Move NMS and IoU off the camera to the host when FPS drops

Crowded scenes are what break detection on a smart camera or stereo 3D camera: the network costs the same per frame, but the suppression pass after it grows with the square of the candidate count, and the device's general-purpose cores pay for it. The fix is architectural, not a setting. This guide is for integrators who must hold 30 fps on a line that fills up. Follow it and you will keep inference on the device, stream raw per-class boxes and scores to an x86 host, and run non-maximum suppression (NMS) with intersection-over-union (IoU) there — with the bandwidth math, the host-side code, and the cases where the move is the wrong fix.

Why does the camera CPU spike when detections pile up?

Because NMS compares every same-class candidate against every other one, the work grows as N × (N − 1) / 2: 100 boxes cost 4,950 comparisons and 1,000 boxes cost 499,500, so a ten-times busier scene costs a hundred times the CPU.

NMS is the deduplication step after a detector: sort the candidate boxes by score, then delete every box that overlaps a higher-scoring box of the same class by more than a threshold fraction. That fraction is IoU — the boxes' shared area over their combined area — and each survivor must be tested against every candidate left in its class.

The candidate count is huge before filtering. An anchor-free YOLO-style head at 640 × 640 input predicts one box per grid cell at three strides: 80 × 80 + 40 × 40 + 20 × 20 = 8,400 raw boxes per frame. A confidence filter at 0.25 removes most of them on a calm line; a full tray leaves hundreds per class.

The accelerator runs the convolutions; suppression runs on the CPU cores beside it, usually single-threaded and reached through a scripting layer. That is why frame rate holds on an empty tray and collapses on a full one while the model itself never changed.

What exactly do you move off the camera?

Only the post-processing: keep the network, its confidence scoring and the accelerator exactly where they are, and move the suppression pass — sort, top-K and NMS built on IoU — onto the host, fed with raw per-class boxes and scores.

The accelerator is built for convolution and already paid for; what scales badly is everything after the score. You trade a few dozen lines of host code for deleting the only quadratic task in the pipeline.

This split — inference on the sensor, output handled by the host — is a documented deployment pattern, not a workaround: the export guides for AI-vision camera platforms describe sending the network's raw output to the host application for post-processing. Exports that baked suppression into the model graph are why this move used to be painful; re-export the network instead of editing the old graph.

For the wider question of what runs where — camera, controller, or both — read Plan edge vision inference deployment across camera and host.

How do you stream raw detections to the host?

Export the model with suppression turned off, filter on the device by confidence only, then stream the survivors as three flat arrays — boxes, scores, class ids — which costs 24 bytes per candidate box.

1. Re-export the model with its suppression stage disabled, so the output is raw boxes, scores and class ids. The setting lives in the export or deployment tool of your camera platform, on the detection output stage — look for options named after NMS, suppression or post-processing, or a raw-detections output mode. The behavioral test: raw output returns more boxes than the scene has objects; suppressed output returns about one box per object.
2. Leave the lightweight, linear filters on the device: confidence threshold 0.25, and a top-K cap of 300 if the tool offers one. They cost almost nothing there and shrink the wire payload by about 28×.
3. Stream three flat arrays per frame on the camera's existing link: boxes as float32 shaped (N, 4) in x1, y1, x2, y2 order, scores as float32 (N,), class ids as int32 (N,), plus a frame counter and timestamp so the host can align boxes with images.
4. On the host, run per-class NMS (code below), then the final top-K, then coordinate transforms and the PLC hand-off — in that order.
5. Time both ends: device frame period before and after the change, and host NMS time per frame. One number on each side is enough.
6. A/B the switch on a recorded worst-case scene: at the same IoU threshold, device-side and host-side suppression must keep identical box sets.

Interface speed matters more than anything else on the camera side. A GigE model such as the MV-SC5060XM-08M-WBN carries images and raw detections on one cable with room to spare; filter smart cameras by interface in the camera finder.

How much bandwidth do raw boxes cost?

Almost nothing, if you filter first: 300 surviving boxes at 24 bytes each are 7.2 kB per frame, or 1.7 Mbit/s at 30 fps — about 57.6 µs of wire time per frame on a gigabit link.

Per box you send four corner coordinates and a score as float32 plus one class id as int32: 5 × 4 B + 4 B = 24 B. For 300 survivors:

300 × 24 B = 7,200 B = 7.2 kB per frame. At 30 fps that is 216 kB/s, or 1.73 Mbit/s — 0.17% of a 1 Gbit/s link. On a 100 Mbit/s Fast Ethernet link the same payload is 1.7% of the link and 0.58 ms of wire time per frame: still fine.

Sending the unfiltered head instead — all 8,400 candidates — costs 8,400 × 24 B = 201.6 kB per frame, which is 6.05 MB/s and 48.4 Mbit/s at 30 fps. On Fast Ethernet that is half the link spent on boxes the host will delete. The decision writes itself: filter by confidence on the device, suppress on the host.

Detection metadata is small next to the images themselves — even a compressed image frame dwarfs the detection payload — so the detection stream never decides your interface; the images do.

How do you run per-class NMS on the host?

Sort each class by score, then greedily suppress at an IoU threshold of 0.45 while the next frame exposes — identical arithmetic to the device's pass, but on a core with headroom and off the capture loop.

The whole function is a few dozen lines, and the assertions below prove it suppresses within a class and never across classes:

import numpy as np

def nms_per_class(boxes, scores, class_ids, iou_threshold=0.45):
    # boxes: float32 (N, 4) as x1, y1, x2, y2
    # scores: float32 (N,), class_ids: int32 (N,)
    # returns kept indices, ascending
    keep = []
    for c in np.unique(class_ids):
        idx = np.where(class_ids == c)[0]
        idx = idx[np.argsort(-scores[idx])]        # highest score first
        while idx.size:
            i = idx[0]
            keep.append(int(i))
            rest = boxes[idx[1:]]
            x1 = np.maximum(boxes[i, 0], rest[:, 0])
            y1 = np.maximum(boxes[i, 1], rest[:, 1])
            x2 = np.minimum(boxes[i, 2], rest[:, 2])
            y2 = np.minimum(boxes[i, 3], rest[:, 3])
            inter = np.clip(x2 - x1, 0, None) * np.clip(y2 - y1, 0, None)
            area_i = (boxes[i, 2] - boxes[i, 0]) * (boxes[i, 3] - boxes[i, 1])
            area_r = (rest[:, 2] - rest[:, 0]) * (rest[:, 3] - rest[:, 1])
            iou = inter / np.maximum(area_i + area_r - inter, 1e-9)
            idx = idx[1:][iou < iou_threshold]
    return np.sort(np.asarray(keep, dtype=np.int64))

boxes = np.array([[0, 0, 10, 10], [0, 0, 10, 8], [20, 20, 30, 30]], np.float32)
scores = np.array([0.9, 0.8, 0.7], np.float32)

# same class: box 1 overlaps box 0 with IoU 80/100 = 0.80, so it is suppressed
kept = nms_per_class(boxes, scores, np.zeros(3, np.int32), 0.45)
assert list(kept) == [0, 2]

# different classes: nothing to suppress within a class, all three survive
kept = nms_per_class(boxes, scores, np.array([0, 1, 0], np.int32), 0.45)
assert list(kept) == [0, 1, 2]

Time it on your own controller rather than trusting anyone's milliseconds:

import time
rng = np.random.default_rng(0)
xy = rng.uniform(0.0, 600.0, (900, 2)).astype(np.float32)
wh = rng.uniform(8.0, 40.0, (900, 2)).astype(np.float32)
boxes = np.hstack([xy, xy + wh])
scores = rng.uniform(0.30, 0.99, 900).astype(np.float32)
classes = np.zeros(900, np.int32)

t0 = time.perf_counter()
for _ in range(100):
    nms_per_class(boxes, scores, classes, 0.45)
print((time.perf_counter() - t0) / 100 * 1e3, 'ms per frame')

On a current desktop core this prints single-digit milliseconds for 900 candidates; the same algorithm in compiled C++ lands under 1 ms. Treat both as the numbers to beat, with the harness above as the acceptance test.

Set the threshold deliberately. 0.45 matches what the device ran. Raise it to 0.55-0.60 when real objects touch — nested brackets, stacked blanks — so neighbors survive; drop toward 0.30 only when one physical part yields duplicates. Thresholds interact with object size and tiling: Tiled inference confidence and IoU thresholds for small objects covers that case.

Keep the pass off the critical path: while frame N is suppressed on the host, the device already exposes frame N+1. If survivors feed a model on the GPU, keep them in device memory — Feeding GPU tensors to a vision model without a CPU round trip shows the pattern. Order the host work detection-first, crops after: Rotated bounding boxes in live inference explains why the reverse order costs you.

Which host should run it, and how much CPU does it need?

Any current x86 vision controller handles it: 900 candidates cost at most 404,550 comparisons per frame, a vectorized pass finishes that in single-digit milliseconds, and one core comfortably serves two to three cameras at 30 fps.

Budget from the measured pass time: at 5 ms per frame and 30 fps, one camera consumes 150 ms of CPU per second — 15% of one core. Four cameras on one controller is 60% of one core, with the rest of the box left for image handling, PLC traffic and the HMI. No GPU is involved; this is plain array math, and Object detection without a GPU covers the wider no-GPU question.

Size the controller to the whole station. For one or two cameras, an Intel Core i3 with 8 GB is more than enough; on a multi-camera line step up to the 12th-I5 model with 32 GB:

All 2 share: Humidity 20% RH to 95% RH (no condensation); Certification CE, KC. Values as printed in each product's data sheet; open a part for its drawing and full specifications.

If adding a controller is the problem, make the camera the host: the LS-SMC-0017 runs an x86 platform, so the raw head output never leaves the process and NMS runs beside it with desktop-class headroom.

Browse vision controllers by processor and port count once your camera count is fixed.

When is moving NMS the wrong fix?

When the candidates are few or the culprit is elsewhere: under about 100 survivors per class the pass is below 5,000 comparisons and cannot cause a collapse, so profile before you move anything.

Break the frame period into exposure, inference, post-processing and transfer, and watch the device's per-process CPU. Three findings change the answer:

• Low candidate counts. If no class exceeds about 100 survivors after filtering, suppression is not your spike. The usual culprits are image encoding (JPEG for a viewer stream is a classic) or the model runtime itself.
• Several cameras on one small board. The cores are shared and the per-camera overheads stack — High CPU usage with multiple cameras on one embedded board works that case.
• No raw-output mode. If the export toolchain cannot disable suppression, the comparisons still run on the device whatever you do downstream. Attack the count instead: confidence 0.25 to 0.40 and a top-K of 100 cap a class at 4,950 comparisons regardless of how crowded the scene gets.

On stereo and structured-light 3D sensors the depth pipeline is usually the heavier load, and moving NMS off will not touch it. A unit such as the MV-DLS300P-04, a laser galvanometer stereo camera, spends its device cycles on depth rather than on box deduplication. When you bench such a camera outside the machine, keep its laser projector beam away from faces and follow the laser-class label on the housing.

What should you check before you switch over?

Ten minutes on the bench: count worst-case candidates, measure the wire bytes, lock the thresholds, and A/B device-side against host-side suppression on a recorded scene before the line ever sees the change.

• Worst-case scene recorded — the fullest tray, worst lighting; every number below comes from it, not from a calm demo.
• Candidates counted per class after the 0.25 filter — any class over 300 confirms the move; all classes under 100 point elsewhere.
• Raw export confirmed — output has more boxes than objects, and the device tool shows suppression off.
• Wire budget measured — detection payload under 5% of the link at full rate.
• IoU threshold locked per scenario — 0.45 default, 0.55-0.60 for touching parts, written into the configuration, not buried in code.
• Host pass timed — the harness above, on your controller, under 5 ms per camera per frame.
• A/B suppression on the recorded scene — device and host keep identical box sets at the same threshold.
• One hour at production rate — device CPU and frame period logged, no drift.

When the checks pass, the system configurator puts camera, link and controller in one build. If your candidate counts are unusual — thousands per class, or a dozen cameras — bring them with your frame-rate target to our contact page and we will size the split.