Running custom object detection models on embedded smart cameras

You have trained a detector on your own parts and want it to run where the image is taken — on the camera, not on a PC. That works, but only for a model sized to the device, and the sizing is arithmetic you can do before you train. This page gives the two numbers that decide it: the int8 model size in kilobytes and the frame rate your compute allows. You get the 4× size saving of 8-bit quantisation, the way input resolution moves compute, a worked example of a 300 kB detector on a small vision sensor with every step shown, and the point at which the right answer stops being the sensor and becomes an x86 smart camera or a vision controller.

What an embedded smart camera asks of a model

Three constraints decide whether your trained network runs on the camera itself. The weights must fit in flash alongside the firmware and your application code. The activations — the intermediate feature maps — must fit in RAM while each layer runs. And one inference, plus image capture and communication, must finish inside the inspection cycle.

Device classes set those budgets. A small vision sensor carries an MCU- or DSP-class processor: plan for weights of a few hundred kilobytes of flash and peak activation memory under about 1 MB. An x86 smart camera runs the same PC-class inference runtimes your training machine does, so capacity stops being the constraint. A vision controller hosts inference for one or several cameras next to the line PLC. Start from the smallest class that clears the checks on this page; it keeps the decision on the device and the wiring short.

The size arithmetic: parameters × bytes

A network's file size is close to its parameter count times the bytes per parameter. A float32 training checkpoint stores each weight in 4 bytes; an int8 quantised model stores each weight in 1 byte.

Quantisation to 8-bit integers is what turns a bench model into one that fits a sensor. A network of 300,000 parameters holds 1.2 MB as float32 and 300 kB as int8 — the 4× cut that brings compact detectors inside a small vision sensor's flash margin. Activations add working memory on top: the input tensor alone, at 96×96 with one grayscale channel, is 96 × 96 × 1 B = 9,216 B ≈ 9 kB, and each layer's output adds more, with the peak set by the widest layer.

The frame-rate arithmetic: MACs against the clock

Detection cost is counted in multiply-accumulate operations (MACs), and MACs scale roughly with the square of the input side, because every output value of every layer is computed from an input neighbourhood. Halving the input side cuts compute about 4×. Going from 160×160 down to 96×96 divides it by (160 ÷ 96)² ≈ 2.8.

Take 100 million MAC/s as the planning number for an MCU-class core in the 300-600 MHz range with single-cycle MAC and DSP instructions. Delivered frame rate lands near half the ceiling once sensor readout, preprocessing and communication take their share. Sustained rates move with memory speed and compiler work, so measure the rate once on your actual device and reuse that number for every later model decision.

Worked example: a 300 kB detector on a small vision sensor

Take a compact one-stage detector: a MobileNet-class backbone with a small SSD-style head, trained at 96×96 on grayscale images. Models of this class carry on the order of 300,000 parameters and about 12 million MACs per inference at that input size.

Size:

• int8 weights: 300,000 × 1 B = 300 kB. The float32 checkpoint would need 300,000 × 4 B = 1,200,000 B = 1.2 MB.
• Input tensor: 96 × 96 × 1 B = 9,216 B ≈ 9 kB.
• Budget check: 300 kB of weights sits inside the few-hundred-kilobyte flash margin of this device class, and peak activation memory for a narrow backbone at this size stays under about 1 MB.

Speed, with the core sustaining 100 million MAC/s:

• Compute ceiling: 100,000,000 ÷ 12,000,000 ≈ 8.3 fps.
• Delivered rate at about half the ceiling: 3-4 fps.

Resolution sensitivity: the same backbone at 160×160 costs 12,000,000 × 2.8 ≈ 34 million MACs per inference, so the ceiling falls to 100,000,000 ÷ 34,000,000 ≈ 3 fps and the delivered rate to 1-2 fps.

Decision for this example: if the line presents at most one or two parts per second per camera and each object spans at least 10-15 pixels in the 96×96 input, the sensor runs the model. If the rate is 5 parts/s or the part detail needs the larger input, move up one device class.

Quantisation: how the model gets to int8

Post-training quantisation converts an existing float checkpoint. You give the converter the model plus a calibration set of a few hundred images, and it fixes a scale and zero point per tensor so that int8 arithmetic reproduces the float network closely. No retraining is involved and the conversion takes minutes.

The accuracy cost is usually small for compact detectors. When the drop exceeds your tolerance, quantisation-aware training inserts the int8 behaviour into training itself and recovers most of the gap while keeping the int8 file size. Calibrate with frames from the camera and lighting you will deploy, and score the quantised model on held-out images before rollout — the float checkpoint's score does not carry over.

Set the input resolution before you train

Input size is a design decision, not a detail to fix after training. The rule: the smallest feature you must detect needs 10-15 pixels across in the network input, which keeps it above the coarse output grid of the detection head.

Work it through the optics. If the smallest defect is 2 mm wide and the field of view at the working distance is 60 mm, the sensor delivers 96 ÷ 60 = 1.6 pixels per mm, so the defect spans 2 × 1.6 = 3.2 pixels — too small for the head. Narrow the field of view to 18 mm and the same defect spans 96 ÷ 18 × 2 ≈ 10.7 pixels, inside the rule. If the field cannot shrink, raise the network input and pay the extra compute from the frame-rate arithmetic above.

The runtime then fits your frame to the network input by resizing and padding; how those dimensions behave is covered in the guide on object detector input dimensions and padding.

Where the sensor class stops, and what to step up to

Four limits end the on-sensor path: more than a handful of classes widens the output head; defects that need pixel-level detail demand input sizes the flash and compute budget cannot afford; a layer your device's converter does not accept; and a cycle faster than 2-3 inspections per second. Any one of them moves the model up the ladder.

For the middle rung, the LS-SMC-0017 smart camera is an x86-platform unit with 8.8 MP resolution, a 1" sensor format, C-mount lens mount and global shutter. Your model exports into a standard runtime format and runs under the camera's PC-class environment, exactly as it does on a bench PC.

When several cameras share one model, a controller hosts the inference next to the line. The MV-VC2140-128G40-1T vision controller pairs an Intel N-series N97 processor with 8 GB of memory, 3-4 LAN ports, 2 USB 3 ports and 4 built-in light channels, which covers a compact multi-camera cell. What a CPU-only deployment delivers is covered in the guide on object detection without a GPU.

For the heaviest models, use hardware built for the job: the MV-VAC3504X-128G01 AI inference terminal carries 5+ LAN ports, 3-4 USB 3 ports and 4 built-in light channels, and its series exists to run neural inference at the line.

Plan which part of the pipeline sits on the camera and which sits on the host with the guide on planning edge vision inference deployment. Once the line runs, model updates become routine; the guide on swapping a neural network model on edge vision hardware shows how to change the model without stopping the cell.

Checks before you commit

• Parameter count × 1 B against spare flash, leaving headroom for the runtime and your application code.
• Peak activation memory against spare RAM, measured with the largest frame you will trigger.
• MACs per inference ÷ sustained MAC rate against the cycle time, with a 2× margin.
• Smallest feature at 10-15 pixels or more in the network input, checked at the real working distance.
• Architecture chosen from what the device's converter accepts, decided before training starts.
• Quantised model scored on held-out images, within the tolerance your defect criteria set.
• Trigger-to-result latency measured end to end, since readout and communication sit on top of inference.

The decision, per case

Work in this order. Set the input size from the smallest feature and your optics. Pick the smallest architecture that clears it and read parameter count and MACs from the export summary. Multiply parameters by 1 byte for int8 and divide MACs by the sustained rate of your device class: if the weights land within a few hundred kilobytes and the delivered rate clears the cycle with a 2× margin, the sensor runs your model, as in the worked example above.

If any check fails once, stop shrinking the network. An x86 smart camera or a vision controller costs less than weeks of forcing an oversized model into a device that was never going to hold it; the smart camera finder filters the x86 rung by resolution, sensor format and mount, and the system configurator walks the camera-light-controller chain end to end. If you are weighing a learned model against classical tools for a new inspection, read the comparison of AI and rule-based smart camera inspection first.