On-sensor AI cameras: when to infer at the camera or the host

AI inspection now ships in two hardware shapes. A new class of smart camera runs the network inside the sensor and sends a verdict down the wire; the familiar shape streams every pixel to a host that does the thinking. Choose wrong and you either own a model that never fits the sensor or a GigE link drowning in frames the decision never needed. Two numbers decide it: the quantized model size in megabytes, and the inspection traffic in megabytes per second. Both are worked through below for a 12 MP camera at 30 fps, with the byte arithmetic shown, a comparison of the three hardware paths, and a decision rule for each case.

What does an on-sensor AI camera do differently?

It runs the network inside the sensor package: the pixel die hands each frame to a CNN processor that holds the quantized weights in about 8 MB of on-chip SRAM. Only the result — a class, a box list, an anomaly score — leaves the camera alongside the image.

The sensor is stacked. Pixels sit on the top die and the AI engine on the die below, moving data between them far faster than any external camera interface could. A first image pipeline turns each frame into a small input tensor for the network, while a conventional second path keeps doing normal camera work — gain, white balance, ROI — in parallel. The stream carries both, and you choose what leaves the camera: the full image, the input tensor, the output tensor, or an image shrunk to a 4 × 4 pixel token when only the verdict matters.

Plan on this envelope for current industrial AI sensors; the converter tool the vendor ships enforces every line of it at deployment:

• Weights: about 8 MB of on-chip SRAM, of which roughly 6 MB is usable budget for weights.
• Input tensors: 256 × 256 px for classification, 320 × 320 px for detection, 512 × 512 px for anomaly detection.
• Decision rate: roughly 30 inferences per second at full-resolution readout.
• Power: the inference engine runs in the hundreds of milliwatts, against tens to hundreds of watts for a desktop GPU.

In the catalogue these devices sit among the smart cameras and vision sensors, which are built for inspection on the device with no PC; the wider camera range is on the cameras topic page.

How much data does on-sensor inference save?

About 100× when the host keeps the 320 × 320 input tensor and the result: a 12 MP camera at 30 fps needs 369 MB/s to stream full frames and about 3 MB/s to send tensors and verdicts.

Worked through for a 12 MP mono camera, 8-bit, at 30 fps:

• Full frame: 4056 × 3040 px × 1 byte = 12.3 MB.
• Full stream: 12.3 MB × 30 = 369 MB/s. A GigE link carries about 118 MB/s of usable payload (125 MB/s line rate less protocol overhead), so full 12 MP frames top out near 9-10 fps on the wire — the interface is the limit, not the sensor.
• On-sensor path: the host takes the 320 × 320 tensor (320 × 320 × 1 B = 102,400 B ≈ 0.10 MB) plus a result record of about 1 KB. Thirty decisions cost 0.104 MB × 30 ≈ 3.1 MB/s.
• Ratio: 369 ÷ 3.1 ≈ 120 — call it 100× with the tensor attached. Results alone, with no tensor, sit another hundredfold lower, in the tens of kilobytes per second.

Two footnotes matter. If you archive every full frame anyway, the wire saves nothing; what improves is decision latency and host load, because inference finishes inside the camera instead of after a transfer-and-queue round trip. When the verdict drives a reject gate, budget that end to end as in image based control latency: budget the full feedback loop. A host-side pipeline that cannot keep up with its own frames has the same arithmetic at larger scale — acquisition faster than processing: fix the frame rate gap.

Will your model fit in the sensor?

Count weights, not checkpoint size: an int8 weight costs 1 byte, so about 8 MB of on-chip memory fits roughly six million weights — MobileNetV2-class, and a quarter of what a ResNet-50 needs.

The six-million line is not the chip's headline number. The same 8 MB holds activations, runtime buffers and the converted model container, so budget about 6 MB for weights. Training checkpoints are float at 4 bytes per weight, which is why MobileNetV2's 3.5 million weights occupy around 14 MB before quantization; published int8 conversions land at 3.3-4.4 MB, right where the byte count predicts. A network that converts under the budget runs at the sensor; one that does not goes to the host, and no setting changes that.

Memory is not the only limit — detail is. The engine answers at tensor scale, 256 to 512 px, so a hairline crack or small laser-etched characters spread across 4,000 px may not survive the downsampling. Test before buying: image your worst defect, downsample it to the tensor size, and check it is still visible. How far sensor pixels can be pushed before detail disappears is its own subject in small pixel camera selection: resolution versus usable detail. Full-int8 conversion typically costs about one percentage point of classification accuracy, and quantization-aware training wins most of that back.

When should the host do the inference?

Whenever any one of three things fails: the int8 model exceeds about six million weights, the question needs more detail than a 256-512 px tensor or more than about 30 decisions per second, or the pipeline needs pre- or post-processing the sensor cannot run.

In practice that covers most deep inspection: full-resolution multi-class detectors, segmentation masks, measurement-grade blobs, crops fed to a second network, frames fused across several cameras, or custom post-processing you want beside the inference. A host also absorbs model churn — when the network changes monthly, a controller takes new weights over the network without touching the camera.

Cameras already installed can gain host AI without being replaced; adding AI surface defect inspection to existing cameras covers that retrofit path.

How does a model get into an AI camera?

You train in float precision, quantize to int8 and convert with the camera maker's toolchain into the packed model the sensor loads once; after that the camera infers offline, with no cloud call and no host in the loop.

The makers of these sensors publish open reference collections — quantized training scripts and sample networks for classification, detection, anomaly detection, pose estimation and segmentation — that already respect the tensor sizes and memory layout. The practical route: start from the reference closest to your task, swap in images of your own parts, retrain, quantize, convert, and upload once from a development machine. The converter rejects anything over the memory budget, so the byte check from the last section happens at your desk, before hardware is ordered. Keep the float model's scores as the baseline; the quantized model must match them on a held-out set of your worst parts, not on the training images.

What hardware sits on each side of the choice?

The on-sensor path is a single device — a smart camera or vision sensor; the host path pairs an area scan camera with a vision controller, or with an AI inference terminal once the networks outgrow CPU-class compute.

On-device inspection hardware runs from a 0.3 MP micro vision sensor for presence and orientation up to 6 MP smart cameras with an integrated 8 mm lens, all inspecting on the device with Fast Ethernet or GigE out the back and results straight into the PLC.

The host path starts with the camera. A 2.5 MP GigE global-shutter model suits tensor-scale classification; 12 MP USB3 global-shutter high-speed models keep full sensor detail for host-side networks. The camera finder filters both by resolution, interface and shutter.

Compute that holds the model follows. A vision controller with a 12th-gen Intel Core i5, 32 GB memory, PoE camera ports and 8 built-in light channels runs classic and mid-size networks beside the cameras, and its 5+ LAN ports hang a whole cell off one box; beyond that, an AI inference terminal takes the GPU-class models. Both live in the vision controllers category.

A host need not be a full PC either: smaller embedded controllers can drive a camera directly, as in this walkthrough of driving a USB camera from a microcontroller host.

Which should your line run?

Run the on-sensor path when a network under roughly six million int8 weights, judging the part at 256-512 px tensor scale, answers the question at up to about 30 parts per second across one GigE drop; run the host path when any one of the three fails.

• Pass/fail by appearance — presence, orientation, sort-by-class, obvious surface defects — at up to 30 parts per second: an on-sensor AI camera, with the verdict reaching the PLC without a PC ever touching the frames.
• Measurement, small characters, hairline defects or full-resolution segmentation: an area scan camera into a vision controller, keeping the pixels and the model together on the host.
• A model over the six-million-weight line, or several networks per part: an AI inference terminal; convert once for the GPU instead of fighting the byte count.
• A mixed line: both at once — AI sensors pre-sort at the camera while the controller judges the border cases, placed as the deployment guide above suggests.

Checks before ordering:

1. Write the decision as a class, box or score, and the part rate in per second.
2. Count network weights and multiply by 1 byte; above about 6 MB, budget host hardware.
3. Downsample your worst defect image to 320 × 320 px and confirm it is still visible.
4. Multiply frame bytes by frames per second and check the link: 118 MB/s carries one 12 MP camera at about 9 fps, and nothing more.
5. Choose a camera whose mount and lens family also suits a controller, so a disappointing quantized score costs a move, not a rebuild.

If your numbers land within a factor of two of the sensor's envelope on either side, or the line mixes both paths, talk to the applications team before standardizing.