Swapping a neural network model on edge vision hardware

You have a working edge vision pipeline: a detector finds objects, a second network classifies or re-identifies them. Now you want to replace one of the two models, and the device refuses to start, or worse, it starts and skips inference on every frame. The fault is almost never the model itself. It is a mismatch between what the new model expects and what the rest of the pipeline supplies.

This guide is for engineers running multi-model inference on embedded accelerators and fanless vision controllers or smart cameras. You will get a repeatable swap procedure: what to align before you compile, how to size buffers from tensor dimensions, how to read the resource warnings, and how to verify the swap in under an hour.

Why model swaps fail on edge hardware

On a workstation, an inference runtime reshapes graphs on load and buffers flex to the input. Edge accelerators do neither. They run models compiled ahead of time into device-specific binaries, and the pipeline allocates memory from the declared tensor shapes at start-up. Three things therefore have to agree:

• The input tensor shape of each model versus the images the pipeline actually feeds it.
• The pre-processing (resize, crop, normalisation) between camera output and model input.
• The runtime version the model binary was compiled for, which must be the same for every model loaded into one pipeline.

Break any one and you see a different symptom, which is why the errors look unrelated. The sections below map symptom to cause, then give the full procedure.

Match the input tensor shape first

The most common failure is feeding the new model images of the old model's size. A detector trained on one resolution swapped in for one trained on another leaves the pipeline sending, say, 544 x 320 frames to a node that declared 672 x 384. The device rejects the data at the node: the classic message is that the input exceeds the available data range, and inference is skipped for that frame.

The fix is to read the new model's declared input dimensions and change the pipeline's image source or resize stage to match, not the other way round. If the model expects 672 x 384, resize the stream to 672 x 384 (or 672 x 383 if the model's aspect logic demands it, but check the export script before rounding). Never pad the buffer silently: a wrong-sized buffer can also produce a misaligned image that passes shape checks and simply gives bad results.

When the two stages differ in resolution, put the resize between them explicitly. The detector's output crops must be resized to the second model's input, every frame, with the same interpolation and normalisation used in training.

Compile all models against one runtime version

When a pipeline loads several compiled models, the device pins a single runtime version. A model compiled for a different version is refused at load with an incompatibility error, even though the model is perfectly valid on its own. This bites exactly when you swap one model: the new binary comes from a newer export toolchain than the old one.

The rule is simple: re-export and recompile every model in the pipeline with the same toolchain version, then declare that version in the pipeline configuration so the device initialises to it. Mixing versions in one pipeline is not supported; aligning them takes minutes and removes a whole class of load-time errors.

Size the buffers from the tensor, not the camera

A related failure: the shapes match on paper but the buffer handed to the node is the camera frame's size. A 672 x 384 three-channel 8-bit input needs 672 x 384 x 3 = 774,144 bytes, roughly 756 KB. If the pipeline hands over a 1.5 MB frame instead, the node reads only the declared range and the rest is ignored, or the transfer is rejected outright.

Worked example: you swap in a re-identification model with a 672 x 384 input while your detector runs on 544 x 320.

1. Detector input: 544 x 320 x 3 = 522,240 bytes per frame.
2. New model input: 672 x 384 x 3 = 774,144 bytes per frame.
3. The hand-off must crop the detector's regions, resize each crop to 672 x 384 and write exactly 774,144 bytes into the second node's input buffer.

Check the write size in code, not by eye. If your framework allocates buffers from the model description, this is automatic; if you manage buffers yourself, assert the byte count before each transfer.

Read the resource warnings before you tune

After a successful load, edge runtimes often print warnings: the model was compiled for fewer execution units than the device has, or a single inference thread is assigned where more would help. These are orientative, calculated as if the model ran alone. With two or more networks sharing the device, the optimal split is different, and the warning for one network does not account for the others.

The safe sequence:

1. Deploy first with the compiled defaults and measure end-to-end latency and throughput.
2. Only if you miss your cycle time, recompile the models with more execution units or threads and measure again.
3. Re-measure after every change; gains on one network can cost the parallel one.

If both models are small, the defaults are often fine. If the pipeline is one large detector plus a per-crop classifier that runs rarely, tune the detector and leave the second stage alone.

Hardware that fits multi-model pipelines

A two-network pipeline needs a controller with enough LAN or USB ports for the cameras, headroom in RAM for both models' buffers, and enough sustained CPU for pre-processing. For GigE cameras on a shared switch, budget network bandwidth the same way; our article on machine vision network congestion: diagnose transfer faults covers the arithmetic for multi-camera links.

With a 32 GB memory option and eight built-in light channels, a controller in this class leaves room for several models and their buffers. Check the vision controllers category for the full range, including models with PoE camera ports that remove separate camera power supplies.

Where inference load is heavy, an AI inference terminal offloads the neural work from the main controller. Pairing one with a standard controller keeps acquisition and I/O on proven hardware while the accelerator handles the networks.

For simpler cells, a smart camera runs inspection on the device with no PC at all, and model updates go through the vendor's deployment tooling.

Step-by-step: the swap

Do the steps in this order; each one catches the class of error the previous step ruled out.

1. Record the old pipeline. Note each model's input width, height, channels, byte depth and normalisation, plus the runtime version of every compiled binary.
2. Read the new model's spec. From its export configuration, write down the same five values. Any difference from the old model is a change you must handle in the pipeline.
3. Recompile everything to one runtime version. Re-export the unchanged model too, so all binaries share a toolchain version, and set the pipeline's declared version to match.
4. Adjust the image path. Set the camera stream or resize stage to the new model's input size. Resize between stages where detector crops feed a second network.
5. Size and assert the buffers. Compute width x height x channels x bytes per pixel for each model and assert the byte count at every hand-off.
6. Load with defaults. Deploy without tuning threads or execution units. Note any resource warnings but change nothing yet.
7. Verify output, not just start-up. Feed known parts or scenes and check detections and classifications against expectations. A pipeline that starts cleanly can still mis-infer if normalisation is wrong.
8. Measure, then tune. Compare cycle time with the old pipeline. Only if you miss target, recompile with more execution units and re-measure.
9. Save the configuration. Back up the pipeline definition and model binaries so a restart cannot leave you with a mismatched set; our article on missing vision models after restart: configuration recovery covers what to persist.

Verify the swap worked

Run these checks before you call it done:

• Pipeline starts with no version or shape errors in the log.
• Frame counters on every node advance; no node skips inference.
• A known test part produces the expected detection and classification, twice in a row.
• End-to-end latency is within your cycle-time budget with the line running.
• A deliberate bad input (blank image, missing crop) fails gracefully rather than crashing the pipeline.

For deployed systems, fold the swap into a wider validation pass; the machine vision software upgrade validation checklist gives a fuller list for changes that touch a running line.

Common mistakes

• Rounding the input size. 672 x 383 instead of 672 x 384 fails shape checks or distorts the image. Copy the declared dimensions exactly.
• Swapping the binary, not the export. Recompiling only the new model and keeping an old-toolchain binary for the other stage invites a version conflict.
• Ignoring normalisation. Mean and scale values travel with the model definition, not the pipeline. A swap that keeps old normalisation often starts cleanly and infers nonsense.
• Tuning before measuring. Acting on single-network resource warnings in a two-network pipeline can make the combined pipeline slower.
• Undersizing RAM. Two large models plus camera buffers can exhaust memory on small controllers. A crash under load looks like an acquisition fault; our article on multi-camera acquisition freezes: callbacks and buffer queues helps tell them apart.

Quick checklist before you deploy

• All models compiled with the same runtime version, declared in the pipeline.
• Input shapes, channels and byte depths recorded for every model.
• Resize and normalisation between stages match the new model's training configuration.
• Buffer byte counts asserted at every hand-off.
• Resource warnings logged but not acted on yet.
• Latency and throughput measured against the old pipeline.
• Pipeline definition and binaries backed up with the deployment.

When the usual answer is wrong

Sometimes you fix shapes, versions and buffers and the model still will not load. Two causes are worth checking. First, model family mismatches: some re-identification or embedding models are trained tightly around a specific class of object, so a detector swaps cleanly while the second-stage model does not transfer. Second, memory: a larger model that runs on a GPU workstation can exceed the edge device's budget once buffers are allocated. The honest fix is a smaller variant of the model, or moving the heavy network to an inference terminal while the edge device handles acquisition.

If you are building the pipeline around a 3D sensor rather than a 2D camera, the same alignment rules apply to the depth and texture streams; see change 3D camera depth resolution and field of view at capture time for how stream configuration interacts with downstream processing.

Ready to spec the hardware side? Start from the vision controllers category or the camera finder, or contact us with your camera count, model sizes and cycle time and we will suggest a configuration.