Run arm64 AI accelerator containers in Kubernetes

The accelerator vendor's all-in-one AI suite downloads as an x86 image, and that stops nothing: an arm64 inference worker never needs it. Build a lean runtime-only container instead — kernel driver on the host, the runtime library, its Python bindings, the pipeline framework and its bindings inside the image — and hand the accelerator to Kubernetes through a device plugin. Models are compiled once on your x86 workstation and shipped to the cluster as files. This guide walks the four steps in order: driver, image, device plugin, smoke-test pod. It ends with the arithmetic that says how many inference pods your workers actually carry, and the version rule that prevents the most common false alarm — hardware that is not broken at all.

Why the suite image is x86-only, and why that is fine

The suite bundles three different jobs: training, model compilation — the dataflow compiler (DFC) — and deployment. Compilation is the x86-only part, and it belongs on a workstation anyway. A worker on the cluster deploys: it loads a compiled model, feeds it frames, returns results. For that it needs exactly five things, and only four of them live in the image:

• the kernel driver, on the host
• the runtime library, in the image
• the runtime's Python bindings, in the image
• the pipeline framework, in the image
• the framework's Python bindings, in the image

Before you build anything, check whether the vendor already publishes an arm64 runtime container. Some ship secure-by-default images that pair their runtime with ONNX Runtime; if one exists for your stack, use it and skip the build. If not, build from the vendor's public arm64 runtime packages. One hard rule: the suite image is a vendor download. Never rebuild it from a copied Dockerfile or re-host someone else's — the runtime packages are the supported path. Licensed x86 tools stay on the workstation for the same reason; software that does end up on arm64 Linux can trip its license checks there, and a CODESYS soft container license failure on arm64 Linux has its own resolution walk-through.

What you need

• arm64 nodes carrying the accelerator, each on an OS release the vendor's driver supports
• the accelerator's kernel driver and the four arm64 runtime packages: runtime library, runtime Python bindings, pipeline framework, framework Python bindings
• an x86 workstation with a container build toolchain that can produce arm64 images
• a Kubernetes cluster; a lightweight distribution on small boards is enough for the control plane
• the vendor's or community's Kubernetes device plugin for the accelerator
• one compiled model per network, built on the workstation
• a registry the workers can pull from

Pick the runtime builds marked _arm64 — the default download is x86, and that suffix is the single most common build mistake. If some nodes also carry cameras, device pass-through on arm64 boards has its own write-up: Run a CSI camera in containers on an embedded board. The same thinking applies to any local device a pod must open.

Step 1 — Driver and runtime on the node, bare metal

1. Power the node off, seat the accelerator module, and boot the vendor-supported OS.
2. Install the kernel driver from the vendor's arm64 package, then reboot.
3. Run the vendor's probe command. It must list the accelerator. If it does not, stop here — no container will ever see the device either.
4. For a bare-metal smoke test, install the runtime packages on the host as well and run one model. The cluster does not need them there; the image carries the runtime.

The rule the whole design hangs on: the kernel driver never goes in the image. It loads on the host at boot, and containers reach it through the device nodes it creates.

Step 2 — Build the arm64 runtime image

1. Start from an arm64 base image that matches the node OS and libc, so the vendor's packages install clean.
2. Install the four userspace packages, taking the _arm64 builds. The package suffix is the trap: extract the runtime for ARM, not the x86 default.
3. Copy the compiled models into the image, or mount them from cluster storage at run time if several networks share a node.
4. Build on the x86 workstation for the arm64 architecture. Cross-builds fail on missing dependencies before they fail on anything interesting; Docker builds for arm64: fix find_package dependencies covers that failure mode.
5. Tag the image with the runtime version and push it to the registry.

The size discipline matters more than the build mechanics: four packages plus models, nothing else. An image that boots a compiler or a training tool is the wrong image for a worker, and the suite image is exactly that.

Step 3 — Hand the accelerator to Kubernetes

Kubernetes does not know a PCIe accelerator exists until something advertises it. That something is the device plugin: a DaemonSet the vendor or its community publishes, which detects the hardware on each node and registers it as an extended resource. Run it with a node selector so it lands only on nodes that carry hardware; a plugin on a plain node registers zero devices and clutters scheduling.

A smoke-test pod then requests one device:

apiVersion: v1
kind: Pod
metadata:
  name: accelerator-smoke-test
spec:
  restartPolicy: Never
  nodeSelector:
    accelerator: enabled        # label only on nodes that carry the module
  containers:
    - name: runtime-test
      image: registry.internal/accel-runtime:4.15.0   # your registry path and version tag
      resources:
        limits:
          example.com/accelerator: 1   # the extended resource name your device plugin registers

Two names in that file are placeholders for your environment: the image path and the extended resource name. Describe the node after the plugin runs and the resource appears under capacity. If your plugin does not mount the device nodes itself, add the mount in the pod spec — the mechanics are the same as handing a camera to a container, which the USB camera CDI spec guide walks through.

Step 4 — Schedule the test pod and prove the device works

1. Apply the smoke-test pod.
2. When it completes, read its logs. The runtime's device-list line must show the accelerator, exactly as the bare-metal probe did in Step 1.
3. Run one compiled model on a test input inside the pod and compare the result against the workstation run. Same input, same output: the pinning is right.
4. Delete the pod, then schedule the real workload with the same resource request.

Decision rule, and it is a firm one: device listed inside the pod means driver, runtime and scheduling all agree. Device missing while the bare-metal probe still lists it means a version mismatch or a missing device mount — not broken hardware. Read the next section before you reseat anything.

Cluster arithmetic: what the workers actually give you

Take a concrete five-worker layout: 20 arm64 cores and 20 GB of memory across the accelerator workers, one accelerator per worker, plus four storage nodes carrying 2 TB each on replicated block storage. The per-node numbers fall out immediately: 20 ÷ 5 = 4 cores and 4 GB per worker.

Choose a reservation for the OS and the Kubernetes agent, set it once per node, and keep it honest. At 0.5 core and 1 GB reserved:

The number that schedules inference is the second-to-last line: five concurrent pods, each owning one accelerator. Tighten the reservation to 1 core and 1.5 GB and the cluster drops to 5 × 3 = 15 CPU cores and 5 × 2.5 = 12.5 GB of schedulable memory; the accelerator count does not move. That is the pattern to internalize: accelerator devices cap inference pods; CPU and memory cap preprocessing. When the plan adds three accelerator modules, the cluster goes to eight pods, and you add CPU headroom only if preprocessing queues grow with it.

Version pinning: the failure that looks like broken hardware

The signature failure of this architecture: the pod schedules, the device node is mounted, and the runtime opens nothing. Treat that as a driver-to-runtime mismatch until proven otherwise. One reported arm64 container ran runtime v4.15.0 against an older host driver — hardware mounted, no communication, and the fix was a version match, not a reseat.

The rule that prevents it: one version across the host driver, the runtime library, the bindings, the pipeline framework and the model files. Upgrade the driver and rebuild the image in the same maintenance window, keep the previous image tag for a one-command rollback, and run the smoke-test pod before real workloads come back. The deeper mechanics of pinning an SDK across a fleet are in Edge inference device crashes: fix with SDK version pinning.

Common mistakes

• Reconstructing the suite image on arm64. It is x86 by design; a rebuilt copy breaks licensing and support. The vendor's public runtime packages are the path.
• A driver inside the image. It loads on the host or nowhere.
• Mixed versions after a node rebuild. The pod schedules, the device disappears; re-pin and re-run the smoke test.
• Third-party inference images without the userspace runtime. Popular ready-made camera and NVR stacks often omit the accelerator's runtime entirely. Derive your image from your own runtime base, not from theirs.
• Expecting devices with no device plugin. The pod sits Pending and its events report insufficient of the extended resource — no plugin is advertising on that node.
• An x86 image on an arm64 node. It fails at start with an exec format error; check the image architecture first, the cluster second.
• A DaemonSet with no node selector. The plugin lands everywhere, registers devices nowhere useful, and masks true capacity.

When not to run this stack

Five nodes need updating, monitoring, storage replication and a person who understands them. Spend that only where the workload pays it back.

• One camera, one model, pass or reject: a cluster is overhead. A single x86 vision controller runs acquisition, processing and inference in one industrial box with PoE camera ports and built-in light channels — no scheduler can move your workload because there is no pod.
• Frequent retraining or recompilation: that stays on the workstation regardless; the cluster only serves compiled models.
• Hard frame-timing on the line: keep the frame grab and the first processing stages local to the camera node and send only heavy post-processing outward; the edge vision inference deployment planning guide walks that camera-to-host split.

For the line-side case, two controllers cover most builds. The entry model pairs an Intel Core i3 with 8 GB of memory, a 128 GB SSD, five or more LAN ports, three or four USB 3 ports and 4 built-in light channels, with the VM6200Pro license included — the MV-VC3301H-128G66.

The heavier model moves to a 12th-generation Intel Core i5 with 32 GB of memory and a 256 GB SSD, five or more LAN ports, PoE camera ports and 8 built-in light channels — the MV-VC5525-256G70-NN.

Both sit in the vision controllers category, carry the 3-year warranty minimum and come with free expert support on the order.

Checklist before you scale out

• Record the host driver version on every node and pair it with the image tag that matches. The mismatch above costs hours; this line costs seconds.
• Confine the device plugin DaemonSet to labeled accelerator nodes so capacity reporting stays true.
• Run the smoke-test pod in CI after every image rebuild, before production pods return.
• Compile models against the deployed runtime version and keep them on the replicated storage volume, not on one node's disk.
• Pull the image on one worker first after each push; a pull failure masquerades as a scheduling failure.
• Keep the previous image tag in the registry for one-command rollback.
• Re-check the reservation arithmetic whenever you add nodes — the accelerator count caps inference pods, and it should match your pod budget.

When the workload justifies the cluster, this stack is stable: driver on the metal, four packages in the image, one version everywhere, devices advertised by the plugin. To keep going, the systems guides collect the rest of the deployment write-ups, and the /contact/ page reaches the applications team when you want a second opinion on the node split.