Why do multiple neural networks crash one edge vision camera?

You stacked face detection, head pose and face recognition on one edge vision camera, and the pipeline runs — until the first face arrives. Then a burst of "Saving face..." lines floods the console and the host raises RuntimeError: Communication exception, with the original message Couldn't read data from stream: 'frame' (X_LINK_ERROR), at frameQ.tryGet(). Some runs never get that far: the process dies with a bare zsh: segmentation fault instead. Nothing is wrong with your models. What follows is the cause order to work through, the SHAVE-core arithmetic that decides how many networks the device holds, the exact library pin and processor line that stop the crash, and the checks that prove the fix held.

Why does the crash land exactly when recognition starts?

Because that is the first moment all three networks, the image manipulator and the script node move frames at the same time — the device stops servicing the stream, and the host reports a communication failure rather than a model failure.

The traceback ends at frameIn = frameQ.tryGet(): the host asked for the next frame and the read came back broken. The variant with no Python traceback at all lands inside the host library's own crash handler, with objects like depthai.cpython-38-darwin.so in the stack. Same event, seen from two sides.

Two details date the failure. The crash appeared immediately after new requirements were installed on a pipeline that had worked, which makes version drift the first suspect. And every run prints the same boot warning — the recognition blob was compiled for 4 SHAVE cores while 13 are available — right before it dies. The warning is context, not the cause; the arithmetic behind it still decides how many networks fit.

What is X_LINK_ERROR actually telling you?

X_LINK_ERROR means the host asked the device for a packet and got nothing back — the transport, the firmware, or an on-board allocation died; in a multi-network pipeline the order is version drift first, script-node memory second, USB link third.

Work that order, not the reverse. The first two tests cost one reinstall and one line of code, and they clear the reported cases; only chase the cable once the pins and the processor line are in place.

How do three networks share 13 SHAVE cores?

Three networks at 4 SHAVE cores each need 12 of the 13 available cores, so they fit with one spare; taking the 6 cores the single-network hint suggests would need 18 and does not fit — with three networks loaded, keep every blob at its compiled 4 cores.

The boot warning compares one blob against the whole device and suggests 6 cores would perform better. The vendor's own caveat: "The issued warnings are orientative, based on optimal settings for a single network, if multiple networks are running in parallel the optimal settings may vary."

The worked arithmetic for this pipeline:

• Detection, head pose and recognition at 4 cores each: 3 x 4 = 12 cores against 13 available, one spare. Fits.
• The same three at the suggested 6 cores: 3 x 6 = 18 cores, five short. The pipeline may still boot, then die under load — exactly the signature above.
• Two networks only: 2 x 6 = 12 cores, so the hint is safe there. The limit lands at three.

Cores are fixed when the blob is compiled, so keeping 4 means declining to recompile, not changing a runtime setting. When you do recompile — swapping a model or shrinking its input — redo this sum the same day; swapping a model on edge vision hardware covers the rest of that job.

Why does the script node's processor choice matter?

Pin every Script node to the LEON_CSS control core with script.setProcessor(dai.ProcessorType.LEON_CSS); in the builds that crashed, unpinned script glue collided with the networks' on-board memory, and that collision surfaces as the dead stream you are chasing.

script = pipeline.create(dai.node.Script)
script.setProcessor(dai.ProcessorType.LEON_CSS)

LEON_CSS is the general-purpose control processor. Without the explicit pin, a script that assembles and forwards frames competes for the same memory the networks use, and the failure arrives as a dead stream rather than a helpful exception. The library build carrying the script-node memory allocation fix, plus this line, is the combination that stopped the crash in the pipeline that surfaced it — recognition held steady instead of dying mid-save. Keep the line in every pipeline that has a Script node, including after you upgrade the library.

Which library version do you pin, and how?

Install the exact versions the pipeline shipped with — python3 -m pip install -r requirements.txt run from the project's own folder — and hold that combination on every machine; the failing pipeline was stable immediately after the pinned reinstall.

This is the same class of failure as SDK version pinning for edge inference crashes, with more networks in play. The full sequence:

1. Reinstall from the project's requirements.txt. That file is the tested combination of library, firmware expectation and blob toolchain.
2. If you need the specific fix build rather than the project pin, take it once, by exact name — depthai==2.11.1.1.dev+7498afce36f34dae5482c2bbf42db317e3e53dd5 from the vendor's own snapshot index — and hold it. Later stable releases carry the same fix; what matters is that the choice is deliberate and identical everywhere.
3. Read the installer's output. In the reported case the resolver warned that depthai-sdk 1.1.2 requires depthai>2 while an older build was in place — two packages disagreeing about the version. One library, one pin, everywhere: the container-build lesson on pinning one library version is the same rule.
4. Match the toolchain. When a blob is compiled with a different OpenVINO release than the pipeline default, set the pipeline's OpenVINO version to the blob's version; a mismatched blob will not behave.
5. Treat device firmware and host library as a pair — camera firmware version mismatch covers matching them without losing detection in the transition.

When does the standard advice make it worse?

Four common moves spend your time on causes that are not there: upgrading every component to its latest release, recompiling blobs for the 6-core hint, catching the exception to retry, and buying a faster host — none of them touches the real failure.

• Upgrade everything. The crash arrived right after new requirements were installed on a working pipeline. Latest-of-each is a combination nobody tested together; a pin is a combination somebody did.
• Recompile for 6 cores. The hint is computed for one network alone; with three loaded it demands 18 cores against 13 and turns a boot-time warning into a load-time crash.
• Catch and retry. Once the stream dies the device stays wedged, and the retry loop collects the same exception. Power-cycle the device, then fix the cause.
• Buy a faster host. The segmentation fault lands on the host, but the failure starts on the device. Host CPU never enters the equation.

When should a network move off the device?

Move a network off the device when three networks at 4 cores each still drop the link or starve the detector; the split that works keeps detection and head pose on the device, recognition on the host, and sizes the camera for the detector.

Planning inference across camera and host frames that decision, and whether the whole pipeline fits on the edge device is the budget question to answer first. Once recognition runs on the host, size the camera for the detector. Worked example for a part-inspection variant of the same pipeline:

• Smallest feature to find: 0.5 mm, at a working floor of 4 pixels across it, so 4 divided by 0.5 mm gives 8 pixels per millimetre.
• Scene width 200 mm, so 200 mm x 8 px/mm = 1,600 pixels across the width.
• A 12 MP sensor in 4:3 delivers about 4,000 pixels wide — a 2.5 times margin over that floor.

For the host side, feed that detector from area scan cameras — a 12 MP global-shutter USB3 model such as the MV-CH120-11UC:

If you would rather keep everything in one housing, smart cameras do the split for you. A smart camera on an x86 platform such as the LS-SMC-0017 runs all three networks as ordinary processes, so SHAVE allocation stops being the constraint:

Compact fixed-pipeline options such as the MV-SC5060XM-08M-WBN keep the networks on the device with the lens and light already matched. Stereo depth sensors that run their own on-board networks follow the same arithmetic — check the core budget before adding one to a 3D sensor pipeline.

What do you check before calling it fixed?

Ten minutes of continuous running per pipeline path — with the SHAVE sum re-checked, the processor line in place, and a clean install from the pinned requirements — is the bar before you call it fixed; thirty minutes before release.

• A clean machine installs from the pinned requirements.txt with no resolver warnings.
• script.setProcessor(dai.ProcessorType.LEON_CSS) is present on every Script node.
• The SHAVE arithmetic holds, redone after any network is added or recompiled.
• Blob toolchain and the pipeline's OpenVINO version agree.
• The device sits on a board port or a powered hub and enumerates at USB 3 speed.
• The soak run reaches full length with frame count and recognition hits stable to the end, including after one mid-test power-cycle.
• The pins are written down where the next engineer finds them.

If the split needs a parts list behind it, start at the camera step of the system configurator or bring the pipeline log to the contact page and we will work the budget with you.