Null frames in a multi camera callback: handle them in the queue

Four depth cameras on one host, one callback, and every so often the callback hands you nothing instead of a frame. The null arrives at random moments, on the master and the slaves, in Python and in .NET, with the sync unit active or bypassed — and most frames are still fine. That pattern is not a broken camera. It is a dropped frame: something on the host — CPU load, link bandwidth, or a lock held a few milliseconds too long — cost that image its delivery slot, and the SDK reported the miss as an empty result. This guide gives you the tests that find which resource ran out, the arithmetic that sizes the load before you order anything, and the queue handling that keeps one missed frame from stopping the line.

What a null frame actually is

When you register a callback at start — the SDK's pipeline.start(config, callback) form — the SDK hands each completed frame set to your function on its own acquisition threads. For ToF and stereo depth cameras the depth map is computed on the host, inside the SDK, once per device. That compute consumes real CPU. When a device's next frame slot arrives and its depth result is not ready, the frame is lost in acquisition; the SDK has nothing to deliver and reports the miss as an empty or null result.

The signature is steady: valid, valid, valid, null, valid. Timing is random, master and slaves drop alike, Python and .NET behave the same, and sync makes no difference. Frames keep flowing after each null, so the link is up and every camera is running. A null here means "nothing was ready for this slot", not "the camera failed".

Keep this separate from the single-image fault where a processing stage inside your own code returns a null image on every run — that one is diagnosed stage by stage in Fix a null image result in a vision pipeline.

The rigs where this bites are continuous-stream depth stations: one ToF depth camera per station, streaming depth to a host that also runs your inspection code.

Likely causes, most probable first

Four causes cover almost every intermittent-null report on a multi-camera rig. Work them in this order.

1. Host CPU saturated by per-device depth computation. The depth map is computed on your cores, once per device, inside the SDK. Load scales with camera count, resolution and frame rate, so a rig that runs one camera cleanly can drop frames at four with no configuration change. This is the first suspect whenever nulls appear only above a certain camera count.
2. Aggregate link bandwidth exceeded. Four streams through one switch uplink, several USB 3 cameras on one controller, or GigE cameras sharing one port family on the host: transfers collide and frames drop in flight.
3. A lock held inside the callback. A global lock shared by four callback threads serializes frame pickup: while device A's thread holds it, device B's frame waits. At 30 fps one frame slot is 33 ms (1,000 ms ÷ 30), and four cameras together deliver a frame every 8.3 ms on average (1,000 ÷ 120). A few milliseconds of waiting per grab eats slots under load — and if the callback does real work such as resizing, converting or logging, the wait multiplies. The right place to copy pixels is covered in Torn frames in a camera callback: copy the buffer before you queue it.
4. Downstream queue backpressure. A full queue makes the producer block or overwrite, and the symptoms look like drops: missing, stale or duplicated frames. See Latest image only returns duplicate frames: drain the buffer queue and Acquisition faster than processing: fix the frame rate gap.
5. Not the sync hardware. Nulls appear with sync disabled too, so exposure timing is cleared. The sync unit aligns when cameras expose; delivery to the host still runs per device on the SDK's threads. Leave it in place and debug the host.

Test each cause in order

1. Read the cores while streaming. Run all four cameras at production settings and watch per-core and per-process CPU (Task Manager on Windows, top or htop on Linux). Cores above about 85% for minutes at a time means the compute budget is spent — cause 1 confirmed. One core per streaming device is the starting budget; depth compute and your callback share it.
2. Run a one-camera baseline. Each camera alone at production resolution and frame rate, ten minutes, null counter on. Zero nulls alone plus nulls at four cameras is aggregate host load, not the cameras. Nulls with a single camera point at that device's acquisition mode or link — 3D camera frame rate low on software trigger: mode and data streams covers the acquisition-side causes.
3. Halve the frame rate. Drop 30 to 15 fps and each camera's byte rate falls by half (the worked example below shows the arithmetic). Nulls stop at 15 fps and you are load-bound: keep 15 fps, or scale the host and go back up. Nulls unchanged: the lock or the physical paths are next.
4. Remove the global lock. One bounded queue per device, the callback pushes without ever waiting, and the consumer merges the queues downstream. If the nulls stop, contention was real — the next section is the permanent version of this test.
5. Check the physical paths. Give each camera a dedicated network port instead of a shared uplink, and put USB 3 cameras on separate controllers. When frames stall or stop updating rather than occasionally dropping, Depth camera frames not updating: fix USB bandwidth stalls is the closer match.

Worked example: four ToF cameras, 1280 × 1024 depth at 30 fps

The inputs, and where each comes from:

• Four cameras — the rig in question: one master, three slaves.
• 1280 × 1024 px at 30 fps — example values, because the original report does not record resolution or rate; substitute your own before ordering hardware.
• 2 bytes per pixel — a 16-bit depth frame, the common depth format.
• One dedicated 1 GbE port per camera — 1 Gb/s ≈ 125 MB/s (1,000,000,000 bit/s ÷ 8).

The arithmetic:

• Bytes per frame: 1280 × 1024 × 2 B = 2,621,440 B ≈ 2.6 MB
• Rate per camera: 2,621,440 B × 30 fps = 78,643,200 B/s ≈ 78.6 MB/s
• In link terms: 78.6 MB/s × 8 = 629 Mb/s ≈ 0.63 Gb/s
• Four cameras: 78.6 MB/s × 4 = 314.6 MB/s ≈ 2.5 Gb/s

The decisions this drives:

• 0.63 Gb/s on its own 1 Gb/s port is 63% sustained — inside the 70% planning rule, which leaves room for control traffic and bursts. A dedicated port per camera passes.
• All four through one shared 1 GbE uplink demand 250% of it. That path fails by arithmetic, not by tuning: the uplink must be 2.5G or 10G, or every camera needs its own port to the host.
• Host side, 314.6 MB/s of incoming copies shares the cores with four depth computations. Halving to 15 fps drops each camera to 39.3 MB/s (2,621,440 B × 15) and the rig to 157 MB/s — the 15 fps test above measures exactly this margin.
• Add color at 1920 × 1080, 3 bytes per pixel, 30 fps and each camera gains 186.6 MB/s ≈ 1.49 Gb/s — more than a 1 GbE port carries on its own. With RGB in the stream, plan 2.5G or 10G per camera from the start.

One rule of thumb covers every path — camera port, switch uplink, NIC, disk: keep sustained use below about 70%. The full four-camera sizing walkthrough, including recording load, is in Multi-camera RGB depth bandwidth: size a four-camera host.

The fix: skip the null and count it

A null is information: that slot is gone, the stream is intact, and nothing downstream should see the gap as data. Handle it in the callback, before the queue:

import queue
import threading

null_count = 0                      # int, total empty deliveries
per_camera = [queue.Queue(maxsize=16) for _ in range(4)]  # one bounded queue per device
count_lock = threading.Lock()       # guards the counter only, never the frame push

def on_frame(device_index, frame):  # runs on the SDK's thread; frame is the SDK's frame object or None
    global null_count
    if frame is None:               # a missed delivery slot: skip it, count it, continue
        with count_lock:
            null_count += 1
        return
    per_camera[device_index].put_nowait(frame)   # never wait inside the callback

frame, device_index, on_frame, null_count and per_camera are names for your own code; queue.Queue, threading.Lock and put_nowait are Python standard library. The signature above is the generic form — match your SDK's actual callback shape, in Python or in .NET.

Three details decide whether this holds up under load. Bound the queues so a slow consumer cannot grow memory without limit — Python frame queue memory leak: bound the queue and reuse frames shows the pattern. Keep the callback to a push: no resize, no conversion, no log calls inside it. If your SDK reuses the pixel buffer between callbacks, copy the data before queuing, or you will queue the next frame's pixels into this slot.

Track the count per camera, not just in total:

How to verify the fix

Soak at production settings for ten minutes. In the worked example that is 30 fps × 4 cameras × 600 s = 72,000 expected frames. A correctly sized host finishes with null_count at zero; accept a soak at up to 0.1% — 72 nulls — and treat anything above as a failed test that sends you back to the CPU and link checks. Frame timestamps should sit on the 33.3 ms period of 30 fps with no gaps, and gaps that line up with nulls confirm the slots were missed in acquisition, not downstream.

Then add the real production load, because the bench test rarely includes it. Recording is the usual missing multiplier: Multi-camera recording to disk without dropped frames shows how to keep disk writes out of the acquisition path.

Prevention: keep the host under the load limit

• Re-run the sizing arithmetic on every change of resolution, frame rate or camera count — the byte rate scales linearly with all three, so doubling fps doubles MB/s per camera.
• Keep sustained use below 70% on every path: camera port, switch uplink, NIC and disk.
• Budget one core per streaming device as the starting point, then confirm with the per-core reading from test 1.
• Keep callbacks thin, queues bounded and per-device, and put the null counter on the line dashboard with an alarm above 0.1% over a rolling minute.

When the load test shows the current host is the constraint, the answer is ports and cores, not new cameras. The MV-VC5525-256G70-NN pairs an Intel Core i5 with 32 GB of memory, 5+ LAN ports with PoE camera ports, 3-4 USB 3 ports and 8 built-in light channels.

A two-camera cell at the example settings carries half that load — 78.6 MB/s × 2 = 157.2 MB/s — and starts lower. The MV-VC3301H-128G66 runs an Intel Core i3 with 8 GB of memory and 5+ LAN ports, without PoE. Either way the tests above decide, not the badge: read the cores and run the bandwidth arithmetic before you order. Browse the vision controllers to compare port counts, PoE and memory classes.

When the usual answer is wrong

• A null on every frame. That is not a drop; it is a pipeline-stage fault — the null image article linked above covers the stage-by-stage probe.
• The callback stops being called. Different failure: Camera streaming callback not called: keep acquisition alive walks the keep-alive causes.
• Stale or duplicated frames instead of empty ones. The consumer is falling behind; drain the queue as in the backpressure article from cause 4.
• One camera drops at every load level. Swap its cable and network port and retest it alone; a single device dropping while three stay clean is a cable, port or device fault, not host sizing.

Next step

Decision, per outcome: nulls stop at half the frame rate — stay at 15 fps or scale the host with the arithmetic above; nulls stop once the lock is gone — keep per-device queues permanently; one camera drops alone — replace its cable or port; every test passes and nulls persist — step the host class up and re-soak. Size the rig from the 3D sensors and vision controller categories, start a build in the configurator, or contact the applications team with your camera count, resolution and frame rate and we will check the arithmetic with you. The rest of the depth-camera diagnostics sit on the 3D topic page.