Detection pipeline runs but no output: debug frame by frame

Your detection pipeline starts, prints a couple of start-up lines, and ends. No window, no error, no detections, while the pose-estimation example from the same collection runs fine on the same USB camera at /dev/video0. That is the classic silent exit on an embedded accelerator stack, and it has two usual causes: the runtime and model packages have drifted apart, or the model's fixed square input — commonly 640×640 pixels — is never delivered because a convert or scale element is missing or mislinked. This guide gives you the tests that separate the two in about 15 minutes, plus the letterbox arithmetic that makes the caps check unambiguous.

A clean exit is the clue, not the absence of one

On a pipeline fed by a live camera, a clean exit is a contradiction. A live source never sends end-of-stream on its own, so if the process stops by itself, reports exit code 0 and leaves the log empty, the application decided to end. Debug the application first, not the camera: the device node is present, and the same camera streams in the pose example, so enumeration, driver and USB link are all proven good. The failure lives inside the graph. Treat it the same way as the GenTL producer silent exit: the framework logs every negotiation it attempts — you only have to ask at the right level.

Why one example works and its neighbour does not

Two example scripts can share a camera and still behave differently, because they build different graphs. The pose model in these collections happens to accept what the camera branch produces. The detection and instance-segmentation models do not: they carry a fixed square input, and in front of it sit convert and scale elements whose only job is to turn whatever the camera negotiates — say 1920×1080 — into that square. Each of those elements is a link that can fail. At build time a failed link usually prints an error; the dangerous case is the branch that builds, attaches the model, and then never delivers a buffer, leaving the application's loop to end as if nothing had happened. Two hand-offs decide the outcome: runtime to model file, and element to element. The tests below check them in that order.

The causes, in order of probability

Work down this list and stop at the first test that answers.

1. Runtime and model package mismatch. The compiled models and the runtime that executes them ship as a pair. In the field case behind this guide, both failing scripts came right after the accelerator's combined runtime-and-model package was updated to the current release — both parts, one move. Test: print the installed runtime version and the model bundle version. Different releases means update both together from the vendor's official channel, then re-test before touching anything else.
2. A dead model branch. A convert or scale element left out or mislinked, or the branch starved by a missing queue. Test: the frame counter in the next section. Zero frames in 10 s means the model never saw one pixel; no amount of threshold tuning can help.
3. Caps that never reach the model input. The scaling chain negotiated something other than the fixed input: 1920×1080 arriving at a sink built for 640×640, or YUY2 where the model wants RGB. This is the same negotiation failure class as v4l2src signal lost not-negotiated: match caps to the sensor format, one stage further down. Test: read the caps string at the sink and compare it, character for character, with the model's stated input.
4. Two copies of the stack. A partial update leaves the previous runtime earlier on the search path, and the process loads it without complaint. Test: print the file path of the runtime module the process actually imported and the model file it opened. Both must point into the release you installed.
5. What it is not. Bandwidth and permissions fail loudly — dropped-frame errors, device-open errors. A 1920×1080 YUY2 stream needs about 995 Mbit/s and cannot fit a USB 2.0 port's 480 Mbit/s line rate, but that failure announces itself with errors. Do not chase it for a quiet exit.

The 15-minute debug, step by step

1. Run the failing script once with output captured to a file, then check the shell's exit status. Expect exit code 0 and a nearly empty file. A traceback or a non-zero code is a louder, easier problem — debug that first.
2. Turn the framework's debug logging on and re-run. For GStreamer-based stacks, the log-level variable GST_DEBUG=3 prints warnings and FIXME messages, caps trouble included, while staying readable. A not-negotiated line naming the elements between the camera source and the inference element ends the investigation at cause 3.
3. Attach the sink probe from the next section and run for 10 s. Read the frame counter.
4. Read the captured caps string and put it beside the model's stated input — the example models in these stacks commonly take 640×640 RGB. A mismatch is cause 3; an exact match with a zero counter is cause 1 or 2.
5. Print the runtime and model bundle versions (cause 1) and the loaded module path (cause 4). Align the two packages if they differ.
6. Re-run the probe. You want roughly 300 frames in 10 s at 30 fps — 30 × 10 = 300 — and a caps string naming the model input exactly.
7. Only now edit the graph, if the counter is still zero. Check that every element between the camera source and the sink is linked in order, that the inference branch has its own queue, and that the filter element states the model input exactly. A one-line sanity build — source, converter (videoconvert), scaler (videoscale), a capsfilter pinning 640×640, then a fakesink, run with verbose output — tells you whether the elements negotiate on their own.

The sink probe: a frame counter and a caps string

A camera can negotiate perfectly and the model can still starve, so measure delivery at the sink instead of trusting the start-up lines — the same principle as measuring the delivered frame rate when detection finds nothing. The probe counts buffers and keeps the last caps string:

# Sink probe for the model input. frame_count and last_caps are our
# own globals; new-sample and pull-sample are the sink element's
# documented signal and method names.
from gi.repository import Gst

frame_count = 0      # int: buffers delivered to this sink
last_caps = "none"   # str: caps of the most recent buffer

def on_new_sample(sink):
    global frame_count, last_caps
    sample = sink.emit("pull-sample")
    if sample is None:
        return Gst.FlowReturn.ERROR
    frame_count += 1
    caps = sample.get_caps()
    last_caps = caps.to_string() if caps else "none"
    return Gst.FlowReturn.OK

Connect it to the model input sink's new-sample signal, run for 10 s, then read the two globals. frame_count near 300 with last_caps naming 640×640 means the branch works and the model side deserves suspicion. frame_count stuck at 0 convicts the graph, whatever the start-up lines claimed.

Worked example: 1080p into a 640×640 model

Bench case: a USB camera negotiating 1920×1080 at 30 fps, feeding a detection model whose input is 640×640 RGB.

Scale to fit inside the square: 640 / 1920 = 0.3333, so the frame becomes 640 × 360, and the letterbox pads (640 − 360) / 2 = 140 black pixels onto the top and bottom. Every detection then runs on that padded square, which is why the sink must negotiate exactly 640×640 and nothing else.

The buffer arithmetic on both sides of the branch:

• Camera side: 1920 × 1080 × 2 B (YUY2) = 4,147,200 B per frame. At 30 fps: 4,147,200 × 30 = 124,416,000 B/s ≈ 124.4 MB/s ≈ 995 Mbit/s.
• Model side: 640 × 640 × 3 B (RGB) = 1,228,800 B per frame. At 30 fps: 1,228,800 × 30 = 36,864,000 B/s ≈ 36.9 MB/s ≈ 295 Mbit/s.

Decision from the numbers: a probe caps string showing 1920×1080, YUY2 or 640×360 convicts the convert-and-scale pair — causes 2 and 3. A caps string that matches the model input exactly with a zero counter, after an aligned update, convicts the runtime: stop editing the graph and fix the packages.

When updating is the wrong move

Updating is the right first test only because version drift is the most common cause; it is not a ritual to repeat. Three habits waste afternoons. Updating one package at a time straddles a release line and manufactures the symptom. Reinstalling everything on every failure leaves extra copies on the search path, and the process keeps loading the stale one — the version-pinning fix for edge inference crashes makes the same point for a louder failure: one pinned release, proven once. And editing the example's input size to match the camera treats the wrong side as the variable: the compiled model is fixed at the size it was built for, commonly 640×640, so the graph, not the camera, delivers the square.

The same debug on a line-side controller

Everything above runs unchanged on a shop-floor vision controller; the method is the same, the branch count is not. One branch per camera means a four-camera line has four sinks to probe, and three healthy preview windows can hide the fourth. Probe every sink at commissioning, log each sink's caps string once at start-up, and bench-test with the same GigE area scan cameras the line will use. Three controllers cover the usual range:

The MV-VC2140-128G40-1T is the single-camera bench unit: 8 GB memory, 2 USB 3 ports, 4 built-in light channels. The MV-VC3301H-128G66 adds an Intel Core i3 processor, 5+ LAN ports and PCIe expansion. The MV-VC5525-256G70-NN is the multi-camera step, with 32 GB memory, PoE camera ports and 8 built-in light channels. If the cell also needs an operator display, industrial panel PCs mount to the same bench and keep the dashboard and the logs on the wall.

Checklist: keep every branch observable

• Pin the runtime and the model bundle to one release number, and upgrade both in the same maintenance window.
• After every upgrade, run the 10-second probe on every sink before trusting the line.
• Log each sink's caps string at start-up and treat exact equality with the model input as the pass condition.
• Keep one Python environment per controller and print the loaded runtime path at start-up.
• Write the debug invocation — GST_DEBUG=3 — into the run notes, so the bench reproduces what the line shows.

If a branch stays dead after an aligned update and a repaired graph, send the probe output and the two version numbers through the contact page and we will read the log with you. To start from a proven base, the vision controller configurator pairs the controller, cameras and lights in one pass. The neighbouring quiet failures live on the systems topic page: a camera that shows in Device Manager but not in your SDK is the same silent class, one layer up.