Camera module code crashes software: isolate vision code from hardware

A vision program that runs for days and starts freezing the moment you edit your inspection module is telling you where the fault lives. The symptoms look like camera trouble: the image pane goes black, the window stops responding within seconds, and no amount of re-plugging changes anything. More often than not the camera is fine — a device that still enumerates and streams in a generic viewer is not the component that died. This guide is for engineers writing custom processing modules on a smart camera or a PC host. In about an hour you will clear or condemn the hardware, pin the crash to a frame number, and size a watchdog from your real frame budget instead of a guess.

The symptom pattern that points at code

Three observations together come close to a diagnosis on their own.

• The freeze appears only after you edit the module, and it reappears at the same stage of processing every time.
• The camera enumerates normally and its status LED stays in the usual run state, which means the device is alive, not wedged.
• Restarting the software without a power cycle reconnects the camera cleanly, so the transport was never dead.

When those three hold, the crash lives in the module code and your edits brought it there. Commit to that reading and debug the code; do not start ordering cameras. The mirror case — no image in any application at all — follows the hardware path in camera shows no image after a short: isolate the fault safely.

Likely causes, in order of probability

Work down this list; each test takes minutes.

1. Unhandled exception in the edited module. A blob touching the image border, a zero-size region after a failed threshold, a division by a width that measured zero. Test: log every exception with its frame number. A code fault repeats on the same frame content.
2. Blocking call on the window's thread. A synchronous grab with no timeout stops the window from processing messages, and the application soon reads as not responding. Test: does the log stop writing at the same second the window freezes?
3. Per-frame overrun. Processing that takes longer than the frame period backs up the queue until the pane goes black. Test: timestamp each grab and each result; a growing gap is the signature. The fixes in acquisition faster than processing: fix the frame rate gap apply directly.
4. Stale buffers. The transport hands you old frames while fresh ones pile up, and processing falls further behind every loop. Test: print the frame IDs you receive; a falling or repeating ID means the buffer policy needs newest-only handling on the transport layer.
5. Resource leak. Memory or handles that grow per frame freeze the host after minutes, with timing that looks random. Test: plot memory over a 30-minute soak; the inspection loop memory leak guide shows what a healthy curve looks like.
6. Genuine camera fault. Least likely when enumeration and the LED are normal, and quickest to clear with the viewer test below.

The five-minute rig that settles it

You need three things: a generic viewer for the camera's transport (a GigE Vision or USB3 Vision viewer, or the operating system's camera service for a UVC device), a second host, and a timer.

Stop your own application so nothing else holds the device. Open the viewer, set exposure and gain to fixed manual values, and stream at your production frame rate and resolution for ten minutes.

• A stable stream for ten minutes clears the camera, its cable, its power supply and the driver. Your module is the fault; go straight to the exception handler below.
• No enumeration, a stream that drops, or corrupted frames moves the suspicion to hardware. Swap the cable first, then retest on the second host before you condemn the camera.

The second host does not need to be elaborate. A vision controller such as the MV-VC3301H-128G66 carries five or more LAN ports, four built-in light-output channels and 8 GB of memory, so the box that runs production can also serve as the known-good bench host: move the camera to a spare port, stream, and you have separated the device from the host operating system in one swap.

An alternative removes the host from the equation entirely. The MV-SC5060XM-08M-WBN is a 6 MP monochrome smart camera with an M12 mount and a GigE interface whose inspection runs on the device itself, so a crash in a PC-side processing loop can no longer stall the line.

Test 2: replay frames with no camera attached

Recording turns a suspicion into a proof. Capture 200 frames from a healthy stream — include the frames that trigger the fault, such as parts touching the edge of the search region — then detach the camera completely and run the module against the files for 30 minutes.

• The freeze reproduces with no device attached: the camera is cleared beyond argument, and you can debug on a bench with no hardware at all. Record-and-replay fixtures show how to structure the harness.
• Thirty minutes of replay never hangs: put the camera back, run live with the watchdog from the next section, and watch the delivery side. The fault sits between the transport and your loop.

Keep the camera behind one thin acquisition interface so the replay harness and the live loop call identical processing code. It is the same idea as the abstraction layers that isolate motion hardware used in control engineering.

Fix the code: handle every frame's failure

A loop that cannot fail cleanly will eventually freeze a window. Wrap the per-frame work so that any single bad frame logs, recovers and continues:

import time

frame_no: int = 0        # frames processed this shift
fail_streak: int = 0     # consecutive delivery failures
watchdog_ms: int = 50    # three frame periods at 60 fps
MAX_STREAK: int = 5      # restarts before a fail-safe stop

while running:
    try:
        frame = grab_frame(timeout_ms=watchdog_ms)
        if frame is None:
            raise TimeoutError(f'no frame within {watchdog_ms} ms')
        result = inspect_module(frame)   # your edited code lives here
        publish_result(result)
        frame_no += 1
        fail_streak = 0
    except TimeoutError:
        fail_streak += 1
        log.warning('frame %d: stream stalled, restart %d of %d',
                    frame_no, fail_streak, MAX_STREAK)
        restart_stream()
        if fail_streak >= MAX_STREAK:
            stop_line_safely()           # never loop on a dead stream
    except Exception as err:
        frame_no += 1
        log.error('frame %d: module raised %r', frame_no, err)
        publish_result(REJECT)           # a code fault must not pass a part

grab_frame, inspect_module, publish_result, restart_stream and stop_line_safely are example names for your own functions; the frame counter, the failure streak and the 50 ms timeout are the parts worth copying. Prove the handler works before you trust it: inject one truncated frame into your replay set and confirm the log records that frame number while the loop continues to the next.

Two details decide whether this survives production. Log the frame number with every exception, because a line like frame 4213 turns a crash into a locator you can replay. And if the capture library is native C or C++ on Windows, a fault inside it can kill the whole process instead of raising a normal exception; a frame-based exception handler catches it so the process lives long enough to write the log.

Size the watchdog from the frame budget

Watchdog timeouts guessed as a few seconds are too slow to save the window and too vague to diagnose anything. Compute them from the loop's real budget. Take a 60 fps loop with a 640 × 480, 8-bit monochrome window:

• Frame period: 1000 ms ÷ 60 fps = 16.7 ms.
• Transfer: 640 × 480 = 307,200 bytes ≈ 0.31 MB per frame, so 0.31 MB × 60 = 18.4 MB/s, well inside GigE's roughly 110 MB/s of practical throughput; 0.31 MB ÷ 110 MB/s ≈ 2.8 ms.
• Exposure: 2 ms, fixed.
• Your module, measured on the replay fixtures: 8 ms.
• Total: 2 + 2.8 + 8 = 12.8 ms, leaving 16.7 − 12.8 = 3.9 ms of margin per frame.

Set the watchdog at three frame periods: 3 × 16.7 ms = 50.1 ms, so 50 ms in the code. Two consecutive late frames cost 33.4 ms and pass; the third trips a stream restart in well under a second — instead of the roughly 5 seconds of unprocessed messages after which Windows declares a window hung.

The same arithmetic explains the black-pane-after-a-few-seconds pattern. Let the module creep from 8 ms to 14 ms: 2 + 2.8 + 14 = 18.8 ms per frame, which is 18.8 − 16.7 = 2.1 ms past the period. At 60 fps the backlog grows 2.1 ms every frame, so after 1,000 frames — about 17 seconds — the queue holds 2.1 seconds of stale frames and the window stops drawing. If the measured module time will not fit the period, recover the milliseconds in slow Python OpenCV code: fix per-pixel loops that kill FPS before touching hardware.

Prevention: make the next fault obvious

Four habits keep a code fault from ever wearing a camera costume again.

• Run inspection in its own process. A module crash then takes down one worker, not the window; an operating-system watchdog restarts it and the exit code lands in the same log.
• Keep the teardown symmetrical with the start-up. Half-released handles and libraries left loaded after a program stop produce hangs that look like hardware; the program stop teardown fault guide walks that checklist.
• Handle reconnection states deliberately. A camera that returns from a reconnect in standby delivers nothing and looks exactly like a hung loop; wake it before you stream.
• Soak every release. Thirty minutes of replay plus thirty minutes live, with memory and per-frame timing plotted. Flat lines pass; a sawtooth that only climbs is a leak.

When it is the hardware after all

The viewer test convicts the camera only when the viewer fails on its own: no enumeration on either host, a stream that drops at random, corrupted frames, or a status LED that never reaches its normal run state. Move the fault with the parts. The same camera failing on the second host convicts the camera; the same camera streaming cleanly there convicts the first host's drivers, power or cabling. Reconnect storms and cable suspects follow USB camera keeps disconnecting: isolate cable and hardware faults. Only then is it time to shop: start in the smart cameras category for an on-device replacement, the vision controllers category for a multi-camera host, and use the camera finder to filter by interface, resolution and shutter.

Checks before you order parts

• Ten-minute generic viewer soak at production frame rate, exposure and resolution — clears device, cable, power supply and driver.
• Replay run: 200 recorded frames, 30 minutes, no camera attached — proves or clears the code.
• Exception log with frame numbers from at least one hang — the same frame repeating is code; random frames with stream stalls is delivery.
• Memory and per-frame timing plots from the soak — flat passes, a climbing sawtooth is a leak.
• Watchdog at 50 ms (three frame periods at 60 fps), proven by pulling the network cable for 2 seconds: the loop logs, restarts and continues.
• A second host, or a spare LAN port on the production controller, for the swap test.

The decision, stated once: a freeze that follows edits plus a clean ten-minute viewer stream — fix the module and order nothing. A viewer that fails on two hosts — isolate cable, power supply and camera in that order, and replace hardware only when a known-good host cannot stream the device. While the line is down, build the replacement camera side around your existing lens and lighting in the configurator: