Smart camera fatal error after days running: network failure recovery

An unattended smart camera that shows a fatal application error after a weekend — web page dead, PLC flashing an I/O-not-responding flag, everything healthy again after a power cycle — is one of the most common failure reports on smart cameras. The hardware is almost never at fault. In most cases the application running on the camera made a network call to something that stopped answering — an FTP upload to a server whose trial licence expired is the classic — or it grew a buffer until memory ran out. This guide ranks the three causes, gives the test that separates them, and sizes the timeouts and the PLC watchdog relay that make recovery automatic.

The symptom pattern and what it tells you

The signature is consistent: the camera's web interface shows a fatal-error banner, the PLC reports the I/O link down, and a power cycle restores everything until the next time. The timescale is the tell — hours to days of clean running, then death, often over a weekend when other machines on the network were also off.

That pattern clears on a power cycle and returns, which rules out the sensor, the lens and the illumination: optics do not heal themselves. What a restart fixes is software state. The page an operator watches (the vision sensor browser display for local operator viewing) is served by the same application that runs the inspection, so when the application dies, the page dies with it.

Two mechanisms produce exactly this signature, and a third looks identical from the PLC's side. Work them in this order. A camera that carries the whole application on the device makes the failure visible in one banner:

The MV-SC5060XM-08M-WBN is typical of the class: inspection, PLC link and web server run as one process tree, so a single hung call takes all of it down.

Cause 1: a blocking call to a server that is down

Start here, because it is the most common and the easiest to prove. The camera saves every failed image by FTP to a PC-side server. The server's trial licence expires, or the PC sleeps over the weekend, and the next upload opens a connection that never completes. With no timeout in the call, the vision loop blocks on a dead socket; supervision eventually declares the application fatal, and the web server goes with it.

The test. Point the FTP step at an address that cannot answer, or pull the network cable mid-upload. If the camera hangs within one save cycle and shows the same banner, you have your cause. Also check the history: if every lock-up coincides with the server or the PC being down, that is your confirmation without touching code.

The fix. Put a timeout on every outbound connection the application makes — FTP, database, PLC tag link, none exempt. Values that work in practice: a 5 s connect timeout, a 10 s data timeout, three attempts with 5 s between them.

Twenty-five seconds is the longest a dead server can hold your inspection. Keep that figure under the PLC's own I/O fault timer so the controller hears about the failure from the camera, not from silence.

Cause 2: memory that only grows

Second most common: an array that gains one element per image, a queue that is never flushed, references opened and never closed. The application runs cleanly for exactly as long as the free memory lasts, then dies — and a weekend is long enough.

The test. Smart-camera management software publishes a system-state view that includes free memory. Log it once an hour while the application runs and plot it: a healthy application holds flat, a leaking one falls on a straight line, and that line predicts the crash.

Worked example with round numbers: the camera reports 400 MB free at startup, you hold a 40 MB reserve, and the log shows a 6 MB/h slope. (400 − 40) / 6 = 60 h. Sixty hours is Friday evening to Monday morning — precisely the reported pattern, and a good sanity check that your measured slope explains your observed uptime.

The fix. Bound every buffer: fixed length, overwrite the oldest entry. Close each reference, queue and file in the same iteration that opened it. If the platform offers a scheduled application restart, a nightly one caps any residual leak at one day of growth.

Cause 3: a wait for input that never comes

Less common but just as fatal: a state machine with an operator step — confirm, acknowledge, clear — reachable on the automatic path. Unattended, that wait never ends, and to the PLC it looks identical to a crash. The test is to trace the state diagram and look for any wait without a timeout; the fix is to give every wait a timeout and a default transition, usually log-and-continue or log-and-safe-stop. An alarm that summons an operator treats the symptom; the timeout removes it.

The multicast traffic is not your problem

Cameras that conform to GigE Vision announce themselves over UDP multicast as part of device discovery, and on most devices you cannot switch that off — nor do you need to. The frames are periodic and small next to any image stream, and they do not crash applications. Runtime traffic is a different matter: make it TCP, point-to-point, from the camera to the PLC and to the single image client. Decide once which side connects to which — TCP client or listener: match your barcode reader to your host walks the same decision for code readers. If discovery frames reaching every port bother you, enable IGMP snooping on a managed switch and they stop. For the wider toolkit of connection problems, start with vision camera network connection troubleshooting.

Automatic recovery: heartbeat, relay, timing budget

Even a well-behaved application deserves a way back without a person. The design that works is a heartbeat plus a relay on the camera's power.

The camera publishes a heartbeat — a pass/fail tag or a dedicated alive bit — at 1 Hz while healthy. The PLC timer resets on every beat; at 30 s with no beat it declares the camera dead and acts: command a shutdown if the camera still listens, wait 5 s, then open the relay for 10 s so the input capacitors fully discharge. Close it and allow 60 s for boot.

One hundred seconds start to finish — budget two minutes of lost inspection and size your part buffer accordingly. After the reboot, gate the PLC on the camera's ready signal rather than on power present: smart camera misses first images after load: fix the ready gate explains why the first frames after a restart need that gate. The same watchdog discipline applies upstream of the power stage — PoE camera stream drops: watchdog timeouts and device recovery covers the camera-side half.

The relay coil can sit on a spare PLC output. If the outputs are full, a channel on a light controller does the same job: the PLC commands the controller, and the channel output drops the coil.

Two 4-channel constant-voltage digital controllers fit this job:

Values as printed in each product's data sheet; open a part for its drawing and full specifications.

If your PLC has no spare input for a heartbeat, a scheduled cut is the fallback — a nightly or weekly window on the relay. A time-of-week timer on a compact smart relay is one way to build it. Prefer the heartbeat: a schedule power-cycles a healthy camera for nothing and a sick one at the wrong moment.

Keeping failed images through the outage

The weekend outage should cost you no history. Write failed images to a fixed-length ring buffer on the camera's local storage and drain it to the server when it answers; the buffer survives reboots, and the uploader resumes where it stopped. To understand what lives on the camera and what only exists in your project file, read smart camera program storage and offline editing explained. If unattended stations are becoming the norm, move the supervision up a level: a vision controller can own the retries, the buffer and the relay logic, and its switching outputs can drive the watchdog coil.

The MV-VC3301H-128G66 is a compact example, with 4 built-in light-output channels for exactly this kind of switching and a bundled vision software license.

Checks before you leave it alone for a week

Run each of these once, on purpose, before commissioning:

• Kill the FTP server and confirm the camera flags the failure within 25 s and keeps inspecting.
• Log free memory for 48 h and confirm the plot is flat, not sloped.
• Stop the trigger for 35 s and time the recovery end to end; it must complete inside 120 s with no operator action.
• After the automatic power cycle, confirm the web client reconnects on its own, with no refresh.
• Fill the ring buffer to capacity once and confirm it overwrites instead of faulting.
• Disable the watchdog logic before any firmware update or program download, and re-enable it once the camera runs.

When the power cycle stops fixing it

If the error returns within minutes of every reboot, or the camera does not come back at all, stop blaming software. Measure the 24 V rail under full load — lights and camera together — looking for sag. Check the enclosure temperature against the camera's rating; a camera at its thermal limit reboots itself and looks hung. Read the switch port counters for resets. Once those three are clean, the two software causes above are again the prime suspects, and the logging you built for the memory test will name the culprit.

If you are rebuilding the station around this failure, the configurator below walks controller, lights and trigger in one pass, and you can bring the failure log to us through the contact page — expert support comes with every order.