Detect a frozen camera stream: alarm when the last frame goes stale

A tile that shows the same picture for a minute is not showing you a camera — it is showing you the last frame your viewer managed to decode. The link may be gone, the camera may be off, or the stream may be stalled, and the tile looks identical in every case. Most camera display widgets expose no connection-status tag you can alarm on, so the dashboard gives no warning at all. Here you will build the missing alarm yourself: watch the frame counter and arrival time in your acquisition code, raise a staleness alarm after three frame intervals, and separate a dead camera from a merely degraded one.

The symptom: a tile that freezes instead of failing

The picture stops updating. No error icon, no blank tile, no log entry — the viewer keeps its last decoded bitmap on screen for as long as the dashboard runs. Power the camera down, pull the cable, kill the client process or starve the link, and the pixel-level result is the same.

Controls tags misbehave the same way elsewhere: a tag faults to its last value when its PLC node goes offline, which is why a frozen value is a classic silent failure in SCADA as well. The cure is the same pattern in both worlds: measure freshness where the data arrives, and alarm on staleness instead of trusting the display.

Likely causes, ranked

Every cause below ends in the same frozen tile, so diagnose at the receiver, not at the screen. Work down the list; the first two cover most cases.

1. Link or power loss. Cable, connector, switch port or PoE budget. No frames arrive at all. Test: check the port's link state and PoE draw, then ping the camera.
2. The acquisition client died or restarted. A crash, an update or a container restart takes the grab loop down while the tile keeps its bitmap. Test: is the grab process running, and is its frame counter advancing?
3. Sender-side stall after packet loss. Over a TCP transport, packet loss with no reconnect can stall the session at some frame; the connection looks open and nothing moves. Test: compare frames received per second against the configured rate.
4. Camera-side hang. The socket stays open, keepalives pass, frames stop. Rarer, and nastier: reconnect logic never fires because the connection is technically fine. Test: force a reconnect and see whether frames return.
5. It is not actually frozen. A degraded link still delivers frames, just late, so the tile updates slowly and reads as stuck. Test: any increase in the frame counter puts you here, not in the dead case.

The fix: watch the frames, not the picture

The display cannot tell you anything; the receive loop can. Keep two numbers in your acquisition code, updated for every delivered frame:

• frame_counter (int, starts at 0): add 1 for every frame the SDK, GenICam client or RTSP library hands you.
• last_frame_ms (float): the wall-clock arrival time at the receiver, in milliseconds, of that frame.

Alarm when the gap between now and the last arrival passes the threshold:

Three intervals is the working margin. One absorbs ordinary jitter and a single dropped frame, the second confirms the gap is real, and the third keeps you clear of false alarms at moderate jitter while still alarming in under half a second at typical rates.

# Illustrative watchdog: variable names are ours. Feed it the counter
# and arrival time your SDK or RTSP client already gives you, and poll
# once per loop.
frame_counter: int = 0        # +1 per delivered frame
last_frame_ms: float = 0.0    # receiver arrival time, ms
threshold_ms: float = 300.0   # 3 x 100 ms at a 10 fps stream

def is_stale(now_ms: float) -> bool:
    return (now_ms - last_frame_ms) > threshold_ms

The test that proves it: start the stream, let frame_counter advance, then stop the camera's stream without stopping your program. is_stale() flips to true within 300 ms of the last delivered frame, and back to false on the first new one.

Two details decide whether this works in practice. If your grab call blocks waiting for a frame, give it a wait timeout at or below the threshold — one frame interval works — so the loop wakes and checks in time; the one-wait-per-frame pattern is in frame didn't arrive within timeout: one wait call per frame. And prefer receiver arrival time over a timestamp embedded in the stream: a camera-side clock can freeze or jump and fool a timestamp check, as camera image timestamps frozen or jumping: fix the clock source explains.

Worked example: a 10 fps tile on a 300 ms watchdog

Inputs, and where each comes from:

• Stream rate: 10 fps — the dashboard tile's configured stream setting; your case is an IP camera URL decoded into a dashboard.
• Rule: alarm after 3 × frame interval, the margin above.
• Grab-call wait timeout: one frame interval, so the loop re-checks staleness inside the window.

The arithmetic:

• interval_ms = 1000 / 10 = 100 ms
• threshold_ms = 3 × 100 ms = 300 ms
• worst-case alarm delay = 300 ms + one 100 ms wait period = 400 ms

Result: the staleness alarm sets 300-400 ms after the last frame. The test: log frame_counter every loop, cut the camera's network at t = 0, and check the log — the last frame lands at t, is_stale() is true by t + 400 ms, and the bit clears on the first frame after the link comes back. Leave it soaking for 24 hours at the production rate: 10 fps is 10 × 60 × 60 × 24 = 86,400 expected frames per day, and the pass criterion is zero false alarms.

Decision: a 10 fps tile gets a 300 ms dead-stream alarm and a 100 ms grab timeout. Scale both from the interval, never from feel; the same arithmetic at 30 fps gives 33 ms and 100 ms.

Frozen or degraded: two alarms plus a reachability check

The dead alarm is the watchdog above: no frames for three intervals. Add two more signals and the frozen tile stops hiding anything.

Degraded warning. Count delivered frames over a rolling one-second window. When the measured rate falls below half the configured rate for three consecutive windows, raise a softer warning: the tile still updates, just half as fast or worse, and the link is worth a look before it dies.

Reachability check. When the dead alarm fires, re-resolve the camera URL and attempt a fresh connection. Resolve or connect fails, and the alarm reads "camera unreachable" — the URL is no longer resolving at all. The socket opens but no frames come, and it reads "stream stalled" — session alive, sender silent. This is the distinction the frozen tile could never give you on its own.

Content check, optional. Hash a downsampled copy of each frame. Thirty identical hashes in a row while frame_counter advances means the sender is repeating one image or the scene is static. Treat it as a hint unless the scene always moves; it is a low-confidence signal by design.

USB cameras get the same treatment with the same counter: the stale-frames pattern for a directly attached USB3 camera is in detect a USB camera disconnected during acquisition from stale frames.

Wiring it to the dashboard: a heartbeat the HMI can watch

The watchdog lives in acquisition code, so publish its state. Write a heartbeat — the current frame_counter and a staleness boolean — to your tag server or PLC register once per second, and alarm the boolean like any other tag.

Then give the heartbeat its own, slower watchdog: if the heartbeat itself sits unchanged for 5 s, the acquisition process is gone and the alarm becomes "acquisition offline". Skip that second watchdog and a crashed grab program freezes its own alarm bit at the last value — stale-data alarms on PLC tags cover this exact frozen-value failure in the SCADA world.

Hardware that hosts the watchdog

The watchdog is a few lines of code; hosting it is the real requirement. You want a machine that runs around the clock unattended, one network port per camera group so a single switch fault cannot take every stream down at once, and CPU headroom to decode every stream while still checking staleness. GigE and 10GigE cameras from the area scan cameras range connect directly to the vision controllers category — vision computers with multiple LAN ports and built-in light-control channels.

These three controllers from that range fit the watchdog role:

Where the cameras run on PoE, the switch port's power reading is your first diagnostic — power collapse and re-read is a classic cause of this exact freeze, and the watchdog-and-recovery pattern is in PoE camera stream drops: watchdog timeouts and device recovery. One controller in that row carries PoE camera ports:

For a new cell, the configurator sequences camera, lens, light and controller in one pass. If you also need a display for the dashboard wall, our sister range of industrial panel PCs covers that side.

Checks before you ship

Run these against the production stream, not a bench mock-up:

1. Cut the camera's network at t = 0. The alarm sets by t + 400 ms on the 10 fps example and upgrades to "camera unreachable" after the failed reconnect.
2. Kill the acquisition process. The heartbeat alarm reads "acquisition offline" within 5 s.
3. Cap the camera's switch port at half the stream's bitrate. The degraded warning fires; the dead alarm stays silent.
4. Restore the link. The bit clears on the first new frame and the log shows exactly one reconnect attempt.
5. Soak for 24 hours at the production rate. 86,400 expected frames on the 10 fps example, zero false alarms allowed.
6. Confirm the clock source if you alarm on embedded timestamps rather than arrival time.
7. Handle partial frames. Discard, requeue and grab a new one — incomplete frames stay incomplete: discard, requeue, grab a new one — and do not increment frame_counter for a frame you threw away.

When the frozen tile is not the network

Three cases trip up a correct watchdog:

• Receiver too slow. Decode backpressure drops frames and the counter stalls while the link is fine. Check CPU on the controller before blaming the cable.
• Display-side freeze. An MJPEG client that errored mid-frame can hold a stale image even while frames keep flowing; the fix is closing and reopening the client socket, as MJPEG stream restart: close the client socket and bust the cache walks through.
• Snapshot tiles. A dashboard that cycles IP camera snapshots instead of running a live stream freezes for a different reason — the cycle stopped. Pair the watchdog with the switch-on-alarm pattern in cycle IP camera snapshot views in a dashboard and switch on alarm.

Fix the rate, compute the interval and threshold, name the three alarms, then pick the box — in that order. More stream-health patterns live on the systems topic page, and if you want the cell design checked over, the contact page reaches the applications team.