Monitor an IP camera stream's health and reconnect automatically

An IP camera tile keeps showing the last frame long after the stream behind it has died, while the connection status still reads connected. The operator ends up watching a frozen picture of a line that has moved on. This guide gives you the monitor that catches it: a stall threshold derived from your frame rate, a frame-freshness check that runs from any timer, and a reconnect policy with backoff that recovers the stream without hammering the camera. The numbers are concrete — a 300 ms stall threshold on a 10 fps stream, 400 ms worst-case alarm latency, a retry backoff that caps at 60 s — and each comes from one line of arithmetic you can redo for your own frame rate.

How do you tell a stalled IP camera stream from a healthy one?

A stream is healthy while its newest decoded frame keeps changing, and stalled the moment it does not: on a 10 fps stream that means a fresh frame at least every 300 ms — three nominal 100 ms intervals — and the socket state plays no part in the judgement.

The failure that bites is not a dropped connection. It is a session that stays open while frames stop: the client reports a live socket, the camera's web page still loads, and the tile repeats one image. Component state makes it worse — many video components hold an internal state such as RUNNING until an exception surfaces, and some failures, an HTTP 400 after an abrupt close being the classic case, leave the stream stalled without any event you can catch. Judge health from the product, not the transport: timestamp every decoded frame and call the stream stalled when that timestamp stops moving.

One trap hides in pipelines that fetch 'the most recent frame': the getter can keep returning the same buffered image with a success status while the producer behind it is dead. That trap, and draining it, is the subject of latest image only returns duplicate frames; the alarm pattern itself — raise one when the last frame goes stale — is laid out in detect a frozen camera stream.

How do you set the stall threshold from your frame rate?

Set the stall threshold at three frame intervals, 3 × (1000 ÷ fps) milliseconds: 300 ms for a 10 fps stream, 600 ms at 5 fps and 100 ms at 30 fps.

Why three intervals and not one? A single missed frame is routine: an exposure spike, an encoder hiccup or a busy switch drops one. Two in a row is suspicious. Three consecutive missed intervals mean the producer has stopped, and waiting longer only delays recovery. The multiplier's cost is 200 ms of extra detection delay at 10 fps, and it buys immunity to ordinary jitter.

Worked example at 10 fps: interval = 1000 ÷ 10 = 100 ms, threshold = 3 × 100 = 300 ms. Frames land at 0.000, 0.100 and 0.200 s, then nothing. At t = 0.500 s the age is (0.500 − 0.200) × 1000 = 300.0 ms — healthy, on the boundary. At t = 0.501 s the age is 301.0 ms — stalled, and the reconnect fires. With a 100 ms poll the alarm is raised by t = 0.6 s at the latest.

What do you need in place before you write the watchdog?

You need four things in place: frame-level access with a timestamp, the slowest frame rate measured rather than assumed, a documented close-and-reopen call on the client, and hardware that stays powered to run the check.

• Frame access. A decoded-frame callback, or any per-frame arrival timestamp you can read from the decoder. The check needs the frames themselves, not the client's opinion of the connection.
• A measured frame rate. Log inter-frame gaps for 60 s at your maximum exposure and use the slowest median gap. A configured fps property can report anything; the guide on reading frame rate when the property returns zero shows why measuring beats asking.
• A documented reconnect. The client's own close-and-reopen call. Reaching into a component's private objects through reflection works until the vendor ships an update and the field moves — and then your watchdog is what breaks the line.
• A home that stays up. A fanless controller or gateway that runs the poll loop even when office PCs reboot.

Any GigE smart camera that serves an HTTP or RTSP stream can be the producer; the smart cameras category and the camera finder filter by interface and resolution. One example of the class:

The watchdog belongs on the receiving machine: a vision controller that stays powered while PCs come and go. Keep camera traffic on its own NIC and subnet; several LAN ports let camera streams, the office network and the PLC link stay apart. One example of the class:

Many HMI video components expose no health attribute at all, and a guide to displaying an IP camera video stream on an HMI terminal shows the level of control you typically get — which is exactly why the watchdog watches frames instead of component properties.

How do you build the frame-freshness watchdog step by step?

Poll a monotonic clock every 100 ms, update a last-frame timestamp from every decoded frame, and treat a gap over 300 ms as the stall signal — six lines of state in total.

1. Measure the true interval. Log arrival gaps for 60 s at maximum exposure. Use the slowest median gap as the interval; if gaps oscillate between 100 and 180 ms, plan on 180 ms.
2. Compute the threshold once. stall_ms = 3 × 1000 ÷ fps_min. For the 10 fps example that is 300.0 ms.
3. Stamp every frame. In the decoded-frame callback, set last_frame_ts = monotonic(). Initialise it to 0.0, meaning no frame yet.
4. Poll every 100 ms. age_ms = (now − last_frame_ts) × 1000, and stalled = age_ms > stall_ms. Keep the poll period at a tenth of the threshold or less so checking adds almost no latency.
5. Act on the rising edge. Fire the reconnect once, when stalled flips from false to true — not on every tick it stays true.
6. Hold the alarm and keep count. Publish stalled to the HMI or PLC until frames resume, and count stalls and reconnects per stream.

# Illustrative watchdog core; variable names are this guide's own, not an API.
last_frame_ts = 0.0                  # float, monotonic seconds; 0.0 = no frame yet
fps_min       = 10.0                 # float, slowest rate measured at max exposure
stall_ms      = 3 * 1000 / fps_min   # float, 300.0 ms
poll_ms       = 100                  # int, timer period in ms

# Test with hand-fed frames at 0.000, 0.100, 0.200 s and nothing after:
#   t = 0.500 s -> age = (0.500 - 0.200) * 1000 = 300.0 ms -> stalled = False (300 is not > 300)
#   t = 0.501 s -> age = 301.0 ms                          -> stalled = True
# Worst-case alarm delay = stall_ms + poll_ms = 300 + 100 = 400 ms at these settings.

When should the watchdog reconnect, and what if reconnect fails?

Reconnect once, immediately, on the first confirmed stall; if frames are not back within 1 s, retry on a backoff of 1, 2, 4, 8, 16 and 32 s that caps at 60 s, holding the stall alarm up the whole time.

Close before you open. A reconnect that stacks a second session on a half-dead first one is how a healthy server starts answering HTTP 400: the old socket still holds the encoder or the port. Tear the session down completely — socket, buffers, decoder cache — then open fresh; MJPEG stream restart walks through exactly that sequence. If the camera dropped into a low-power or standby state while the link was down, wake it before expecting frames; camera stuck in standby after reconnect covers that case.

Give the camera 1 s after a clean reopen to deliver its first frame before calling the attempt failed — first-frame latency after session setup routinely runs a few hundred milliseconds. During backoff the stall alarm stays raised and retries keep counting; a backoff is not health. One optional belt-and-braces for links with a history of silent stalls: a full reconnect every 300 s during healthy operation, costing a 1 s picture gap, clears state you cannot see. Run it only where a one-second video gap is acceptable.

How do you verify the watchdog catches real stream failures?

Inject three faults and time the alarm: pull the camera's Ethernet (expect the stall flag within 400 ms at 10 fps), drop its packets with a firewall rule (same 400 ms, link still up), and point the URL at a closed port (one stall cycle, then steady backoff).

• Fault 1 — link gone. Unplug the camera's patch lead. Frames stop, the flag flips within 300 ms of the last frame, the alarm lands within 400 ms worst case, retries follow at 1, 2, 4 s. Pass if the HMI flag matches within one poll tick.
• Fault 2 — silent stall. Add a client-side firewall rule that drops the camera's packets without refusing them. The NIC stays up and the socket looks alive; only frame freshness catches this, which is the whole reason the watchdog watches frames instead of link state.
• Fault 3 — server refuses. Point the stream URL at a closed port. The client errors immediately and frames never start; expect one stall cycle and then steady backoff, never a hot reconnect loop.
• False-positive soak. Run 30 min at maximum exposure with the line idle. Zero stalls expected; one false stall means your threshold came from the nameplate rate instead of the measured one.

Publish last-frame age, stall count and reconnect count per stream next to the video itself; live video and metrics to a custom dashboard shows one pipeline for that. A week of counts tells you whether 300 ms fits your line or needs the exposure correction from the threshold section.

What breaks an IP camera stream in the field?

Field failures rank in this order: camera reboot or power blink, an overloaded stream server that slows below your threshold, a half-closed session that answers HTTP 400, a camera sitting in standby, then plain bandwidth starvation.

1. Camera reboot or power blink. A PoE switch hiccup restarts the camera and the stream ends cleanly; reconnect handles it. Most common cause, and the reason the backoff exists.
2. Overloaded stream server. A server feeding many clients slows below your threshold, so frames arrive late rather than never. Fix at the source — serve stills at a lower rate or split cameras across links — before touching the threshold.
3. Half-closed session. The abrupt-close class of failure: the server rejects the next request with HTTP 400 and the stream sits stalled. The full close-and-reopen from the reconnect section is the cure; the 300 s periodic reset is the bandage.
4. Standby states. The camera answers ping but sends no frames until woken; see the standby guide above.
5. Bandwidth. Bitrate scales with pixels and frame rate: a 6 MP stream carries about 3.75 times the data of a 1.6 MP one at the same fps, because 6 ÷ 1.6 = 3.75. Keep camera traffic on its own ports and subnet instead of sharing the office LAN.

The systems topic page collects the rest of the transport-level failures — heartbeat timeouts, buffer queues, stale buffers — that reach you as stalls on the client's side.

When is automatic reconnect the wrong answer?

Auto-reconnect is right for operator tiles, recording and non-interlocked guidance, and wrong anywhere a machine actuates on the video: the watchdog restores a convenience feed, never a safety function.

Two neighbours of this problem need different treatment. If the robot acts on results computed from the stream, a fresh trigger and result per cycle matters more than a self-healing tile — vision result stale every cycle is the failure that pattern prevents. If the camera speaks GigE Vision rather than MJPEG over HTTP, a transport-level keepalive governs instead: why the GigE Vision heartbeat timeout stops your stream explains why a paused application kills the stream and what to set.

Decision per case: an operator tile or recorder gets the freshness watchdog, immediate reconnect, backoff to 60 s and an alarm to the HMI. A guidance stream gets a per-cycle trigger with the freshness check as a sanity gate, and no silent reconnects between cycles. Anything a person's safety leans on gets no video dependence at all.

One decision is left: where the watchdog lives. Put it on hardware that stays powered and has a port to spare, and contact the applications team if your streams cross VLANs or a managed switch you cannot touch.