RTSP IP camera streams in an HMI: display and PTZ control

An IP camera is the cheapest way to give an operator eyes outside the machine: one Ethernet drop, power from the switch, and a PTZ head that covers the whole cell. The catch is that most HMI runtimes ship no RTSP widget, so teams fall back to browser windows and spawned helper processes. Neither is needed. Decode the RTSP IP camera stream inside your HMI process, draw it into a plain image control, and drive pan, tilt and zoom over the network with ONVIF or the vendor's HTTP calls. The steps below run on the same machine that hosts your inspection software, and every number in them is one you can measure before go-live.

What you need

• An IP camera that serves RTSP in H.264 or H.265, with PTZ through ONVIF Profile S or documented HTTP commands. Fixed cameras work for display alone.
• An HMI runtime that can draw a bitmap into an image control on a timer, or host a decode library.
• A decoder: FFmpeg as a helper process, or a.NET RTSP client used in-process.
• A machine to run it on. Vision controllers keep the HMI, the decoder and the inspection software on one fanless box; where the HMI lives on its own panel instead, an industrial panel PC plays the same part.

The shop's controllers cover the ends of that range:

The first pairs an Intel Core i3 with 8 GB of memory and 5+ LAN ports, and includes the VM6200Pro vision license. The second runs an Intel N-series processor with 8 GB, a 1 TB HDD and 3-4 LAN ports.

Get the RTSP URL and prove the feed

1. Open the camera's stream settings and copy the RTSP address exactly. The path is vendor-specific and often encodes a channel or profile number, so copy it rather than reconstruct it.
2. Paste the address into VLC and watch for a minute. If VLC stutters, fix the network first — no decoder recovers a feed the switch cannot deliver.
3. Note three numbers that size everything downstream: resolution, frame rate and codec.

If the operator only needs a still every few seconds, stop here: the snapshot URL route skips the decode work entirely.

Decode frames into an image control

Two routes work, and both end in the same place: a bitmap you draw.

• FFmpeg as a helper process. Your runtime spawns it, it holds the RTSP session and decodes, and raw frames cross a pipe for you to scale and draw.
• A.NET RTSP client in-process. RtspClientSharp hands you decoded frames directly; its SimpleRtspPlayer sample is the decode-to-bitmap reference, and a standard JPEG encoder covers the still-export case.

Then wire the draw loop:

4. Size the image control to the widget, not to the stream, and scale the frame once per draw.
5. Draw the newest frame on the display timer and drop anything older. A queue turns one frame time into a growing one.
6. Count dropped frames per minute and show the counter during commissioning. A 25 fps stream into a 30 fps timer must drop 5 frames per second somewhere — make that explicit instead of accidental.

The same grab loop is what you would reuse to decode barcodes from a live RTSP or MJPEG video stream; display is just its consumer. And if the widget is small, cut the decoder CPU load with a low-resolution preview stream instead of downscaling the main stream yourself.

When the full decode is more than you need

Three cheaper routes cover real cases. For a live view in a runtime that renders web content, restream RTSP for display in an HMI browser and let a browser engine decode. To choose between refreshing stills and a true stream, the snapshots versus streams comparison lays out the trade.

Platforms that ship a built-in camera viewer usually support a narrow codec and profile set, so check RTSP limits of a built-in IP camera viewer widget before you plan around one. Panel-class HMIs document their own route: configuring RTSP camera streams on a unified comfort panel shows the media-player add-on approach.

Size the load: a worked example

Take a 1080p25 main stream, a 640×360 substream and four cameras on one controller:

• Frame budget. 1,000 / 25 = 40 ms. Decode plus draw must finish inside it, or frames drop.
• Substream load. 640 × 360 = 230,400 pixels against 1920 × 1080 = 2,073,600 — exactly 1/9, about 11 percent of the main-stream decode work.
• Bandwidth. A 1080p25 H.264 stream at a typical 4 Mbit/s default is 4 / 8 = 0.5 MB/s. Four cameras make 16 Mbit/s: 16 / 1,000 = 1.6 percent of a 1,000 Mbit/s link, or 16 percent of a 100 Mbit/s port. Leave headroom for the inspection cameras.
• Latency floor. One frame captured (40 ms) + one frame in the camera's encoder (40 ms) + under 2 ms on the wire + one to two frames of decoder buffer (40-80 ms) + one display refresh (16 ms at 60 Hz) = about 140-180 ms. Whole seconds mean queueing.
• CPU. Software H.264 decode of 1080p25 costs roughly 10-15 ms of one core per frame — a planning figure to measure on your machine. At 12 ms, four streams take 4 × 25 × 12 = 1,200 ms of core time per second, about 1.2 cores before HMI logic. Drive the widget from substreams instead and the same math gives 1.2 / 9 ≈ 0.13 cores.

That decode headroom is what separates the controller classes:

The MV-VC5525-256G70-NN pairs an Intel Core i5 with 32 GB of memory, PoE camera ports and 8 built-in light channels. Put the RTSP cameras on their own port and subnet, and keep the PoE ports for the inspection cameras.

Move the camera over the network

PTZ has a standard: ONVIF. A camera conformant to Profile S exposes video streaming and the PTZ service together, so pan, tilt, zoom, presets and home answer the same calls whatever the badge on the housing. Discover the camera on the subnet, then drive it from an ONVIF command-line tool before any HMI work: get-presets lists the stored positions, goto-preset moves to one, and goto-home returns the head to its stored home position. Prove repeatability at the bench. When the camera hangs off a serial PTZ driver instead of the network, PTZ camera control from an HMI over RS-485 is the closer guide.

Continuous moves take a speed per axis, normally scaled 0 to 1, plus a duration. Send them while the operator holds the button, at 5-10 commands per second, and keep each burst 200-500 ms with an explicit stop on release. A lost stop leaves the head slewing; a short burst bounds the worst case to a small drift.

Cameras without ONVIF usually expose the same axes as HTTP GET commands with the values in the query string. The pattern is identical to sending commands to a smart camera over TCP in C#: open the connection, write the command, read the acknowledgement, keep one connection alive.

Digital zoom deserves its own arithmetic, because it is cropping. At 4× on a 1080p stream the widget shows (1920 / 4) × (1080 / 4) = 480 × 270 pixels — 129,600 pixels, 1/16 of the frame — upscaled 4 × 4 on screen. Only optical zoom adds detail, so store each station's optical framing in a preset and keep zoom out of the jog buttons.

Wire PTZ into the HMI

• Jog. Eight direction buttons plus zoom in and out; press starts a continuous move, release sends stop.
• Presets. Fill a dropdown from get-presets at runtime start. A preset repeats because the camera stores the coordinates — its accuracy does not depend on your command timing.
• Park. Send goto-home on shift start and after idle time, so the default view is stable for the next operator.
• Permissions. Allow PTZ in manual mode only; in automatic mode the head stays parked and the view stays deterministic.

Verify the three things that break

1. Latency. Point the camera at a stopwatch running next to the lens and compare on screen. Expect the 140-180 ms floor from the worked example; anything near a second is buffering.
2. Recovery. Pull the camera's cable for ten seconds during a run. An RTSP session does not resume by itself, so the client must detect the drop and re-open — build that loop, then test it.
3. Repeatability. Run goto-preset twenty times and watch the framing. Preset accuracy belongs to the camera; confirm it before anything downstream depends on the view.

Watch decoder core time during commissioning with every stream live, not one at a time.

Common mistakes

• Queueing frames to smooth the picture. Drop late frames instead; a queue converts the 140-180 ms floor into whole seconds.
• Previewing from the main stream. The substream does 1/9 of the decode work and a small widget cannot tell the difference.
• Long continuous-move bursts with no stop. Keep bursts 200-500 ms and stop on release.
• One flat network for inspection cameras, RTSP cameras and the plant. Give the RTSP cameras their own port and subnet.
• Treating digital zoom as detail. At 4×, the widget shows 1/16 of the pixels.
• Testing presets only inside the finished HMI. Prove them at the bench with a command-line tool first.

Once the split between controller and panel is settled, /configure/ builds the full set from the camera outward, the systems topic page collects the neighbouring wiring patterns, and /contact/ reaches the applications team when the architecture fights back.