IP camera images into vision software: processable pixels

An IP camera puts a picture in your browser and a JPEG on your disk, and neither is a pixel array your code can index. The stream arrives compressed, wrapped in a transport protocol and, on many cameras, reachable only through the maker's library. This guide is for engineers wiring a network camera into their own inspection software. It gives the three documented routes to the pixels, the decode and reshape steps with their traps, the frame rate each route sustains with the arithmetic shown, and the point at which a machine-vision camera is the better buy.

How do you get an IP camera image into your own code?

Three documented routes exist, and every one ends in bytes you decode yourself: the maker's SDK callback, an RTSP stream, or an HTTP JPEG snapshot and MJPEG feed. The browser picture and the saved file never contain indexable pixels on their own.

The SDK route calls the library the camera maker publishes: a documented function or callback hands your code a frame buffer. RTSP streams video over RTP, normally on TCP port 554, at a path the camera's manual lists. A media library such as FFmpeg, GStreamer or OpenCV's capture class connects, decodes and delivers frames. The HTTP routes are simpler: a snapshot URL returns one JPEG per GET, and an MJPEG endpoint sends a continuous multipart/x-mixed-replace body of boundary-delimited JPEG frames. The walkthrough on acquiring an RTSP stream from an IP camera in your vision program covers the connection side in detail.

If the picture only needs to be seen, not processed, that is a display problem with its own answers — displaying vision camera images on an HMI shows one display path. Processing needs the decoded buffer itself, which is the next distinction.

What separates a JPEG from processable pixels?

A JPEG is a compressed byte stream, not pixels: only after a decoder expands it into width × height × 3 numeric values can you index pixel (x, y) — 640 × 480 RGB expands to 921,600 bytes.

Inside a JPEG the picture exists as frequency-domain coefficients, so pixel (319, 239) is not in the file until a decoder produces it. Image bytes versus pixel data covers that boundary in general. H.264 makes it stricter: frame N is predicted from earlier frames, so no frame decodes alone without its group of pictures. The decoder needs the SPS and PPS parameter sets before the first picture appears. A saved JPEG is self-contained; an H.264 excerpt without its parameter sets is not.

How do you turn the SDK's flat buffer into an indexable image?

Reshape the flat buffer into rows × columns × 3, but only after checking three things: the row stride, the channel order and the vertical orientation — a 640-pixel RGB row needs at least 1,920 bytes of stride.

Work through these steps once per camera model:

1. Take the frame from the documented interface and note the width, height and channel count it reports.
2. If the payload is compressed, decode it: JPEG through a decoder library such as libjpeg-turbo, H.264 through a media library that also converts its planar YUV output to RGB.
3. Read the stride, also called row pitch. Rows are often padded to a 4-byte boundary, so address pixel (x, y) as y × stride + x × 3, not (y × width + x) × 3, whenever stride exceeds width × 3.
4. Reshape the flat buffer to height rows, width columns, 3 channels.
5. Confirm channel order (RGB or BGR) and vertical orientation; some libraries deliver rows bottom-up.
6. Prove it with a test target: point the camera at solid red, read the center pixel and confirm the red channel reads near 255 with the other two near 0 before any downstream math runs.

Steps 3 and 5 cause most first-week bugs, and both fail silently: the picture looks right while the numbers are wrong. The raw buffer conversion walkthrough and the pixel-format verification check cover both in code. Where your pipeline keeps frames encoded for transport and decodes at the end, the unencoded capture path shows what you save by cutting both steps.

Which route works best: vendor SDK, RTSP or snapshot polling?

Pick the vendor SDK when it documents a decoded-frame callback, RTSP with a native media library for continuous motion at the camera's stream rate, and snapshot polling when one checked image every few seconds is enough — realistically 1–10 fps.

The RTSP latency comes from client-side buffering and B-frame reordering, and both shrink with settings the library exposes. Snapshot polling costs one HTTP round trip plus a decode per image, which an interpreted scripting layer turns into single-digit frame rates; the compiled decode itself is far faster, as the next section shows. When the consumer is PLC logic rather than an operating-system process, the same decoded frames cross into the runtime over its data-access layer — retrieving camera images over ADS documents one such path. And when the job is reading codes from a live stream, decoding barcodes from an RTSP or MJPEG stream runs this exact pipeline end to end.

How many frames per second can you actually process?

Budget 13.6 fps for uncompressed 640 × 480 RGB on a 100 Mbit/s link — 12.5 MB/s ÷ 0.92 MB per frame — while the same frames as MJPEG need under 20 Mbit/s, so the decode step, not the wire, sets the ceiling.

Worked example: a 640 × 480 camera streaming MJPEG at 25 fps with a typical 10:1 compression.

• Pixels per frame: 640 × 480 × 3 = 921,600 bytes.
• On the wire: 921,600 ÷ 10 = 92,160 bytes, about 0.09 MB per frame.
• Stream rate: 92,160 × 25 = 2,304,000 bytes/s = 2.3 MB/s = 18.4 Mbit/s, which 100 Mbit/s carries with more than 5× headroom.
• Uncompressed, the same link allows 12.5 MB/s ÷ 0.9216 MB ≈ 13.6 fps, below the camera's 25 fps — the reason the camera compresses.
• Decode cost: roughly 1–3 ms per VGA JPEG on one current x86 core, against a 40 ms frame period.

A vehicle at 30 m/s (108 km/h) covers 30 ÷ 25 = 1.2 m between frames. A 25 fps patch check therefore samples its position comfortably for presence and queue flags. Two patches 5 m apart time that vehicle at 5 ÷ 30 = 0.167 s ≈ 4.2 frames; half a frame of timing quantization is ±12%, good enough for an alarm and not for enforcement. For tighter speed figures or per-vehicle capture, use the hardware-triggered route at the end of this guide.

One thing the IP camera never supplies is a per-frame capture timestamp or exposure value, so your timing starts when the frame arrives. Chunk data on camera frames shows how machine-vision cameras carry both beside the pixels, and RGB camera frame rate limits runs the same pixel-format-versus-rate arithmetic for direct camera streams.

How do you process just the pixels that matter?

Crop to your region of interest before any per-pixel math: a 200 × 200 patch moves 120,000 bytes per frame instead of 921,600 — 7.7× less through your loop — though the JPEG still decodes full-frame first.

JPEG has no partial decode: the decoder expands all 921,600 bytes no matter how small your patch is, so decode cost stays fixed and the crop happens afterwards. What the crop saves is everything downstream: copying, conversion and interpreted per-pixel work all drop by 7.7×, and 25 frames × 120,000 bytes = 3 MB/s is a light load for compiled array operations. Compute one mean red, green and blue value per patch per frame and you hold a color-change signal at a handful of operations per frame. Keep the rest of the frame out of the loop entirely — zeroing pixels outside an ROI shows the array-level way to mask without per-pixel script.

What hardware runs the decode without dropping frames?

A current vision controller decodes several VGA-rate streams without strain; choose by network ports and CPU headroom — keep 50% CPU free, plan one compact unit per two to four 25 fps VGA streams, and let PoE camera ports carry power over the same cable.

The margin follows from the decode budget: at 1–3 ms per frame one core sustains a handful of VGA streams, and the headroom absorbs conversion, copies and your own processing. Interpreted script loops, not the controller, are the usual bottleneck; where a PLC does the deciding, image processing on a PLC sets out what that class of hardware absorbs.

A compact fanless unit such as the MV-VC2140-128G40-1T carries three to four LAN ports, two USB 3 ports, four built-in light channels and 8 GB of memory on an Intel N-series processor.

For several cameras on one box, the MV-VC5525-256G70-NN adds PoE camera ports, five or more LAN ports, eight built-in light channels and 32 GB of memory on a Core i5-class processor, so one Ethernet cable per camera carries frames and power together.

Browse the vision controllers and check four things before ordering: the camera documents its RTSP path or an SDK with decoded-frame access; the session count covers every consumer; the PoE budget covers every camera; and your decode library is licensed for the way you ship it. The configurator walks ports, memory and mounting in order.

When is an IP camera the wrong camera for the job?

Change to a machine-vision camera when the job needs hardware triggering, a global shutter, raw pixel values or guaranteed timing — an IP camera's compressed, buffered stream cannot tell you the exact moment any frame was captured.

Four disqualifiers cover most cases. Triggering: an IP camera exposes no hardware trigger wired to the exposure, so you cannot freeze a moving part at a known position. Shutter: fast motion read by a rolling shutter shears the geometry, while machine-vision lines offer global shutters. Data: compressed pixels shift values, so measurement and radiometric checks fail. Timing: latency sits in the 100 ms class at best, so the answer arrives after the part has moved on. When any of those applies, the area scan cameras are the starting point and the camera finder filters by shutter, resolution and interface. The MV-CU020-90GC, a 2.5 MP GigE color camera with a 1/1.7" sensor, C-mount lens mount and a global shutter, is the class of part that answers those requirements.

If the deployment keeps its IP cameras, the pieces above — documented route, decode, reshape, ROI, rate budget — are the whole project, and the systems topic page collects the surrounding guides. Send the camera model, the rate you need and the decision the pixels drive through the contact page, and we will size the decode hardware or spec the machine-vision alternative.