Streaming camera video over UDP without broken frames

A webcam or board camera feeding a viewer program over UDP often plays like a television losing signal: bands of the picture tear, shift or freeze while the rest keeps updating. The cause is almost never the image format itself. It is datagrams lost or reassembled out of order between the sender and the receiver. This guide is for anyone writing that transport by hand, in Qt or any other toolkit. You will get the bandwidth math that shows when raw frames cannot fit the link, the datagram size and per-packet header that make reassembly possible, and the pacing, JPEG compression and one-pass conversion that turn a torn stream into a steady one.

What a torn picture actually tells you

Work the failure chain in order, because every link in it produces the same symptom on screen. UDP offers no ordering and no delivery notice: a datagram arrives whole or not at all. Anything larger than the path MTU travels as IP fragments, and losing one fragment discards the entire datagram. Standard Ethernet runs a 1500-byte MTU, so a careless 921,600-byte frame is dozens of independent chances to lose the picture.

The second failure is detection. A frame split into 100-byte pieces with no frame number and no index cannot tell a lost piece from a late one: one missing fragment silently corrupts everything after it, and the client paints the result anyway. That painted partial frame is the tear you see. Third is volume: raw 640 × 480 colour overruns both the link and the socket queues. Fourth is the client, which rebuilds pixels one call at a time and falls behind the arrival rate.

When decode plus conversion takes longer than the frame budget, the receive queue backs up and the drops return no matter how correct the sender is. The frame-rate gap has its own walkthrough in acquisition faster than processing: fix the frame rate gap, and stalled client previews in preview dropped frames: diagnose stalls and resource limits.

Worked example: a 640 × 480 24-bit stream at 15 fps on a 100 Mbps link

Inputs, each with its source:

• Frame: 640 × 480 pixels, 3 bytes per pixel (24-bit BGR) — the webcam case this article answers.
• Target rate: 15 fps, the usual minimum before motion reads as video.
• Link: 100 Mbps Ethernet with a 1500-byte MTU, so the largest unfragmented UDP payload is 1472 bytes (1500 − 20 IP header − 8 UDP header).
• Design payload: 1400 bytes per datagram, leaving headroom for tunnels and stacked headers.
• Per-datagram overhead: 28 bytes of IP and UDP header.

The rules, in words: a frame splits into whole datagrams of payload size, every datagram pays 28 header bytes on the wire, and a burst must finish inside the time between frames or it collides with the next one.

The arithmetic: 640 × 480 × 3 = 921,600 bytes per frame. That is ceil(921,600 / 1400) = 659 datagrams, each 1428 bytes on the wire. Wire rate: 659 × 1428 × 8 × 15 = 112,926,240 bit/s ≈ 112.9 Mbps, above the 100 Mbps line rate before any other traffic. The burst check removes all doubt: 659 × 1428 × 8 / 100,000 ≈ 75 ms, longer than the 66.7 ms between frames at 15 fps (1000 / 15). Raw cannot fit this link at any pacing.

Compress first, then split. Mid-quality JPEG on photographic content typically lands between 8:1 and 15:1, so plan on the cautious 8:1: 921,600 / 8 = 115,200 bytes, ceil(115,200 / 1400) = 83 datagrams, and 83 × 1428 × 8 × 15 ≈ 14,222,880 bit/s ≈ 14.2 Mbps. The burst now takes 83 × 1428 × 8 / 100,000 ≈ 9.5 ms of the 66.7 ms budget, and the stream uses about a seventh of the link.

The 100-byte split is the worst of both worlds: 9,216 datagrams per frame, 138,240 per second, and 28 header bytes for every 100 of payload — 141.6 Mbps of wire traffic that also floods the send queue.

Decision: on a 100 Mbps link, per-frame JPEG at a mid-quality setting is the design — 83 paced datagrams per frame, about 14.2 Mbps, 9.5 ms bursts. On gigabit Ethernet end to end, raw passes the arithmetic (112.9 Mbps is about 11% of the line), but it still needs the sizing, header and pacing fixes below.

Size each datagram from the MTU

Keep every payload under the smallest MTU on the path. Standard Ethernet caps the unfragmented UDP payload at 1472 bytes; 1400 is the design number because VPN encapsulation and stacked headers shave bytes off in the field. One send call per frame is not an option either way: IPv4 limits a single UDP datagram to 65,507 bytes. The other extreme is just as costly — 100-byte pieces pay 28 header bytes per 100 of payload and issue fourteen times as many send calls as the same frame in 1400-byte pieces. If you administer every hop, jumbo frames raise the MTU to about 9000 bytes and cut a raw frame from 659 to about 103 datagrams; anything else stays at 1400.

Put a reassembly header in every datagram

UDP can lose, duplicate or reorder datagrams, so each one must say what it is. Put a fixed binary header at byte 0 of every datagram — in every one, not once per frame, and not as text tags appended after the pixels. Text tags found with a substring search cost a scan per datagram and still cannot tell you which fragment never arrived. One workable layout:

The receiver then runs five rules:

1. Parse the 8 header bytes before touching the payload, with a byte order agreed on both ends; fixed little-endian is the simple choice on the platforms you are likely to pair.
2. Keep one reassembly buffer per frame ID, and only for the two newest IDs, so a datagram lost in the network cannot reserve memory forever.
3. Store each fragment once, at its index.
4. When indices 0 through count − 1 are all present, join the payloads and decode. Before that, paint nothing.
5. When a new frame ID arrives, discard the older unfinished frame.

Pace the sender and count the drops

Send each frame's datagrams as one tight burst, then idle until the next capture tick. The burst size decides whether the link absorbs it: the 83-datagram JPEG burst finishes in 9.5 ms, while the 659-datagram raw burst needs 75 ms and collides with the next frame. Pace against the capture clock, not against a guessed wait of a few milliseconds — the clock is the real constraint, and a fixed delay that is too short simply overruns the queue again.

Check every send call's return value: writeDatagram reports the bytes sent, or −1 on error. A rising count of −1 means the send queue is full and frames are dying before they leave. On the receive side, count the datagrams that arrive per frame ID; sent minus received is your loss rate as a plain number, and the buffer-then-process discipline that keeps such pipelines alive is worked through in high speed camera recording without dropped frames.

Compress each frame to JPEG, not a lossless squeeze

qCompress is lossless, so it removes exact byte repeats only — and camera frames are full of sensor noise, exactly the part lossless compression cannot squeeze. Photographic data rarely shrinks by half, so the bandwidth decision above stands unchanged. JPEG is lossy and built for photographs: encode the finished frame first, then split the encoded bytes into datagrams. Splitting after compression is what took the example from 659 datagrams to 83; compressing fragments, or compressing after splitting, throws that away.

Measure your own ratio instead of trusting a figure: encode twenty real frames from your camera, average the encoded sizes, and recompute the budget. If the link is far narrower — a shared 10 Mbps run, Wi-Fi, a VPN — per-frame JPEG will not fit and a video codec with inter-frame prediction is the next step. That is a larger build, so buy it only when the math demands it.

Machine vision answers the raw-versus-compressed question at the protocol level, which is why camera interfaces stream raw: in-camera JPEG compression: why machine vision cameras stay raw covers that trade-off.

Rebuild the QImage in one pass

A loop that calls setPixel once per pixel issues 307,200 guarded calls per frame — bounds check, format check and write, each time — and falls behind 15 fps before the network is even involved. Build the image from the buffer directly instead: construct the QImage over the received bytes with the constructor that takes a data pointer, width, height and bytes per line, and pass the capture's real stride rather than width × 3. Image libraries pad rows to alignment boundaries; the stride the sender already logs as widthStep is the number to use. For 640 × 480 colour the padding happens to vanish, because 640 × 3 = 1,920 bytes is a multiple of 4 — but a 641-pixel-wide frame pads each row to 1,924 bytes and shears the picture if you ignore it.

The buffer is BGR, so wrap it with Format_BGR888 (Qt 5.14 and later have it) or swap the two channels into a line buffer while copying. Either way it is one linear pass over 921,600 bytes — about 0.2 ms at a 5 GB/s copy rate, by estimate — followed by a single convertToFormat(Format_ARGB32) if the widget wants ARGB32, or a direct paint of the RGB image. Other raw formats need the same one-pass discipline; YUYV to RGB conversion in MATLAB: decode raw camera frames works an example.

Keep one test on the bench: send a solid red frame. Blue on screen means channel order. A diagonal shear means stride. A torn band means transport. Three failure modes, three distinct signatures, no guessing.

When the link or the camera should carry the load

If the two programs meet across Wi-Fi, a VPN or the public internet, switch the stream to TCP. Retransmission turns loss into a slightly later frame; UDP turns it into a tear. Paced UDP with per-frame JPEG wins only on a clean, managed LAN where drops sit at zero and you control the pacing.

If the job is machine vision — a camera feeding an inspection program — stop before hand-rolling any of this. GigE Vision cameras packetize the image within the MTU, recover lost packets at the driver level and stream raw frames over the same Ethernet cable, which is why the area scan cameras category spans every resolution without a custom protocol in sight. The camera finder filters them by interface, resolution and shutter. A compact model such as the MV-CU020-90GC puts a 2.5 MP colour picture from a 1/1.7-inch global shutter sensor on the wire over GigE with no transport code of yours at all.

On the receiving end, the vision controllers category is the shortlist: fanless industrial computers with multiple camera-side LAN ports and built-in light channels. The MV-VC3301H-128G66 combines 5+ LAN ports, an Intel Core i3 CPU, 8 GB of memory and 4 built-in light channels in one unit.

If the endpoint is a shop-floor display rather than your own program, the panel-native route already exists — streaming IP camera live video to HMI panels skips the custom transport entirely.

Before you ship the stream

Run this list against the code before the first live frame:

Sender

• Payload of 1400 bytes on standard Ethernet; a jumbo size only on a network you configure end to end.
• The 8-byte header — frame ID, fragment index, fragment count — at byte 0 of every datagram.
• One burst per frame, timed by the capture clock, with no guessed waits.
• Every send call's return checked; −1 counted and logged once per second.

Wire

• MTU confirmed on every hop the stream crosses; a raw frame is 659 chances to lose one datagram.
• No other bulk traffic sharing the link during the test window, or the budget is fiction.

Receiver

• Fragments buffered per frame ID; incomplete frames discarded, never painted.
• Decode plus conversion measured against the frame budget — 66.7 ms at 15 fps.
• The solid-colour test frame run once, to separate transport, stride and channel-order faults.

If this stream feeds a machine-vision build, start the camera configurator to pair the camera with its lens and lighting, or reach the team through the contact page with your link budget. More capture-pipeline debugging lives on the systems topic page.