RS232 image transfer missing last bytes: overrun and buffer fixes

Your serial camera module synchronizes, initializes and starts sending, yet every image arrives short by the last 4-10 bytes. That pattern is almost never a broken camera. It is the receiving side ending its read, or discarding bytes, before the frame finishes. This guide is for engineers moving picture data over RS232 into a vision controller, a PLC or an embedded board. You will pin down where the bytes die — sender, cable or receiver — with a byte count. Then apply the read-until-length loop, the timeout and the buffer habits that complete the transfer every time, and know when the honest answer is to move the image off serial entirely.

Where the bytes die: three places, one count

The pattern tells you more than the byte count does. A transfer that stops short by roughly the same 4-10 bytes every attempt is deterministic: read logic ends the transfer, or the sender stops, before the frame completes. Bytes missing from random places, or torn rows in the picture, point at receiver overrun or signal corruption instead.

Count before you theorize. Add up the bytes you actually received and compare them with the declared frame size from the module's protocol document: payload pixels plus header, checksum and end bytes. One counted run sorts the suspects:

• Expected total minus the same small tail on every attempt: read logic or the sender.
• A different total each attempt, gaps in random places: UART overrun or the physical link.

Overrun has a precise meaning here. It occurs when a new byte finishes arriving before your program has read the previous one out of the receive buffer; the UART sets its error flag and a byte is lost. Give the port the same discipline as any serial link — get the first reply under control before you trust a stream. A port that answers a handshake but truncates payloads has a read-logic problem, not a wiring one.

Cause 1: the read returns before the frame lands

Most serial libraries make this easy to get wrong. A read call returns whatever is in the buffer when it fires; it does not block until your requested count has arrived. A read sized for the whole frame that fires on the first chunk leaves the tail on the wire, and the code above it treats the short buffer as the complete image.

The other classic is the termination character. If the port is configured with one enabled, every read ends at the first byte that matches it — and binary pixel data will contain that value somewhere. Disable termination for binary transfers and frame by byte count instead. Closing the port or flushing buffers straight after the read returns discards bytes still sitting in the driver or the FIFO, so leave the port open until the count is complete.

The fix is a loop that keeps reading until the declared total is in:

expected_total = 76817         # int: 12-byte header + 76,800 pixels + 5-byte footer
received = 0                   # int, bytes counted so far
frame = bytearray()            # empty image buffer

while received < expected_total:
    chunk = port.read(min(expected_total - received, 4096))  # returns what is there
    if len(chunk) == 0:        # idle gap: sender stopped or link broke
        break
    frame.extend(chunk)
    received += len(chunk)

if received != expected_total:
    log_transfer_error(received, expected_total)  # record both numbers every time

The names are illustrative — each serial library has its own read call, and the loop shape is what carries over. The test that proves it: run the loop on a known frame, and it must return exactly expected_total. If it stalls at expected_total minus the same handful of bytes on every attempt, your receiver is fine and the sender is stopping; go to cause 3.

Framing by declared length is the same habit as sending images over a TCP socket with a length header: the declared length tells the receiver when to stop, never the timeout. It also matches the one-wait-per-frame rule for a frame that misses its timeout — one blocking wait per frame, not a polling loop that gives up early.

Cause 2: the receiver's UART overruns

Overrun is a traffic jam at the silicon. The next byte completes before the previous one has been read out of the receive buffer, the overrun flag sets, and bytes are lost. Your slack depends on the hardware: a 16550-class PC UART buffers 16 received bytes, while the small UARTs inside microcontrollers may hold only 1-2.

The arithmetic at 115,200 baud with 8N1 framing: a byte arrives every 86.8 µs, which is 11,520 bytes per second. A 16-byte FIFO therefore buys about 1.4 ms of headroom — 16 × 86.8 µs — between services of the port. Per-byte processing, per-byte logging or any decode work inside the read loop burns that headroom at once. An 8-bit microcontroller doing per-byte work cannot sustain 11,520 bytes per second; a PC loop reading in chunks can.

Test it instead of guessing. After each read, query the line status — every serial API and every UART datasheet names its overrun and framing flags — and log the counts. A rising overrun count together with gaps in random places confirms this cause.

Fixes, in the order to try them:

1. Read in chunks of 1-4 KB instead of byte by byte.
2. Move all decoding, logging and display out of the receive loop.
3. Enable RTS/CTS hardware flow control if the module supports it and the cable carries the wires, so the sender pauses while your buffer drains.
4. With no flow control available, lower the baud rate until the worst-case loop time fits inside the byte interval.

Cause 3: the sender stops or the signal corrupts

If the receiver counts cleanly and the flag registers stay quiet, the bytes were never sent or never arrived intact. Three checks, in order:

Sender limits. Some modules stop a transfer early: a firmware payload cap, a buffer boundary, or a checksum failure it never reports. Hex-log one transfer and compare the received bytes with the module's frame layout; the last bytes you received show whether the sender stopped mid-pixels or after a footer.

Footer blind spots. Modules that append a checksum and an end byte look short when your parser stops counting at the last pixel. Read the protocol description: a declared footer belongs in expected_total. A missing 5-byte tail is sometimes a 4-byte checksum plus an end marker the code never counted.

Signal quality. RS232 pushed to high baud rates over long or unshielded runs corrupts bytes randomly, not in a clean tail. Framing errors in the status register, with missing counts that vary between attempts, point here. Keep serial image runs to a few metres of shielded cable with latched connectors, and keep the link serviceable — a ready-made DB9 female-to-open lead beats a soldered pigtail:

Framing races are not unique to pictures, either. Null-filled tails and races in a PLC receive buffer are the same disease in another protocol: a reader consuming a buffer the writer has not finished filling.

The numbers: transfer time and the timeout that fits

Timing decides both your timeout and whether serial is viable at all:

A timeout under the computed transfer time guarantees truncation, and the miss then looks exactly like cause 1. Set the timeout from the arithmetic, not from what feels safe on the bench.

The last row carries the decision. If your inspection cycle needs the image in under about 1 second, no standard serial rate delivers even a QVGA frame, and a megapixel frame misses by roughly two orders of magnitude. That is the trigger for the architecture change below, and the wait discipline carries over when you move: cap the bulk-transfer wait so a stalled device cannot hang the loop.

The durable fix: keep the habits, move the payload

Whatever the transport, the habits stay: count received bytes against the declared total on every transfer and alarm on any mismatch; no termination character on binary data; one receive loop with chunked reads; overrun and framing flags checked and logged; timeout at 1.5 × the computed transfer time.

The architecture change is simpler than it sounds. Serial is the right wire for commands, configuration and results — not for payloads. Put the camera on GigE or USB3 and keep RS232 for what it does well: the fixed-position RS232 telegram that carries vision results to a PLC is exactly that split, images on the fast link and a few dozen bytes on the serial one. A short frame that slips into a latest-image slot fails twice, so drain or version your buffers the way duplicate frames from a latest-image-only queue get fixed.

A fanless vision controller closes the gap in one box: camera LAN ports, USB3 for the cameras that need it, built-in light channels and room for the inspection software. For a first cell, an Intel Core i3 with 8 GB of memory carries several GigE cameras and the lighting from one enclosure:

When every camera should take power and data down the same cable, step up to PoE camera ports and 8 built-in light channels:

The controller side is settled at that point; the camera side is not automatic, so check whether the camera port supports PoE before you order cabling. Ethernet also lifts the distance limit to 100 m per run on standard structured cabling, against a few careful metres for high-baud serial. The camera data cables range includes locked RJ45 leads; for a cabinet 15 m from the cell, this is the run:

Browse the vision controllers category or the camera cable finder to match ports to cameras before you order.

Prevention checklist

• Count every transfer. Received bytes against declared total, logged and alarmed — a silent short read is the bug that reaches production.
• Frame by length. Termination characters off for binary payloads; the declared frame length ends the read.
• One receive loop, chunked reads. 1-4 KB chunks, nothing else inside the loop, no per-byte logging.
• Read the flags. Overrun and framing counts after each read, in the log rather than in memory.
• Timeout from arithmetic. 1.5 × computed transfer time, and a zero-byte return treated as an idle fault.
• Right wire for the payload. A few shielded metres for high-baud serial; GigE or USB3 once cycle time matters.

If the table says serial cannot carry your frames, contact the applications team with your frame size and cycle budget, or start the system configurator at the cable and controller step. The systems topic page gathers the companion fixes for streams, timeouts and buffers.