Feed socket-streamed frames to YOLO without temp files

You have a camera on one PC and a YOLO detector on another, and the detector's stock demo command accepts only a camera it opens itself or a video file path. It never reads a socket. The workaround people reach for first — write each received frame to a temp JPEG, run file-mode detection on it, delete, repeat — works and is slow: at 30 fps you own 33.3 ms per frame, and the disk detour spends a third of that before inference starts. The right shape is shorter. Read the bytes off the socket, decode them in memory with cv2.imdecode, and pass the resulting NumPy array straight into the detector's Python API. Here is the wire protocol, the code for both ends, the bandwidth arithmetic and the hardware choices.

The decision: bytes in, arrays to the detector, never files

The demo binary has two input modes: a camera it opens itself, or an image file whose path you name. Neither reads a socket, which is where the temp-file habit comes from. The library underneath takes an image already in memory: every current Python wrapper for YOLO accepts a NumPy array, and the classic C bindings ship a helper that copies an array into their internal image struct. The server loop is therefore:

socket bytes → cv2.imdecode → NumPy array, shape (H, W, 3), dtype uint8, BGR → detector call → boxes → JSON back to the client

Files appear only when you choose to log a frame. The same decode-in-memory pattern serves other stream readers; Decode barcodes from a live RTSP or MJPEG video stream applies it to code reading.

Size the stream first — the arithmetic that settles it

Check that the link can carry the frames before you write a line of code:

Worked example, 640 × 480 MJPEG at 30 fps:

• Raw BGR: 640 × 480 × 3 B = 921,600 B ≈ 0.92 MB per frame.
• JPEG at quality near 80 on natural video: 40-70 KB per frame; use 50 KB as the average.
• 50 KB × 30 fps = 1,500 KB/s ≈ 1.5 MB/s ≈ 12 Mbit/s.
• A gigabit link moves about 940 Mbit/s of payload, so headroom is roughly 78×.

The stream is small; that is why the temp-file detour was never necessary. Resolution is what bites. A 12 MP machine-vision camera such as the LS-CMR-0166 sends 4000 × 3000 frames at 30 fps over USB3. Raw BGR at that size is 36 MB per frame, and 36 MB × 30 fps = 1,080 MB/s ≈ 8.6 Gbit/s — it does not fit a gigabit LAN. JPEG at about 2.5 MB per frame gives 75 MB/s ≈ 600 Mbit/s, which fits with margin.

Commit to this rule: compress on the client above roughly 1 MP on a gigabit link. Sending raw arrays instead of JPEG is defensible at 640 × 480 — 27.6 MB/s — and unencoded capture to a NumPy array: skip JPEG encode and decode shows that side of the trade.

Step 1: Frame each image and send it

TCP is a byte stream, so you delimit frames yourself. Length-prefix framing — four bytes that say how long the payload is, then the payload — is one line on each end and survives any packet split.

1. Capture into a NumPy array on the client; camera module image acquisition without saving files shows capture straight into arrays.
2. Encode with cv2.imencode(".jpg", frame).
3. Prefix the payload length as a 4-byte big-endian integer.
4. Send header and payload in a single sendall call.

import socket
import struct

import cv2
import numpy as np

SERVER: tuple = ("192.168.1.20", 5555)          # your server's address

# from your capture call; shown here with its type and shape
frame: np.ndarray = np.zeros((480, 640, 3), dtype=np.uint8)

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(SERVER)

ok, encoded = cv2.imencode(".jpg", frame)       # encoded: np.ndarray, uint8
payload: bytes = encoded.tobytes()
header: bytes = struct.pack(">I", len(payload))
sock.sendall(header + payload)

# the test: the header must round-trip the payload length
assert ok is True
assert struct.unpack(">I", header)[0] == len(payload)

Step 2: Read one whole frame and decode it

conn.recv(n) returns up to n bytes, not exactly n, so wrap it in a helper that loops. cv2.imdecode then does the whole job: it takes the bytes as a uint8 NumPy array and returns the decoded image, inferring the codec from the data itself — the sender can switch JPEG for PNG and the receiver does not change.

import socket
import struct

import cv2
import numpy as np

HEADER_LEN: int = 4
drop_count: int = 0

def recv_exact(conn: socket.socket, count: int) -> bytes:
    """Read exactly count bytes; raise when the peer closes early."""
    buf: bytearray = bytearray()
    while len(buf) < count:
        chunk: bytes = conn.recv(count - len(buf))
        if not chunk:
            raise ConnectionError("peer closed mid-frame")
        buf.extend(chunk)
    return bytes(buf)

# conn, addr = sock.accept() after bind and listen
header: bytes = recv_exact(conn, HEADER_LEN)
(length,) = struct.unpack(">I", header)
payload: bytes = recv_exact(conn, length)

buf: np.ndarray = np.frombuffer(payload, dtype=np.uint8)
frame = cv2.imdecode(buf, cv2.IMREAD_COLOR)

if frame is None:
    drop_count += 1                     # torn payload: skip it, count it
    continue
assert frame.shape == (480, 640, 3)    # your camera's height and width

cv2.imdecode returns None — it does not raise — when the payload is truncated or not an image. Test for None on every frame; a torn frame skipped is invisible, a crash on frame 40,000 is not.

Step 3: Hand the array to the detector

Pass the array in as-is. Current wrappers resize and letterbox internally to their input size, typically 416 or 640 px, and return boxes; how that resize pads the image changes box coordinates, and object detector input dimensions: height, width, and padding covers what to watch. Classic C bindings take one extra step: copy the array into their image struct with the helper they ship, in memory, then call detect. Either way the disk stays out of the loop.

Two efficiency habits are worth the keystrokes: reuse the decode buffers rather than allocating per frame — reuse image buffers instead of per-frame allocation shows the gain — and when the wrapper allows it, hand frames to the GPU once, as feeding GPU tensors to a vision model without a CPU round trip describes. Preprocessing beyond that is optional. The one case that pays on a live stream is a static scene, where averaging several frames beats sensor noise; calculating average images from ten camera frames shows that loop.

Step 4: One decode thread, one inference thread, one-slot queue

Put the socket read and the detector call on separate threads, joined by a queue of exactly one slot. The arithmetic says why. A detector at 35 ms per frame processes 1,000 / 35 ≈ 28.6 fps against 30 fps arriving — nearly balanced. On CPU at 80 ms it is 12.5 fps against 30, a deficit of 17.5 fps: an unbounded queue grows 100 frames deep in 100 / 17.5 ≈ 5.7 s, and your verdicts describe where the part was. A one-slot drop-oldest queue caps latency at one frame period plus one inference, permanently.

import queue

latest: queue.Queue = queue.Queue(maxsize=1)

def decode_loop(conn: socket.socket) -> None:
    while True:
        header: bytes = recv_exact(conn, HEADER_LEN)
        (length,) = struct.unpack(">I", header)
        payload: bytes = recv_exact(conn, length)
        frame = cv2.imdecode(np.frombuffer(payload, dtype=np.uint8),
                             cv2.IMREAD_COLOR)
        if frame is None:
            continue
        try:
            latest.put_nowait(frame)
        except queue.Full:
            try:
                latest.get_nowait()         # drop the stale frame
                latest.put_nowait(frame)    # keep the freshest one
            except queue.Empty:
                pass

import json

def run_detector(frame: np.ndarray) -> dict:
    # stand-in for your YOLO wrapper call; returns one verdict dict
    return {"found": False, "class": "", "score": 0.0}

def inference_loop(conn: socket.socket) -> None:
    while True:
        frame: np.ndarray = latest.get()    # blocks until a fresh frame
        verdict: dict = run_detector(frame)
        reply: bytes = json.dumps(verdict).encode("utf-8")
        conn.sendall(struct.pack(">I", len(reply)) + reply)

When the camera outruns the detector, the queue only hides it; acquisition faster than processing: fix the frame rate gap ranks the real fixes.

Step 5: Send the verdict back on the same socket

The reply uses the same framing, so the client reuses recv_exact:

(reply_len,) = struct.unpack(">I", recv_exact(sock, HEADER_LEN))
verdict: dict = json.loads(recv_exact(sock, reply_len))

Keep the JSON small: found, class, score and the one box your PLC or robot acts on. Budget the round trip from parts you now know — about 3 ms to encode, 1 ms of LAN on one switch, 2 ms to decode, 35 ms of inference, 1 ms for the reply — roughly 42 ms from shutter to verdict at this scale.

Hardware for the two ends

The pipeline runs on any two PCs. When the cell becomes permanent, put parts meant for the job at each end.

Client. A webcam proves the software in an afternoon. A machine-vision camera earns its place with exposure control, a hardware trigger and a deterministic frame rate. Area scan cameras span sub-1 MP to beyond 25 MP; size them by the smallest feature you must detect, not by the stream — the camera resolution calculator turns field of view and object size into a pixel count before you buy.

The LS-CMR-0166 is the client camera of the worked example above; compress before the wire and its frames cross a gigabit LAN with room to spare.

Server. The detector wants a CUDA-capable GPU, so the server needs a PCIe slot for the card. Pick a card that draws its power from the slot and the choice stays simple. Vision controllers are built for this mounting and I/O.

The MV-VC5525-256G70-NN pairs a 12th-Gen Core i5 with 32 GB of memory and PoE camera ports, with PCIe expansion supported. The MV-VC3301H-128G66 is the Intel Core i3 option with 8 GB of memory and PCIe expansion supported.

No server at all. The other escape is not to stream.

The MV-SC5060XM-08M-WBN is a 6 MP smart camera with an 8 mm M12 lens that runs inspection on the device and reports over GigE. One station, fixed geometry, no pipeline to maintain: when the task is one camera and one verdict, smart cameras delete the whole network.

Check the pipeline before it ships

1. Assert the decoded shape equals your camera's (H, W, 3) on the first frame, not only in the lab.
2. Torn-frame drill: trim the last 100 bytes from one payload before decoding. imdecode must return None and the skip counter must advance — no crash.
3. Sixty-second soak at 30 fps in: expect about 1,800 frames received; at 35 ms inference, about 1,710 processed and roughly 90 dropped. The drop counter is the health signal; a growing backlog is the failure.
4. Pull the cable mid-stream: the ConnectionError fires, the socket closes cleanly, and the next connect works.
5. Ship the counters inside the verdict JSON during bring-up, then delete them.

When files on disk are the right answer

Temp files per frame are wrong. Files per event are often right:

• Traceability: when a part is rejected, append the payload bytes you already hold — one write of an already-encoded JPEG, no re-encode, no decode.
• Debugging: save the torn payloads that made imdecode return None; replaying them offline finds framing bugs in minutes.
• Retraining: keep a rolling capture when a model update is coming; the saved JPEGs are exactly what the detector saw.

In every case the file records a frame you already processed. It never sits between the socket and the detector.

Start at the camera step when the client hardware is the open question, or browse the GigE and USB3 camera finder when the sensor is already decided. For server sizing against your frame rate, ask the applications team.