Stream depth frames to a remote PC over gigabit ethernet

A depth pipeline often splits in two: a small board beside the camera grabs frames, and a desktop host does the heavy processing. Whether that split works is a bandwidth question, and bandwidth is arithmetic, not guesswork. A 640×480 depth stream at 30 fps is 18.4 MB/s of data, so a single gigabit ethernet link carries depth and color together with half its capacity still free. This how-to sizes the stream, chooses between the SDK's own network mode and your own TCP framing, specifies a header that rebuilds each frame byte-exact on the host, and lists the acceptance checks that prove frames arrive gap-free before the inspection code goes live.

What you need

• A depth camera and SDK running on the embedded board, delivering 16-bit depth (Z16) and, if you align to it, color, at full rate. The 3D sensors category covers laser profilers, structured-light and ToF cameras, and the camera finder filters them by type.
• A gigabit ethernet path from board to host: a direct cable on the bench, or two ports on a switch that carries nothing else during the test. If the link never comes up at all, work through 3D camera not pingable on a Linux controller: fix subnet and link first.
• A cable rated for the run. The camera cables category carries RJ45 data cables in fixed lengths from 3 m to 15 m, and the accessories finder filters by length.

For the bench hop between board and host — 3 m of high-flex, drag-chain-rated GigE cable:

For the longer run across the cell — 15 m of RJ45 camera data cable with a locking stud:

Step 1: size the stream before writing any code

Every later decision falls out of one multiplication.

A sustained TCP stream on ordinary gigabit hardware reaches about 110–118 MB/s of that 125. Plan new streams to stay under roughly 80 MB/s, which leaves retransmits, the control channel and any other traffic their share of the link.

Worked example — depth plus color, 640×480 at 30 fps:

1. Pixels per frame: 640 × 480 = 307,200.
2. Depth, 2 bytes per pixel: 307,200 × 2 = 614,400 B per frame; × 30 fps = 18,432,000 B/s ≈ 18.4 MB/s.
3. Color, 3 bytes per pixel: 307,200 × 3 = 921,600 B per frame; × 30 fps = 27,648,000 B/s ≈ 27.6 MB/s.
4. Total payload: 46.1 MB/s. TCP and IP headers add about 3 % (40 B per 1460 B packet), so the wire carries roughly 380 Mb/s of the 1000 available.

Decision: both streams fit one gigabit link with half of it free. The wire itself needs 1,536,000 B × 8 ÷ 1000 Mb/s = 12.3 ms per frame, so capture, copying and host processing share the remaining 21 ms of the 33 ms frame period.

The last row is the case that breaks gigabit; a later section decides it. When several cameras feed one host, run the same sum per camera — Multi-camera RGB depth bandwidth: size a four-camera host works it through for four.

Step 2: pick the transport — SDK network mode or your own framing

Two transports work; pick by support, not preference.

1. SDK network mode. Some depth SDKs expose the camera on ethernet: a server runs on the board and the host SDK adds it as a network device, so the host pipeline builds real frame objects and alignment calls run unchanged. The catch is support: networked modes whitelist fewer resolution and frame-rate combinations than the local interface, and at least one SDK shipped the feature as an experiment and later removed it. Read the list in your SDK version and treat it as the contract.
2. Your own TCP framing. You serialize the depth and color buffers on the board and rebuild them on the host with code you own. Any resolution and rate the hardware supports is available, but the host processes your buffers — which is why the naive version segfaults.

Decision: if your SDK version lists your exact resolution and fps on the network path, use it and stop here. Otherwise build the framing in the next section. If a desktop is not an option either, a vision controller beside the switch is the middle path — see Offload 3D inspection processing to a vision controller.

Step 3: frame the bytes so the host can rebuild them

A segmentation fault is the process reading memory it does not own — the classic definition — and in streaming code it almost always means the frame contract is missing, not a broken SDK.

The field names below are illustrative; the contract is yours. What a depth receiver needs:

1. A fixed 32-byte header, little-endian: magic (uint16), version (uint8), stream_id (uint8, 0 for depth and 1 for color), sequence (uint32), timestamp_ns (uint64), width (uint16), height (uint16), stride_bytes (uint16), bytes_per_pixel (uint8), flags (uint8), payload_bytes (uint32), header CRC32 (uint32).
2. One write per frame: header, then exactly payload_bytes of image data. Turn on the no-delay socket option (TCP_NODELAY) so the stack sends the first bytes of a frame immediately.
3. Send stride_bytes from the SDK, never width × bytes_per_pixel: drivers pad rows, and a 640-wide 16-bit row is 1,280 B that a driver may pad to 1,344 B — assuming 1,280 shifts every row after the first.
4. Ship calibration once at connect: depth intrinsics, color intrinsics and the transform between them, as a small versioned config message. The host needs all three to align; Aligned depth streams: data layout and which intrinsics apply explains which matrix does what.
5. On the host, read exactly payload_bytes into a pre-allocated buffer, check magic and CRC32, and advance the sequence counter. On any miss, drop the frame and count it — never process a partial.
6. Send depth as the native two bytes and convert on the host: floats double the payload to 36.9 MB/s at this size and buy nothing.

Step 4: keep the sender light

1. Serialize straight from the grabber's callback buffer: one copy of 1.5 MB costs a fraction of a millisecond on desktop memory and about a millisecond on embedded-class memory. No per-pixel work belongs on the board.
2. Leave the NIC's checksum and segment offload enabled; that offload matters most exactly when the board's CPU is slowest.
3. If frames vanish before they reach the socket, the loss is upstream: a camera behind a USB hub can starve before the network ever sees it — Depth camera fails on a USB hub: why raw streams refuse to start.

Step 5: rebuild and process on the host

1. Allocate one buffer per stream at startup and read into it forever; per-frame allocation is where receivers bleed.
2. Once the CRC passes, the buffer is byte-exact — run alignment and measurement on it. Match the depth and color resolutions before aligning or the maps drift; Depth to color alignment breaks up: match stream resolution first is that failure mode.
3. If the inspection code insists on SDK frame objects on the host, the network mode from step 2 is the clean route: your own framing cannot feed the SDK pipeline unless the SDK documents a constructor for external buffers.

Step 6: verify with numbers, not eyeballs

1. Baseline the path first: a plain throughput test over the same cable should report about 110 MB/s or more on a direct gigabit hop. If it reports 40, fix duplex, cable or switch port before touching code.
2. Prove the framing with a loopback: send 1,000 synthetic frames of 614,400 B and require sequence 0–999 with no gaps, 1,000 of 1,000 CRC matches, and 18.4 MB/s at the receiver — step 1's arithmetic reproduced end to end.
3. Soak 10,000 real frames: fps locked at 30, zero sequence gaps, zero CRC failures. Occasional misses on a shared switch mean the 80 MB/s budget is already spent.
4. Watch latency with a periodic sync marker: a direct gigabit hop adds well under a millisecond of queuing at this load. If the capture side itself looks slow in software, compare against 3D camera capture time slower in the API than in the viewer.
5. If fps sags only when a second camera joins, test one stream at a time to isolate the shared bottleneck — Three depth cameras drop to low FPS: test one stream at a time.

When gigabit is not enough

The 1280×720 case from step 1, decided:

• Depth alone at 30 fps: 55.3 MB/s — fits.
• Depth at 30 plus color at 15 fps: 55.3 + 41.5 = 96.8 MB/s — inside the ~118 MB/s usable rate but at 82 % of it; fine on a dedicated link, not a shared one.
• Both at 30 fps: 138.2 MB/s — no framing, checksum or driver setting fixes arithmetic that exceeds the wire.

Fixes, in the order that costs least: halve the color rate (96.8 MB/s), drop color (55.3 MB/s), compress the depth payload (smooth depth fields typically halve with a fast LZ-class codec such as LZ4; noisy edges compress poorly, so plan on 2×, not 5×), or put 2.5 GbE at both ends — 312.5 MB/s at the wire, about 280 usable, which carries 720p at full rate with margin.

Compressed video streams are a different product: an MJPEG camera feed into a SCADA or DVR client is built for display, and configuring and troubleshooting an MJPEG camera stream in SCADA is a different problem from moving raw Z16 — display-grade compression discards the pixel values depth math needs. If the application tolerates seconds of delay instead of milliseconds, skip live streaming entirely and record on the board for post-processing — 3D camera recording to H5 files for offline processing.

Before you order: the quick checklist

• Sum width × height × bytes × fps for every stream and keep the total under about 80 MB/s — the budget that leaves the link usable for everything else.
• Confirm NIC, switch port and cable are all gigabit-class, and the cable length sits inside its rated run.
• Confirm the SDK version supports your resolution and fps on its network path, or commit to your own framing.
• Read stride from the SDK at startup instead of assuming width × bytes per pixel.
• Send intrinsics and extrinsics once at connect, versioned, before the first frame.
• Make the receiver check magic, CRC32 and sequence numbers, and count every drop.
• Pass a 10,000-frame soak: fps locked, no gaps, zero CRC failures.
• Keep wire time plus capture plus processing inside the frame period — 12.3 ms of wire per 1.5 MB frame at 30 fps leaves 21 ms for the rest.

The decision, one line each: 640×480 depth and color at 30 fps streams over one gigabit link at 46.1 MB/s with room to spare; 720p survives with depth at 30 and color at 15 or less; anything past 118 MB/s moves to 2.5 GbE or compression. Pick the cable for the run through the cable configurator step, browse the other depth guides on the 3D topic page, or start a quoted multi-camera build at the contact page.