You stream 1280x720 at 48 fps over Wi-Fi and measure 100-120 ms from photons on the sensor to photons on the screen. That is five to six frame periods, and arguing about which stage owns them without data wastes integration weeks. The method that settles it: timestamp the same frame at every handoff — exposure start, raw buffer callback, encoder input, encoder output, socket receive, decoded frame ready, display flip — then difference the consecutive stamps. This guide shows where the clocks go, how to instrument an encode path your framework normally wires for you, how to carry a per-frame timestamp across a TCP stream, and how to read the table that comes out. You leave with a per-stage budget for your own stream and one stage to attack first.
How much latency should each stage of a 48 fps pipeline have?
One frame period at 48 fps is 1000 / 48 ≈ 20.8 ms, so 100-120 ms end to end is 4.8 to 5.8 frame periods; each frame-quantized stage you can name — exposure plus readout, encode, decode, display — contributes whole periods, and only the network contributes jitter.
Run the arithmetic once: 1000 / 48 = 20.8 ms; 100 / 20.8 = 4.8 frames; 120 / 20.8 = 5.8 frames. A stage that must finish a whole frame before the next handoff is quantized to the period. An encoder holds at least one frame to build a reference, a decoder mirrors that hold, and a display scans out on vsync boundaries. Exposure is the one term you set directly, and it can be any fraction of the period. Five periods spread over six handoffs is a healthy pipeline; one stage at three periods, 62.5 ms, is a fault. When the picture closes a control loop instead of feeding a viewer, this table becomes the loop's delay budget — see image-based control latency: budget the full feedback loop.
Where do I timestamp the same frame to split the delay?
Stamp one identified frame at every handoff and subtract consecutive stamps: exposure start, raw buffer callback, encoder input, encoder output callback, socket receive, decoded-frame-ready, display flip — seven stamps give six stage numbers for that frame.
Three rules keep the numbers honest:
- Every device-side stamp comes from one monotonic clock. Wall-clock time goes in the log file, never into a difference.
- The anchor comes from the camera. Industrial area scan cameras with chunk data enabled embed a timestamp and the exposure value in each frame — chunk data on camera frames. Learn which edge it latches, because chunk data timestamps latch at exposure end, and a full period of error lives on that one setting.
- Pair rows by the frame's presentation timestamp (PTS) or frame ID, never by arrival order.
| Consecutive stamps | What the difference measures |
|---|---|
| Raw callback - exposure start | Exposure, sensor readout, transfer to host |
| Encoder input - raw callback | Your code: format conversion, copies |
| Encoder output - encoder input | Encoder pipeline depth |
| Socket receive - encoder output | Network, including retransmissions |
| Decoded ready - socket receive | Decoder pipeline depth |
| Display flip - decoded ready | Render queue and vsync wait |
The six differences sum to glass-to-glass latency for that frame, even though the stages work on different frames concurrently. That overlap is why 48 fps throughput and 110 ms depth coexist in a healthy design.
How do I measure encoding latency between input and output?
Break the framework's automatic connection between the camera output port and the encoder input port, enable both ports yourself with a callback each, and log PTS plus monotonic clock in the camera callback and again in the encoder-output callback; the difference on one PTS is the encoder's latency.
- Set the camera port's timestamp mode so buffer headers carry the raw system-time-clock value, not a counter reset at start-up. With a reset counter, PTS values are offsets you cannot line up against your own clock.
- Replace the framework-managed connection between camera output and encoder input with two ports you enable yourself, each with a callback you register.
- In the camera-buffer callback, read the PTS from the buffer header, read the monotonic clock, append both to a log, then forward the buffer to the encoder input.
- In the encoder-output callback, read the PTS, read the clock again, append. Return buffers immediately; copy the payload first if you must process inside the callback — torn frames in a camera callback start exactly there.
- Pair the two records by PTS and subtract: encoder output clock - input clock for that PTS is the encoder's latency.
- Write every frame's row to a CSV, not only the summary, so you can re-cut the data later.
Expect one period, 20.8 ms, from an encoder holding a single reference frame; each B-frame in the structure adds another full period. Two periods or more means the encode structure is your first target, not the radio. On an industrial camera the same split is the sensor-embedded timestamp against host receive time — camera buffer timestamp latency.
How do I carry the timestamp across the network?
Prefix each frame with your own header — a 32-bit length and a 64-bit timestamp — because an H.264 elementary stream carries no per-frame timestamps; a container, or your own framing, is the only way the number survives the trip.
Two facts about the transport do the rest of the work:
- TCP is a byte stream. One write on the sender is not one read on the receiver, so parse your length prefix and accumulate bytes until the frame is complete. Never infer frame boundaries from where a read returns; a decoder finds frames by start codes, not by your socket calls.
- Sender and receiver run different clocks. Measure the round trip and halve it, or exchange a clock offset once at session start, before differencing stamps taken on the two hosts.
The Wi-Fi hop is the one stage that is jitter rather than frame-quantized, so report its median and its 95th percentile; the tail is what a viewer notices. For radio-link behavior in exactly this situation, read troubleshooting Wi-Fi video latency on a robot.
What does a solved 110 ms budget look like?
Stamps of 0.0 ms at exposure start, 21.6 at the raw callback, 21.9 into the encoder, 44.3 out of it, 79.0 at the socket, 99.9 decoded and 110.1 on the display sum to 110.1 ms, or 5.3 frame periods, with the network the largest stage at 34.7 ms.
Suppose one frame's stamps land like this, exposure locked at 5.0 ms and the receive-side clock corrected by a measured offset:
| Handoff | Stamp (ms) | Stage | Stage latency (ms) |
|---|---|---|---|
| Exposure start, camera timestamp | 0.0 | — | — |
| Raw buffer callback | 21.6 | 5.0 exposure + 16.6 readout and transfer | 21.6 |
| Encoder input | 21.9 | Your code | 0.3 |
| Encoder output callback | 44.3 | Encoder | 22.4 |
| Socket receive | 79.0 | Network | 34.7 |
| Decoded frame ready | 99.9 | Decoder | 20.9 |
| Display flip | 110.1 | Render and vsync | 10.2 |
Show the arithmetic: 44.3 - 21.9 = 22.4 ms encode; 79.0 - 44.3 = 34.7 ms network; 99.9 - 79.0 = 20.9 ms decode; 110.1 - 99.9 = 10.2 ms render; total 110.1 ms; 110.1 / 20.8 = 5.3 periods. The decision falls straight out of the table. The network owns 34.7 ms — 1.7 periods, the only non-quantized stage — so a wired link or a tighter retransmission budget buys 25-30 ms, and nothing else here is worth touching first. Encode at 22.4 ms is 1.1 periods, one reference frame, normal; drop B-frames only if you need that last period back.
Which camera settings skew the per-stage numbers?
Exposure time and shutter type own the first stage: your raw-callback stamp trails exposure start by the full exposure plus readout, so an unlocked auto-exposure makes the first difference wander by milliseconds from frame to frame while you measure.
- Lock exposure for the whole run. When exposure must change frame to frame, that loop has its own one-frame delay to account for — set exposure every frame.
- Shutter type sets where the first stage's time goes. A rolling shutter spreads exposure across the frame, so know whether your anchor is first-line or last-line exposure. A global-shutter head such as the MV-CU020-90GC pins the whole frame to one exposure instant.
- On USB3, host-side staging masquerades as readout when the usbfs memory allowance is tight — USB3 camera low frame rate and latency. A head like the LS-CMR-0166, 12 MP at 4000 × 3000 and 30 fps, makes any host-side queue obvious because the sensor side is steady.
- For an absolute marker, the camera's strobe output pulses with exposure and a scope sees it — camera strobe output timing.
For the head itself, a global-shutter GigE camera keeps the readout stage flat while you chase the others:
- Sensor format
- 1/1.7"
- Lens mount
- C-mount
- Interface
- GigE
- Resolution
- Up to 3 MP
When is the per-stage budget wrong?
The budget understates reality whenever a stage holds more than one frame — encoder lookahead, a jitter buffer, a decoder queue or vsync each add whole extra periods, so a six-stage pipeline can measure eight periods deep while every individual stage reports one.
A stage that quietly holds several frames is invisible to a single-frame measurement; it shows up as depth you cannot explain. Compare the first second of a run with the last: a stage whose latency creeps upward is a queue filling under load, not a slow component. Encoded image pipeline latency: find hidden frame buffering walks that hunt, and pipeline latency that grows toward one second is a worked case where queue depth, not the radio, owned the delay.
The other blind spot is frame repetition: a receiver that re-displays the last decoded frame shows a healthy display stage while the content ages. Per-stage stamps miss it; the photodiode check below catches it.
What do I check before trusting the numbers?
Capture 60 seconds — 2,880 frames at 48 fps — take the median and the 95th percentile of every stage, then confirm the stage medians sum to an independently measured end-to-end figure; a mismatch larger than one period means a stamp sits on the wrong clock.
- One monotonic clock per device; stamps cross devices only through the embedded timestamp or a measured offset.
- Rows paired by PTS or frame ID, never by arrival order.
- Median and 95th percentile per stage; the mean follows the Wi-Fi tail, not the viewer's experience.
- First second compared with last second; a growing stage is a queue, not a slow part.
- Exposure locked and encode structure fixed for the whole run.
- Callbacks return buffers at once; copy when you must process — reuse image buffers.
- Absolute cross-check: drive an LED from the camera's strobe line, put one photodiode on the LED and one on the screen, and read the gap on an oscilloscope. This replaces the mirror test — a camera pointed at its own display — whose feedback loop strobes the screen hard enough to matter for photosensitive viewers.
The table also tells you when hardware is the fix. A head whose frames carry chunk-data timestamps removes the clock-anchor problem, and vision controllers have the margin to log every stamp without becoming a stage themselves — the MV-VC5525-256G70-NN brings 32 GB of memory and PoE camera ports to the same cell. Pair the two ends and run the method:
- Sensor format
- 1/1.7"
- Interface
- USB3
- LAN ports
- 5+ LAN ports
- USB 3 ports
- 3-4 USB 3 ports
Start in the camera finder for the head, or let the configurator build the chain from camera to host:
The configurator matches a camera, lens, light, light controller and cables to each other, step by step, and lists only parts that fit what you already picked.
Questions
Why does my total latency land near whole frame periods?
Every frame-quantized stage adds exactly one period, 20.8 ms at 48 fps, so five such stages give 104 ms and six give 125 ms — which is why 100-120 ms keeps showing up. If one stage measures a non-integer number of periods, it is mixing in jitter or a partial exposure; the integer part still tells you how many frames that stage holds.
Median or mean when I report each stage?
Median for the budget, 95th percentile for the worst case. In a 60-second run at 48 fps you collect 2,880 frames, and a few percent of frames arriving 100 ms late pulls the mean up while the median does not move. The median is what the pipeline delivers most of the time; the percentile is what the viewer remembers.
Can I embed a timestamp in the H.264 elementary stream itself?
No. An elementary stream has no per-frame timestamp field you can set from the capture side; a decoder finds frames by start codes. Wrap the stream in a container, which carries presentation timestamps, or send your own header — a 32-bit length and a 64-bit timestamp — ahead of each frame and strip it on receive.
How do I measure glass-to-glass latency without instrumenting the pipeline?
Drive an LED from the camera's strobe output, aim the camera at the display, and put one photodiode on the LED and one on the screen. The oscilloscope gap between the two pulses is exposure-to-photon latency with no clocks to trust, and it also catches frame repetition that per-stage stamps miss.
Does raising the frame rate reduce end-to-end latency?
Yes, for the frame-quantized part. Halving the period halves every whole-frame stage: the five-period pipeline that costs 5 × 20.8 = 104 ms at 48 fps costs 5 × 10.4 = 52 ms at 96 fps. Exposure time and radio jitter do not scale with frame rate, so measure those separately before crediting the faster sensor.


