A stream can decode smoothly while your inspection application works on images captured seconds earlier. Faster decoding helps only when decoding is the bottleneck; it does not empty a queue that keeps growing downstream. This procedure is for applications that already receive usable RTSP images but need fresher inspection data. You will separate frame age from processing time, locate accumulated waiting, and choose a buffering policy that matches the inspection task.
1. Define the freshness requirement first
- Set a maximum acceptable image age at the point where the inspection decision is used.
Frame rate describes throughput. Frame age describes how old the scene information is. An application can report a steady frame rate while processing a delayed sequence, so throughput alone is not a latency test.
Define two measurements:
- Processing-start age: elapsed time from image capture to inspection starting.
- Decision age: elapsed time from image capture to the completed decision becoming available.
For example, a hypothetical application might allow a maximum decision age of 200 ms. If inspection and result delivery together require a budget of 50 ms, the image must reach processing within 150 ms. These are application requirements, not camera specifications.
Decide what happens when that deadline cannot be met: discard the result, request another observation, or declare the inspection unavailable. Never silently treat an old result as current.
For measurement methods involving an optical event and a visible output, use the guide to measuring camera latency glass to glass. Display latency and decision latency have different endpoints, so document which endpoint you measure.
2. Establish trustworthy timestamps
- Record timestamps at acquisition, application handoff, inspection start, and decision completion.
Use a monotonic host clock for intervals measured entirely within the application. Record a frame identifier alongside each timestamp so events from different images cannot accidentally be paired.
A media presentation timestamp is not automatically a capture timestamp. It may describe a stream timeline without sharing the host clock's origin. Confirm its meaning before subtracting it from a host timestamp.
The capture time must be mapped into the same clock domain as the host measurements. That mapping needs a known offset and acceptable drift. Reconnection, clock adjustments, or timestamp discontinuities can invalidate it.
If trustworthy capture timing is unavailable, label your measurement honestly: handoff-to-decision delay is useful, but it excludes upstream waiting. Validate total delay separately with a controlled visual change and an independently observed decision output.
For one hypothetical frame, capture occurs at 10.000 s, inspection starts at 10.140 s, and the decision completes at 10.185 s:
- Processing-start age = 10.140 − 10.000 = 0.140 s, or 140 ms.
- Decision age = 10.185 − 10.000 = 0.185 s, or 185 ms.
- Inspection duration = 10.185 − 10.140 = 0.045 s, or 45 ms.
Against a 200 ms decision deadline, that frame has only 15 ms of remaining margin. Repeat the measurement under load rather than accepting one favorable observation.
3. Find the stage that accumulates waiting
- Compare the same stream through progressively more of the application.
Begin with continuous acquisition that immediately releases decoded frames. Add color conversion, then inspection, then result delivery. Keep the source profile and transport settings unchanged during these comparisons.
A display-only test is not equivalent to an inspection application. Different output stages can have different synchronization behavior, buffer limits, memory transfers, and scheduling costs.
| Observation | Likely area to investigate | Useful isolation test |
|---|---|---|
| Delay grows only after inspection starts | Processing cannot keep up, or application queues are unbounded | Replace inspection with immediate frame release |
| Delay appears before application handoff | Source encoding, transport, jitter buffering, or decoding | Measure upstream stages independently |
| Delay jumps after a network interruption | Reordering, retransmission, or recovery behavior | Record packet-loss events and frame-age changes |
| Throughput is steady but images remain old | A persistent backlog or deliberately buffered stage | Log queue occupancy and capture timestamps |
| Decode time improves but decision age does not | Another stage dominates waiting | Compare per-stage timing for the same frames |
Hardware decoding can reduce decoder execution time and host processing load. It does not remove encoder lookahead, network buffering, application queues, or copies into inspection memory.
If acquisition does not start reliably, resolve pipeline format negotiation errors first. If property access fails, use the separate RTSP property-access diagnostics. Neither problem should be confused with sustained frame-age growth after acquisition works.
4. Bound buffers throughout the decoded path
- Inventory every place a complete image can wait, then set an explicit limit or policy.
Check the decoder output, conversion stages, application sink, worker queue, display queue, and recording branch. A small final queue cannot compensate for an older backlog upstream.
For each queue, document:
- Its capacity in frames, bytes, or time.
- Whether a full queue blocks upstream work or discards data.
- Whether it discards the oldest or newest entry.
- Whether its contents are compressed packets or complete decoded frames.
- How you observe occupancy and discarded-frame counts.
Consider a hypothetical stream delivering 30 frames/s while inspection takes 40 ms per frame:
The final number describes queued processing work, not a direct measurement of the currently delivered frame's age. It nevertheless explains why adding memory can turn a temporary slowdown into seconds of delay.
Blocking a full application queue may simply move the backlog upstream. A freshness-oriented design usually needs a deliberate discard boundary for complete decoded frames, while acquisition continues draining the stream.
5. Use a latest-frame mailbox when skipping is allowed
- Separate continuous acquisition from inspection, and keep only the newest pending decoded frame.
Use this policy when the task asks, “What does the scene look like now?” It is unsuitable when every passing part requires a separate inspection record.
A practical design has an acquisition worker and an inspection worker. The acquisition worker receives complete decoded frames and atomically replaces the pending frame. The inspection worker takes the newest pending frame whenever it becomes available for more work.
Acquisition worker:
receive a complete decoded frame
attach frame identity and timing metadata
replace the pending frame under synchronization
release the replaced frame
notify the inspection worker
Inspection worker:
wait until a pending frame exists or shutdown is requested
take ownership of the pending frame under synchronization
clear the pending slot
validate timing and freshness
process the frame
validate decision age before publishing the result
Replacement applies only to the pending frame. The acquisition worker must not overwrite memory that inspection is still reading. Use explicit ownership transfer, reference-counted buffers with a valid lifetime, or a deliberate copy.
The mailbox prevents an arbitrarily long application backlog, but it does not interrupt an inspection already running. A slow inspection still delays the next opportunity to take a newer frame.
Track received, replaced, processed, and rejected frames separately. Otherwise a healthy freshness policy may look like unexplained frame loss.
For acquisition threads that stall while delivering images, review callbacks and buffer queues in multi-camera acquisition. For lifetime faults around shared image storage, use the guide to camera frame memory ownership.
6. Tune jitter buffering without hiding failures
- Reduce receiver buffering in measured increments while monitoring late frames and decoding failures.
A jitter buffer trades waiting time for tolerance of irregular packet arrival. Reducing it can improve freshness on a stable connection, but an excessively small allowance can increase incomplete images or recovery events.
Change one setting at a time. For example, test configured allowances of 150 ms, 100 ms, and 50 ms under the same workload. These are experimental settings, not promised end-to-end delays. Confirm the receiver's documented interpretation of each setting.
Record decision-age percentiles, maximum observed age, discarded images, decoding errors, and recovery duration. A lower median is not an improvement if long interruptions cause unacceptable missed inspections.
Transport selection also involves tradeoffs. Reliable delivery can add waiting during loss recovery; datagram delivery can lose packets. Choose against measured network conditions rather than assuming one transport always wins.
Source encoding settings matter too. Where documented controls are available, examine frame reordering, lookahead, and reference-frame intervals. Less buffering may require more bandwidth or change image quality.
Do not reopen the stream for every image. Keep acquisition active, and reconnect only through a bounded recovery state machine after a genuine failure. Reset or revalidate timestamp mapping after reconnection.
7. Verify freshness under overload and recovery
- Test the application with realistic processing, recording, and communications active.
A brief idle test rarely exposes queue growth. Run long enough to encounter sustained load and repeat representative interruptions in a controlled test environment.
Include these cases:
- Normal production workload.
- A temporary increase in inspection duration.
- A slow display or recording consumer.
- Loss of the stream followed by reconnection.
- Timestamp discontinuities or missing timing metadata.
- Shutdown while acquisition and inspection are both active.
Log queue occupancy alongside frame age. A queue that keeps growing indicates a throughput mismatch; a bounded queue with high age points toward upstream delay, slow active work, or incorrect timestamp interpretation.
At inspection start, compare measured age plus the remaining execution budget against the deadline. Check age again before publishing because execution may exceed its budget. This freshness gate is an inspection data-validity check, not a machine safety function.
When measurements disagree with observed conveyor position, review scene timing and stale pick data. A fresh image can still produce an invalid action if later result handling introduces delay.
8. Choose hardware around the acquisition requirement
- Decide whether compressed streaming is actually the right interface for the inspection.
RTSP can suit monitoring and continuously observed scenes. When every object requires a distinct exposure or tightly associated timestamp, compare the complete streaming design with direct acquisition from area scan cameras. Use the camera finder to narrow interface and shutter requirements.
The LS-CMR-0053 uses GigE and a rolling shutter. The LS-CMR-0166 uses USB3 and a rolling shutter. They are acquisition alternatives to evaluate, not examples of RTSP support or guaranteed latency.
- Sensor format
- 1/1.7"
- Lens mount
- C-mount
- Interface
- GigE
- Resolution
- 12 MP
- Sensor format
- 1/1.7"
- Interface
- USB3
- Resolution
- 12 MP
- Shutter
- Rolling
A different interface does not automatically solve exposure delay, host queuing, or rolling-shutter motion effects. If the task requires exposure timing tied to an external event, review hardware-triggered image acquisition and verify the required features for the selected camera.
For vision controllers, evaluate camera connectivity, processing capacity, and software compatibility separately. The MV-VC2140-128G40-1T is a controller candidate to assess against those requirements. The supplied catalog information does not establish decoder compatibility or application latency.
- LAN ports
- 3-4 LAN ports
- USB 3 ports
- 2 USB 3 ports
- Light outputs
- 4 built-in channels
- Memory
- 8 GB
9. Complete the acceptance checklist
Before committing the design, confirm that:
- The deadline refers to decision age, not merely decoder execution time.
- Capture timing is trustworthy, or the measurement limitation is explicitly documented.
- Every decoded-frame queue has a known capacity and discard policy.
- Pending-frame replacement cannot invalidate an image being processed.
- Skipping frames is acceptable for the inspection requirement.
- Slow recording or display work cannot silently accumulate inspection backlog.
- Reconnection resets timing assumptions and rejects stale pending data.
- Overload tests meet both freshness and inspection-coverage requirements.
If every item must be inspected, a latest-frame mailbox is the wrong remedy. Increase processing capacity, reduce the acquisition workload, or coordinate acquisition with the process so required observations are not discarded.
Start a camera configuration when the acquisition interface needs reconsideration. For an existing RTSP application, bring frame-age logs, queue limits, and the inspection deadline to applications support before selecting replacement hardware.
Questions
Why can RTSP images remain delayed with hardware decoding?
Hardware decoding accelerates one part of the path. Images may still wait in source encoding, receiver buffering, memory conversion, or application queues. Compare timestamps for individual frames across those stages before changing hardware. A growing decoded-frame queue usually requires a throughput or buffering-policy change.
Should the receiver jitter buffer be set to zero?
Choose the smallest documented allowance that meets your loss, recovery, and freshness requirements under representative conditions. A zero setting may expose ordinary arrival variation and produce incomplete images. Measure decision age and usable inspection coverage together rather than optimizing only the shortest observed delay.
When is a latest-frame policy appropriate?
Use it when an inspection needs the newest scene state and skipped observations are acceptable. Do not use it as a substitute for sufficient capacity when each object must be inspected. In that case, coordinate acquisition with production or provide enough processing throughput for every required image.
Can a stream timestamp measure capture-to-decision latency?
Not by subtraction alone. Establish what the timestamp represents and how its clock relates to the host clock, including offset, drift, and resets. Without that relationship, report host-side intervals separately and validate total latency using an independently observed visual event and decision output.