Recorded video replay for machine vision pipelines

Recorded footage lets you debug image processing without keeping a camera connected or repeating a production run. The challenge is making decoded frames behave like a predictable input stream, rather than merely opening a video player. This procedure is for developers adapting an existing camera pipeline to recorded sequences. You will define the frame contract, preserve presentation order, control playback timing and validate stereo inputs. For individual test images rather than continuous recordings, start with practicing machine vision with offline images.

Decide where recorded frames enter the pipeline

Connect replay at the earliest documented input that accepts host-supplied images. This might be an application callback, an input queue or a processing function that normally receives camera buffers. Use documented interfaces rather than modifying protected components or assuming that every camera-facing node accepts arbitrary frames.

An MP4 file is a container, not an image format. A decoder converts its encoded video stream into image buffers. Your replay adapter then supplies those buffers, with their associated timing and identification data, to the processing pipeline.

There are three practical integration choices:

Start after acquisition when you only need to test an algorithm. Use a supported replay facility when stream relationships and device metadata matter. Check camera acquisition software compatibility before assuming that an existing acquisition interface also accepts files.

Establish the recording and frame contract

Before writing the adapter, record what the receiving pipeline expects. A successful decode does not guarantee a compatible input image.

Define these fields explicitly:

• Image width and height, including any crop or rotation.
• Pixel type, bit depth, channel order and memory layout.
• Row stride, which may include padding beyond the visible pixels.
• Frame identifier and presentation timestamp.
• Buffer ownership and the point when memory may be reused.
• End-of-file behavior, including whether looping is permitted.

A decoder might return packed color images while the pipeline expects a single grayscale plane. Convert deliberately and verify the result with a saved reference frame. Do not treat one channel of a color image as an interchangeable substitute for grayscale.

Compression, chroma subsampling and prior image adjustments can change edges and fine texture. If the goal is defect measurement, retain representative original images where possible. A replay result is only as representative as the recording supplied to it.

Feed decoded frames in six steps

1. Open the intended video stream and inspect it. Confirm that the file opens, contains video and produces a valid first frame. Record the decoded dimensions and available timestamps. Treat a reported frame count or nominal frame rate as useful metadata, not proof that the entire file is readable. For opening failures, follow the checks in diagnosing video capture backend fallback.

2. Decode sequentially in presentation order. Let the decoder handle compression dependencies and frame reordering. Encoded packet order is not necessarily display order. For a baseline test, read continuously from the beginning rather than repeatedly seeking to calculated positions. Seeking can land near a keyframe and require additional decoding before the requested image becomes available.

3. Normalize each frame to the agreed contract. Apply the required color conversion, layout conversion and explicit resize or crop. Reject unexpected dimensions instead of silently changing them. If calibration, masks or measurement regions depend on image coordinates, update them consistently or keep the original image geometry. Save a few normalized frames for visual inspection before connecting the complete pipeline.

4. Attach stable identifiers and timestamps. Give every emitted image a sequential identifier and preserve its source presentation timestamp when available. Use relative timestamps if the recording contains no trustworthy absolute capture time. If you generate timestamps from an assumed constant rate, label them as synthetic. They describe the replay schedule, not independently measured camera timing.

5. Submit through a bounded queue with clear ownership. Confirm whether the receiving API copies data or retains a reference. Do not overwrite a buffer while downstream processing still uses it. Set a queue capacity and choose what happens when it fills: block the producer, discard according to a declared policy or stop with an error. Avoid an unlimited queue that hides a processing bottleneck.

6. Handle completion and restart explicitly. At the end of the recording, stop submitting new images and drain outstanding work before reporting completion. Distinguish decoder errors from a normal end of stream. For a repeated run, reset algorithm state and frame identifiers, or introduce an explicit replay-run identifier. Do not let an automatic loop silently join the final scene to the opening scene.

Keep decoding, scheduling and processing as separate responsibilities. This makes it possible to change playback speed without changing pixel conversion, or replace a decoder without changing inspection logic.

Calculate pacing and memory before running long tests

Choose a replay mode according to the question being tested. An exhaustive regression run normally processes every frame as fast as downstream work allows. A timing-sensitive demonstration needs source-paced delivery. A live-like overload test may deliberately discard late frames, but that policy must be visible in the results.

For source-paced replay, schedule each image against a monotonic clock using its presentation timestamp. Do not add a full frame-period sleep after processing each image.

At normal speed, playback speed equals 1. At double speed, it equals 2. For a constant-rate example, 30 frames/s gives a frame period of 1000 / 30 = 33.3 ms. If processing takes 20 ms, the remaining nominal interval is 33.3 - 20 = 13.3 ms. Sleeping another 33.3 ms would extend the cycle to about 53.3 ms, reducing delivery to approximately 18.8 frames/s.

Variable-rate files need their actual presentation intervals. A single nominal rate cannot reproduce irregular timing. If the decoder does not expose usable timestamps, use a declared constant-rate approximation rather than claiming original timing fidelity.

Estimate queue memory separately:

For a hypothetical 1920 × 1080 image with 3 channels and 1 byte per channel, one packed frame requires 6,220,800 bytes. Eight queued frames require 49,766,400 bytes, approximately 49.8 MB using decimal units. Padding, decoder buffers and intermediate images add further memory.

If the original recording already contains missing frames, replay cannot restore them. Review recording without dropped frames when capture completeness is uncertain.

Treat stereo replay as a paired recording problem

An ordinary single-view video is suitable for many detection and tracking tasks. It is not a substitute for synchronized left and right observations in a calibrated stereo pipeline.

Stereo reconstruction needs corresponding views of the scene, known camera geometry and calibration matching the recorded image dimensions. If the scene moves, mismatched exposure times can create false correspondence even when both recordings appear individually smooth.

Use a recording format that preserves paired frame identifiers or a shared capture-time reference. Two files labeled with the same nominal frame rate are not sufficient evidence of synchronization. Independently assigning the same replay timestamp to unrelated frames does not make their original exposures simultaneous.

Preserve these relationships throughout replay:

• Left and right stream identity and ordering.
• Original pairing information or documented synchronization metadata.
• Intrinsic parameters and distortion models for each view.
• Relative camera pose and the applicable rectification transforms.
• Image geometry, including any crop, resize or rotation applied during recording.

As an illustrative calculation, an object moving laterally at 0.5 m/s travels 0.5 × 0.010 = 0.005 m, or 5 mm, during a 10 ms timing mismatch. The resulting image error depends on scene geometry, but the two frames no longer describe exactly the same moving scene.

Use stereo calibration from paired checkerboard images to establish geometry, then follow live stereo rectification checks when validating corresponding image rows. If calibration changes software formats, check importing stereo calibration parameters rather than copying arrays without checking their conventions.

Verify ordering, pixels and timing independently

Begin with a short sequence that includes visible motion and an identifiable beginning and end. Use a 100-frame segment for an initial exhaustive test. If every decoded frame is submitted exactly once, expect 100 submissions and 100 completed results after draining the pipeline.

Record separate counters for decoded, submitted, completed and discarded frames. A single displayed frame counter cannot reveal where losses occurred.

For stateful algorithms, reset trackers, accumulated statistics and background models between independent tests. If randomness is involved, record the relevant seed or compare results using an appropriate tolerance.

Measure wall-clock delivery timing separately from source timestamps. Correct metadata does not prove that frames reached the consumer on schedule. Conversely, slow exhaustive processing can still produce valid per-frame inspection results when real-time delivery is not the test objective.

Choose capture hardware only when the recording requires it

You do not need a new camera to process an existing compatible recording. Hardware selection becomes relevant when you need more representative footage, controlled exposure or synchronized source images.

For new single-view recordings, compare area scan cameras through the camera finder. Confirm that the chosen acquisition software can save the required image data and metadata. The LS-CMR-0053 is a GigE capture option to evaluate against those requirements.

The LS-CMR-0166 is a USB3 capture option. Neither selection establishes file-replay compatibility; that depends on the software and recording workflow.

For depth acquisition, compare 3D sensors, but confirm whether the intended recording preserves raw views, processed depth, calibration or some combination. These outputs are not interchangeable.

If replay will run on vision controllers, verify decoder availability and benchmark the actual processing chain. Size the system from measured decode, conversion and processing times, not from camera interface bandwidth alone.

Final checklist and next step

Before accepting the replay adapter, confirm the following:

• The selected input interface officially accepts host-supplied frames.
• Pixel format, dimensions, stride and ownership match the receiver.
• Presentation order survives decoding and asynchronous processing.
• Source timestamps remain distinguishable from delivery timestamps.
• Queue capacity and late-frame handling are explicit.
• End-of-file drains outstanding work and reports completion.
• Stereo recordings retain valid pairing and matching calibration.
• Test reports identify which live-acquisition functions were not exercised.

Start with exhaustive processing, then add source-paced delivery once image correctness and ordering are established. Keep overload simulation as a separate test mode so dropped frames cannot be mistaken for missing inspection results.

For a new capture setup, use the camera configurator. If you need help matching a recording workflow to acquisition hardware, send the frame contract and timing requirements through applications support.