Robot training data collection: camera setup and timing

A robot can generate hours of video without producing a trustworthy training example. Images become useful when you can connect them to the robot's physical state, the active calibration and a verified task outcome. This guide is for teams adding data capture to an operating vision cell. Start with a bounded, read-only collection pipeline, then prove image quality, timing and traceability before expanding coverage or training models.

Define the training example before choosing hardware

Decide what the future model must predict. Object detection needs image annotations; grasp assessment needs observations around contact and a verified outcome. Learning motion behavior may require sequences containing observations, issued commands and measured states. A commanded position is not evidence that the robot reached it.

Define an episode as one task attempt, with explicit start and end events. Give every episode a unique identifier, including failed or interrupted attempts. Keep capture faults separate from task failures: an incomplete recording cannot establish what happened physically.

A practical record contains these linked elements:

Retain the original observation separately from annotations and derived coordinates. That lets you revise labels without silently changing the evidence. For task transitions and request sequencing, the guide to vision picking request timing explains why software events and physical motion need separate records.

Choose the camera around motion and integration

Start with area scan cameras when a packaged camera suits the installation. Consider board-level cameras when the assembly genuinely requires a compact module and you can handle the additional integration work. Neither package style proves that a camera supports your required trigger, timestamp or software interface.

The LS-BCM-0003 is a USB3 board-level camera with a global shutter. Treat it as a candidate for motion-sensitive capture, subject to verification of the required acquisition features.

The LS-CMR-0166 uses a USB3 interface and a rolling shutter. Consider that shutter behavior explicitly when the scene contains moving parts or the camera moves with the robot.

Use the camera finder to narrow interface, shutter and package choices. The board-level camera selection guide covers the additional decisions involved in integrating a module.

Do not infer HDR operation, synchronization protocols, trigger support or timestamp precision from a camera's interface or shutter type. Confirm those features in the manufacturer's documentation before committing the acquisition architecture.

Calculate sampling and exposure from the task

Choose spatial sampling from the smallest feature that affects the training label, not from a desired video resolution. As an illustrative starting assumption, suppose a feature should span at least five pixels across its width. That is a planning target, not a guarantee of detection accuracy.

For a field width of 240 mm and a relevant feature width of 0.5 mm:

Use the resolution calculator to explore different feature sizes and sampling targets. Then verify actual contrast, focus and annotation consistency with representative images.

Motion sets a separate exposure constraint. Estimate blur using the relative image-plane motion between the camera and the feature. For a moving camera, conveyor speed alone may not describe that motion.

This estimate assumes approximately constant lateral motion during exposure. Rotation, perspective and changing depth can produce different blur across the image. Inspect the fastest and nearest visible features rather than checking only the image center.

Establish a stable lighting baseline first. Add approved lighting variation later, recording exposure, gain and lighting configuration where available. Automatic adjustment can make successive images look consistent while concealing changing acquisition conditions. The guide to verifying per-frame exposure and gain explains how to associate settings with individual images.

Put camera and robot timestamps on a verified timeline

A timestamp is useful only when you know which event it describes. Camera metadata might refer to an exposure event or another acquisition event. A host timestamp typically describes a software observation. Robot telemetry may describe measurement time, publication time or receipt time.

Document those meanings before pairing records. Also identify each clock, its resolution, any offset correction and how drift is measured. Never subtract unrelated clock values simply because both are expressed in milliseconds.

A position-error budget gives timing requirements a physical basis. For approximately constant relative speed:

This budget covers only timing-related position uncertainty. Calibration, telemetry accuracy and mechanical variation consume additional error allowance.

Suppose images arrive at 60 frames/s while robot states are measured at 100 Hz. The image interval is about 16.7 ms; the state interval is 10 ms. With perfectly aligned clocks, regular sampling and no missing states, nearest-sample pairing can still leave a 5 ms mismatch. At 800 mm/s, that corresponds to 4 mm of travel.

Interpolation between bracketing measurements may reduce sampling error during smooth motion. It does not remove clock offset, unknown telemetry delay or errors during abrupt motion. Preserve both bracketing records and mark missing intervals rather than extrapolating silently.

Where documented hardware synchronization or clock synchronization is available, verify its behavior across the complete system. The guide to multi-camera exposure synchronization distinguishes matching frame delivery from matching exposure events.

Keep collection outside the production control path

Use a bounded acquisition queue and a separate writer or processing worker. The acquisition callback should hand off a frame reference or a deliberate copy, record metadata and return promptly. Image compression, annotation and storage management belong outside time-sensitive acquisition work.

Choose an explicit overflow policy. A diagnostic collector might discard an entire incomplete episode and record the reason. Another installation might suspend collection until storage recovers. Neither approach should silently block the robot's control loop or change its safety behavior.

Vision controllers are candidates when a dedicated vision box fits the cell architecture. The MV-VC2140-128G40-1T is a vision controller with built-in light outputs. Its suitability still depends on documented software support and measured acquisition, processing and storage performance.

Keep telemetry collection read-only through supported interfaces. Rate-limit subscriptions or polling according to the controller documentation. Do not add collection tasks to a motion program without reviewing their execution cost and operational effect.

If a general-purpose host is more appropriate, compare industrial mini PCs against the measured workload. A processor specification alone does not establish sustained recording performance.

Record collector health independently: queue depth, missing frames, state gaps, write latency and clock-correction status. The guide to high speed camera recording without dropped frames covers the underlying acquisition and writer bottlenecks in more detail.

Budget storage using actual payloads

Calculate uncompressed payload first, then measure the complete recording format. Compression ratios depend on scene content and encoding choices; they are not a safe fixed assumption.

For an illustrative image stream containing 2400 × 1600 pixels, one byte per pixel and 60 frames/s:

These are decimal storage units. The calculation excludes metadata, container overhead, temporary files and duplicate copies. Validate disk capacity and sustained writing separately; sufficient free capacity does not prove adequate throughput.

Event windows can reduce retained data without weakening every example. A five-second window containing two seconds before an event and three seconds after it requires about 1.152 GB at this payload rate. Retaining 120 non-overlapping windows requires approximately 138 GB before overhead.

A pre-event buffer still requires continuous acquisition and memory handling. Event selection reduces retained storage, not necessarily incoming bandwidth. If windows overlap, define whether shared frames are stored once or copied into each episode.

When intensity values matter, preserve the original pixel representation or document every conversion. The guide to saving camera images without losing intensity data explains why a visually acceptable export may not preserve the original measurements.

Select useful variation instead of more repeated video

Begin with a coverage matrix: part family, orientation, approach direction, speed band, lighting condition, occlusion and outcome. Add conditions only when they could affect the intended model. Thousands of nearly identical successful cycles can hide a complete absence of useful failure examples.

Retain a baseline sample of ordinary cycles alongside unusual episodes. Useful selection signals include low perception confidence, disagreement between inspection stages, operator correction and independently measured task failure. A low-confidence prediction is a reason to review an example, not proof of its correct label.

Separate these outcome classes:

• Confirmed task success, supported by an independent observation.
• Confirmed task failure, with the relevant evidence retained.
• Interrupted or incomplete attempt.
• Unknown outcome requiring review.
• Capture fault that prevents reliable interpretation.

Lighting variation should represent the conditions the deployed system must handle. Change one factor at a time during approved trials so you can identify its effect. Keep a stable reference condition between trials to detect focus shifts, exposure changes or calibration drift.

Do not manufacture hazardous failures to balance a dataset. Use approved test procedures, simulation or offline replay where appropriate. Protect confidential scene content and restrict access to recordings according to the site's data-handling requirements.

Verify traceability before releasing a dataset

Run a short acceptance collection before expanding to unattended recording. A useful first target is 100 consecutive episodes across representative operating conditions. This is an integration check, not statistical proof of model performance.

1. Reconstruct each episode. Confirm that images, robot states, configuration and outcome records join through explicit identifiers.
2. Check continuity. Count missing images, missing telemetry intervals and incomplete writes rather than judging playback appearance.
3. Measure timing. Use an approved observable event or documented timing signals to assess offset and variation across operating load.
4. Inspect image quality. Review fast motion, field edges, occlusion and lighting transitions at original resolution.
5. Verify coordinate meaning. Confirm units, frame names, tool definition and the calibration active during capture.
6. Exercise collector faults. Test queue saturation and writer interruption in an approved setting, confirming that collection faults do not alter control behavior.
7. Replay independently. Reconstruct a sample on a separate analysis session without relying on undocumented live settings.

For geometry changes, follow vision calibration recovery after camera or robot changes. A calibration file remaining readable does not establish that it still describes the physical installation.

Split training, validation and test data by meaningful groups such as episode, collection session or part batch. Randomly splitting adjacent frames can put nearly identical views on both sides of the evaluation. Keep the held-out test set fixed while refining selection and annotation procedures.

Dataset approval and model deployment are separate gates. Training on new recordings must not automatically replace the production model or change robot behavior.

Check before ordering and know when to change direction

Before committing hardware, complete this short checklist:

• Confirm the smallest relevant feature and the required field coverage.
• Measure worst-case relative motion and test the proposed exposure.
• Verify documented timestamp meaning, trigger behavior and telemetry access.
• Check camera software compatibility using the intended operating environment.
• Benchmark sustained acquisition and writing with the actual pixel format.
• Define calibration versioning, episode identifiers and independent outcome evidence.
• Approve collection fault behavior and a separate model-release process.

The usual answer is not always another camera or a higher frame rate. If failure happens after visual inspection, a downstream verification signal may provide better labels. If the scene is stationary during a reliable dwell, selected still images may be more useful than continuous video. If state timing is ambiguous, increasing image frequency does not repair the pairing.

Start a component shortlist with the camera configurator. Bring representative images, motion estimates and timing requirements to applications support before ordering; those details determine whether the proposed capture pipeline can produce usable, traceable examples.