Camera pipeline format negotiation: fix startup errors

A camera can appear in the device list yet fail when an application starts acquisition. Detection confirms that a device exists; it does not prove that the requested image format can travel through the capture pipeline. For an embedded inspection prototype or production vision application, the fastest diagnosis is to separate source capture, conversion and application delivery. Establish a working native stream first, then add processing stages individually.

Read the startup error as a boundary failure

A capture pipeline connects several contracts: the camera's output, the driver's buffers, any decoder or converter, and the application's receiving format. A mismatch at any boundary can stop startup. The final error may identify the component that gave up, rather than the component that introduced the mismatch.

Start with unsupported source settings and incompatible receiving formats before changing drivers or replacing hardware. Use this diagnostic order:

Record the first meaningful error and the settings immediately preceding it. Repeatedly changing several settings destroys that evidence.

Identify the acquisition path before editing formats

First establish how the camera is supposed to deliver images. A standard video device, a manufacturer SDK and a GenTL producer are different acquisition paths. A USB3 connection alone does not establish that a camera supports a generic video-device interface.

For a camera exposed through V4L2, inspect the actual device node. The following command lists advertised formats and their associated sizes and frame intervals:

v4l2-ctl --device=/dev/video0 --list-formats-ext

Here, /dev/video0 is an example, not a guaranteed device assignment. Confirm the device identity and capture capability before using it. Some physical devices expose multiple nodes with different functions.

For an SDK-controlled camera, use the manufacturer's documented acquisition application and feature enumeration instead. Obtain software and documentation from the manufacturer or an authorized distributor. Do not assume a generic video utility can access that camera.

The closest companion procedure, diagnosing rejected USB camera modes, covers the source-side negotiation problem. Here, the additional task is proving that every downstream stage accepts the source's actual output.

Prove one complete native mode

Treat a mode as a combination, not three independent choices. A listed resolution does not imply that every pixel format or frame interval is available at that resolution.

1. Close other applications that might own the camera.
2. Select an explicitly advertised pixel format, width, height and frame interval.
3. Disable automatic format conversion in the test path where possible.
4. Apply the request and read back the accepted settings.
5. Capture frames without display, recording, conversion or inference.

For a V4L2 device advertising YUYV at 640 × 480 and 30 fps, the following is an example native-capture test. Use it only when that combination and memory-mapped streaming are supported:

v4l2-ctl --device=/dev/video0 \
  --set-fmt-video=width=640,height=480,pixelformat=YUYV \
  --set-parm=30 \
  --get-fmt-video --get-parm \
  --stream-mmap --stream-count=60 --stream-to=/dev/null

This discards captured data rather than saving images. It tests buffer delivery, not image correctness. At the requested rate, the nominal capture duration is 60 ÷ 30 = 2 seconds, excluding startup overhead.

Compare the readback with the request. A driver may adjust dimensions, substitute a format or accept a different frame interval. Build downstream settings from the accepted values.

If a feature remains unchanged, use the separate procedure for camera pixel formats that cannot be changed. A successful property-setting call is not sufficient verification.

Budget the data crossing each boundary

Negotiation specifies what a buffer means; bandwidth determines whether the selected path can sustain it. Calculate both native and converted payloads, because conversion can increase memory traffic even when the camera stream is unchanged.

For an unpacked, uncompressed image, the minimum payload estimate is:

For the hypothetical YUYV stream above, packed image data uses 2 bytes per pixel on average:

• Frame payload: 640 × 480 × 2 = 614,400 bytes.
• Native payload rate: 614,400 × 30 = 18,432,000 bytes/s, approximately 18.4 MB/s.
• Converted three-channel, 8-bit output: 640 × 480 × 3 × 30 = 27,648,000 bytes/s, approximately 27.6 MB/s.

These are image-payload estimates, not interface capacity requirements. Row padding, transport overhead, buffer copies and intermediate images add traffic. A converter must also read its input and write its output.

Compressed streams require a different calculation based on measured encoded payloads. Their buffer sizes can vary with image content and compression settings.

If startup succeeds but sustained delivery fails, follow finding the bottleneck when a camera misses its frame rate. Do not mislabel a growing processing backlog as an initial format-negotiation failure.

Add decoding and conversion one stage at a time

Once native capture works, write down the format at every boundary. Include dimensions, pixel representation, plane arrangement and row stride. Where relevant, record colorimetry and whether buffers reside in ordinary system memory or another memory domain.

A useful diagnostic chain is:

confirmed source mode
    -> decoder, only if the source is compressed
    -> pixel-format converter, if required
    -> application-compatible receiving buffer
    -> inspection algorithm

A decoder and a color converter solve different problems. A compressed image cannot generally be treated as a rectangular array of raw pixels. The guide to encoded image bytes versus pixel data explains that distinction.

Likewise, selecting a different format name does not necessarily convert the buffer. Some pipeline stages only constrain or describe a stream. Actual conversion requires a component that supports the input format, output format and memory representation.

For each added stage:

1. Keep the working source mode unchanged.
2. Specify only the output constraints that the next stage requires.
3. Confirm the negotiated input and output descriptions.
4. Acquire a short sequence and inspect representative frames.
5. Save the working configuration before adding another stage.

Match the receiving buffer before running the algorithm

The application's input contract includes more than width and height. Check channel count, element depth, channel order, row stride and buffer lifetime.

For example, three-channel RGB and BGR images occupy similar amounts of memory but use different channel meanings. Packed YUYV is not interchangeable with either. A grayscale image also cannot be assumed to contain three independent color channels.

Use a small capture-only application that reports:

• Accepted image width and height.
• Pixel format, channel count and element depth.
• Row stride and delivered buffer size.
• Frame identifiers and timestamps, when available.
• Capture success or failure before processing begins.

Do not reshape a buffer merely because its byte count appears plausible. Padding can make row stride larger than width multiplied by bytes per pixel. Multiplanar formats require additional handling.

For doubled, striped or incorrectly interpreted color images, follow matching array types to the delivered pixel format.

Also check buffer ownership. Some acquisition interfaces reuse memory after a frame is released. An asynchronous processing worker must retain the buffer through the documented mechanism or copy the required pixels before reuse.

A successful preview does not prove application compatibility. The preview may contain a decoder, converter or memory transfer that your application has not included.

Choose hardware by its documented software path

Replacing a camera is not the first remedy for a downstream conversion error. Before ordering, confirm that the proposed camera's documented acquisition path works with your operating system, processor architecture and application interface.

Use area scan cameras when comparing camera options, and board-level cameras when integration requires a board-level form. The shared camera finder helps narrow catalog choices, but interface filtering alone does not establish software compatibility.

The LS-CMR-0166 is a USB3 area scan camera with a rolling shutter. Treat it as a catalog candidate, not evidence that a particular generic capture backend is supported.

The LS-BCM-0003 is a USB3 board-level camera with a global shutter. Its catalog interface does not establish generic video-device compatibility either.

For either candidate, request confirmation of the required acquisition API and supported software environment. Also confirm available pixel formats, trigger behavior and delivery of metadata needed by the application.

The board-level camera selection guide addresses integration decisions beyond pipeline startup. Mechanical suitability and shutter choice remain separate from negotiating a valid stream.

Verify the repair and recognize the exceptions

Use a staged acceptance test rather than a single successful frame. For a hypothetical 30 fps stream, 300 delivered frames represent approximately 10 seconds of acquisition. This is a useful initial check, not a production reliability qualification.

Run the same mode through native capture, converted capture and the complete application. At each stage, compare delivered frame counts, timestamps, errors and image interpretation. Extend testing to the intended operating duration and triggering pattern before deployment.

The usual format-first diagnosis is wrong when native capture already fails because of device access, disconnection or an unsupported acquisition path. It is also incomplete when images arrive correctly but the inspection algorithm rejects them. Exposure, focus and algorithm requirements belong to later validation.

Before accepting the repair, check that:

• The intended physical camera is selected reliably.
• The complete source mode is advertised and its accepted values are recorded.
• Native capture succeeds without hidden conversion.
• Every decoder and converter has a verified input and output contract.
• Application buffers have correct layout and ownership.
• Restarting the application reproduces the working configuration.
• Full processing sustains the required delivery rate without an expanding queue.

Keep the minimal working capture configuration alongside the production configuration. It provides a repeatable baseline when software or hardware changes.

For a new system, start with the camera configurator. For an existing startup failure, send the advertised modes, accepted readback and first failing pipeline boundary through technical support before selecting replacement hardware.