Undistortion pipeline crash when aligning depth to color

A vision pipeline can stream color and depth correctly until color undistortion and depth-to-color alignment run together. Adding inference may expose the failure sooner, but that does not prove the model is responsible. The fault may sit in shared preprocessing, buffer ownership, or a software-version interaction.

This troubleshooting procedure is for engineers isolating that combination on an embedded vision device. You will reduce the processing graph, identify the smallest failing configuration, and build a reproducible report. You will also distinguish a manufacturer-supported correction from a graph change that merely hides the symptom.

1. Make the processing interaction the first test

When each operation works separately but their combination crashes, investigate the interaction before replacing calibration files or retraining a model. This is a diagnostic priority, not proof of a particular internal defect.

Undistortion remaps image pixels. Depth alignment maps depth samples into another camera's image coordinates. Inference may introduce additional resizing, format conversion, buffering, or scheduling. These operations can share resources even when your application presents them as separate graph nodes.

Start with three questions:

• Does color undistortion work without depth alignment?
• Does depth alignment work without explicitly enabling color undistortion?
• Does their combination fail with inference completely removed?

If the third test fails, model weights and inference postprocessing are not necessary to reproduce the fault. Keep them out of the investigation until acquisition and geometry processing work together.

The real-time image undistortion guide explains the processing choices behind remapping. Here, the objective is narrower: identify which combination changes a working stream into a failing runtime.

2. Freeze the environment and capture the failure

Before updating software, save the failing configuration. Otherwise, you may lose the only reproducible case while changing several dependencies together.

Record the following in a small test manifest:

• Device identity and hardware revision, where available.
• Host operating system and architecture.
• Application build identifier and build settings.
• Runtime, language binding, driver, and device firmware versions.
• Camera streams, requested dimensions, actual output dimensions, and frame rates.
• Pixel formats, row strides, and image layouts.
• Undistortion source, alignment target, and calibration identifier.
• Queue limits, blocking behavior, and buffer ownership rules.
• Time from startup to failure and the last completed processing stage.

Save complete startup logs and the first fatal message, including its subsystem and diagnostic code. A later host exception may only report that the device connection disappeared after the original failure.

Repeat the original case three times without changing settings. Note whether it fails during graph startup, on the first frame, or after sustained streaming. A repeatable startup failure suggests a different test path from memory growth over several minutes, although neither timing pattern proves its cause.

Compare requested settings with delivered frame metadata. A stream resized upstream may no longer match assumptions made by an alignment or inference stage.

If component versions are inconsistent, use the checks in camera firmware and SDK alignment. Do not assume that matching version labels alone establish compatibility; consult the manufacturer's documented compatibility information.

3. Build a minimal graph and run the isolation matrix

Use the manufacturer's smallest supported acquisition example as your starting point. Obtain examples, runtimes, and firmware only from the manufacturer or its authorized distributors.

Keep one device, one application, and only the streams required for the experiment. Disable recording, visualization effects, extra consumers, and automatic restart while collecting the first failure. Maintain the documented stream connections needed by the device.

Run the following cases with identical image dimensions and queue settings wherever the supported API permits them:

“Off” means the documented control is disabled. Some devices still perform internal rectification or other geometry processing required for depth generation. Do not interpret a disabled application setting as proof that every internal warp is absent.

1. Run Test A and confirm continuous frame delivery.
2. Run Test B and inspect the delivered color output.
3. Run Test C and verify that aligned depth is actually produced.
4. Run Test D before adding any inference stage.
5. Run Test E only after Test D passes reliably.

For an initial screen, use a fixed observation window, such as 60 seconds per case, and repeat each case three times. If the original failure takes longer, extend the window beyond that observed failure time. These are screening intervals, not reliability guarantees.

If only Test D fails, submit that graph as the primary reproducer. If D passes but E fails, add inference preprocessing and inference execution separately where the documented API supports that separation.

4. Test the likely causes without changing everything

Use the matrix result to choose the next experiment. The order below prioritizes tests that remove large parts of the graph with small configuration changes.

Shared preprocessing or runtime interaction

Symptom: Both geometry operations pass separately, but their combination fails even without inference.

Test: Reproduce the failure in a supported minimal example. Repeat with a documented, compatible software stack recommended by the manufacturer.

Action: Report the smallest failing graph and request confirmation of the supported correction. Do not infer an internal allocator defect solely because an unrelated graph edit changes the outcome.

Unsupported geometry or output combination

Symptom: Failure depends on a particular output size, alignment target, or explicit remapping configuration.

Test: Compare every relevant setting with the documented combinations. Try a documented baseline configuration, changing only one setting per run.

Action: Use a supported combination while the manufacturer reviews the reproducer. A smaller image that works does not establish that the original dimensions are unsupported.

Buffer ownership or asynchronous access

Symptom: Device processing continues, but the application crashes after passing images to another thread or consumer.

Test: Remove asynchronous consumers. Process frames synchronously, or make an application-owned copy using the documented API before releasing the source frame.

Action: Correct ownership and disposal order. The frame memory lifetime guide covers access to buffers that outlive their original frame object.

Queue pressure or memory exhaustion

Symptom: Failure appears after queues fill, or changes substantially when consumers slow down.

Test: Log queue occupancy, frame age, and available memory where supported. Reduce queue depth and stream rate independently.

Action: Bound buffering and define intentional backpressure or frame-dropping behavior. Retest the original geometry combination rather than treating reduced throughput as a confirmed software repair.

Incorrect coordinate assumptions

Symptom: The pipeline runs, but color edges and depth boundaries disagree.

Test: Inspect calibration metadata, output intrinsics, image dimensions, and the coordinate frame associated with the aligned output.

Action: Follow the alignment metadata checks. Calibration problems can explain incorrect geometry, but they should not be assumed to explain a runtime crash without evidence.

5. Estimate buffering before blaming the model

A memory estimate helps explain why adding a consumer can expose a previously hidden limit. It does not prove that memory caused the crash.

For tightly packed, uncompressed images, the payload estimate is:

Consider a hypothetical stream with 1280 × 720 color pixels, three bytes per color pixel, and two bytes per depth pixel. Assume depth has already been aligned to the same dimensions.

• Color: 1280 × 720 × 3 = 2,764,800 bytes.
• Depth: 1280 × 720 × 2 = 1,843,200 bytes.
• One color-depth pair: 2,764,800 + 1,843,200 = 4,608,000 bytes.
• Four queued pairs: 4 × 4,608,000 = 18,432,000 bytes, approximately 17.6 MiB.

That estimate excludes padding, source depth images, remapping tables, intermediate images, inference tensors, and runtime allocations. Multiple consumers may also retain different frames simultaneously.

Use reported row stride rather than width times bytes per pixel when images contain row padding. For planar color, account for every plane and its actual layout.

Log timestamps alongside memory measurements. A growing age difference between acquisition and consumption may reveal backlog before memory exhaustion becomes visible. The asynchronous frame-matching guide explains how to preserve associations when processing stages finish at different times.

6. Validate a supported correction, not a graph accident

A useful manufacturer report contains the minimal reproducer, environment manifest, isolation matrix, and complete first-failure log. Include exact reproduction steps and the expected behavior. State which combinations pass rather than reporting only that the full application crashes.

Ask for the documented release or supported package containing the correction. If a development build is proposed, confirm its intended test scope and production support status before deployment. Do not treat an informal statement that an issue is fixed as verification on your device and graph.

Preserve a rollback package for the previously qualified configuration. Then change the software stack as a controlled experiment, keeping stream settings and graph structure unchanged.

Do not make an unused node, disconnected processing stage, or undocumented ordering change the production repair. Such edits can alter resource assignment or timing without correcting the underlying fault.

If processing must temporarily move to the host, first confirm that the manufacturer supports access to the required raw streams and calibration data. The offline rectification guide provides a way to investigate geometry independently, but offline success does not validate the embedded runtime.

A temporary configuration without alignment is acceptable only when the application genuinely does not need aligned coordinates. Never silently substitute unaligned depth for aligned output.

7. Verify geometry, endurance, and recovery

Rerun every case in the isolation matrix after applying the proposed correction. A fix for the combined geometry case must not break the independent streams or inference path.

Validate more than survival:

• Confirm delivered dimensions, pixel formats, intrinsics, and timestamps.
• Check several stationary objects at different depths and image locations.
• Compare color edges with depth boundaries, allowing for occlusion and invalid depth.
• Test the original stream rate, queue depth, and inference workload.
• Restart the application and device repeatedly.
• Exercise documented disconnection and recovery behavior.
• Check for persistent memory growth and increasing frame age.

Use a visible depth discontinuity rather than relying only on a flat wall. A flat surface can look plausible even when pixel correspondence is wrong. Check the center and corners, because an incorrect warp may be more apparent away from the optical center.

For regression screening, a 30-minute run at 30 frames per second represents 30 × 60 × 30 = 54,000 frame intervals. Record actual delivered frames, dropped frames, and restarts rather than assuming the requested rate was maintained.

Choose final acceptance limits from your application requirements. A screening run is not proof of production reliability, and visual agreement alone is not a metrology qualification.

8. Close the issue before changing hardware

Do not replace the sensor merely because a software interaction is difficult to isolate. First establish whether the manufacturer supports the required combination of undistortion, alignment, and inference.

If a separate architecture review becomes necessary, start with the 3D sensors category and camera finder. Select the sensing method and supported output interface before evaluating processing hardware.

The MV-DPS200P-02 is identified in the catalog as a projected structured-light camera. Its catalog listing does not establish support for the processing graph investigated here.

For a separate processing architecture, review the vision controllers category. The MV-SVC2506-16G60-128G is a vision-controller candidate with camera GigE ports. Confirm acquisition compatibility and software requirements before selecting it.

These catalog entries are not a verified camera-controller pair, nor evidence that either reproduces or resolves the fault. Compatibility must come from documented interfaces and application validation.

Before closing the issue, check that you have:

• A saved minimal failing graph and its passing counterpart.
• A documented software correction with a rollback path.
• Passing geometry, inference, endurance, and restart tests.
• Defined behavior for missing frames and invalid depth.
• A regression test that can run after future updates.

For help reviewing an alternative acquisition architecture, send the stream requirements and compatibility information through applications support. Keep the reproducible failure case as the reference for deciding whether a software correction or hardware change is justified.