Depth camera object detection: connect frames to inference

A depth camera can work in its own viewer yet fail when an object detector tries to open webcam index 0. That index belongs to a particular video backend; it does not identify every camera connected to the computer. This procedure is for connecting an already detected camera to an inference application. You will verify acquisition through the supported SDK, pass a correctly formatted color image to the detector, and preserve the coordinates needed for overlays or later distance measurements.

Choose the acquisition path before changing the detector

Start with the camera's supported acquisition interface. A depth device may expose several streams through its SDK without exposing a conventional webcam endpoint. Some devices expose both interfaces, but their available formats and controls can differ.

An index such as 0 is not a persistent device identity. Another connected camera or a different backend can change which device it selects. Prefer a documented device identifier when the SDK provides one.

For ordinary color-image object detection, first acquire color only. Depth is not required to draw a detection box or assign a class label. A model trained on color photographs should not receive a depth visualization unless it was specifically trained for that representation.

If the SDK itself cannot find the device, follow camera driver and package detection checks before debugging inference.

Prepare a known-good acquisition baseline

Before writing the bridge, establish what the camera actually delivers. Keep the detector disabled during this stage.

1. Install the supported SDK and language bindings from the manufacturer or its authorized distributor. Follow the documented compatibility requirements for your operating system, runtime, and firmware.
2. Open the official viewer and identify the connected device. Verify that a color stream exists rather than assuming that every depth camera includes one.
3. Select a documented color profile and record its width, height, frame rate, and pixel format. Use a profile that already works in the viewer.
4. Close the viewer before starting your application. Multiple clients can conflict if the device or SDK does not support shared acquisition.
5. Run the SDK's simplest color acquisition example, without loading the detector.

Record the exact startup error if acquisition fails. A missing device, unavailable profile, occupied interface, and timeout are different faults requiring different fixes.

If individual streams work but a requested combination fails, use the camera pipeline format negotiation checks. Do not keep changing detector settings to compensate for an acquisition failure.

Open one SDK pipeline and inspect its frames

The acquisition layer should own the device and expose frames to the rest of the application. Avoid letting the detector's default video loader open a second connection behind it.

6. Open the selected device through the SDK and enable the verified color profile. Obtain the actual active profile after startup, since requested and negotiated settings may differ.
7. Wait for a valid frame using the documented timeout behavior. Check frame validity before accessing its data.
8. Log the frame dimensions, pixel format, row stride, timestamp, and sequence identifier where available. Confirm that timestamps and frame identifiers advance.
9. Display the image directly, without resizing, normalization, detection, or overlays. Save several frames containing both stationary detail and motion.

A useful initial test is a run of 300 frames. Treat this as a debugging sample, not proof of production reliability. Count received frames and record timeouts rather than silently skipping every failure.

Your log should distinguish acquisition time from processing time. Camera timestamps may use a different clock from the host, so do not subtract them from host timestamps without a documented clock relationship.

When you later add recording or multiple consumers, retain a single acquisition owner. The camera stream fan-out design explains how to distribute frames without independently opening the device for each task.

Convert the frame to the detector's input contract

A successful SDK frame is not automatically a valid detector input. Write down the detector's expected input contract before implementing conversions.

Check these properties explicitly:

• Channels: Does the model expect color, grayscale, or another representation?
• Channel order: Is the incoming image RGB, BGR, or a packed format requiring conversion?
• Data type: Does preprocessing expect unsigned bytes or floating-point values?
• Numeric range: Are values expected in 0–255, 0–1, or a documented normalized range?
• Layout: Does the inference interface accept height-width-channel arrays or channel-height-width tensors?
• Shape: Is the input size fixed, dynamic, or restricted to particular dimensions?
• Ownership: Will the image memory remain valid until inference finishes?

10. Convert the SDK frame into an array using its documented dimensions, format, and stride. Do not assume that every row contains only tightly packed image pixels.
11. Apply only the transformations missing from the detector's existing preprocessing path. Check that resizing, channel swapping, scaling, and normalization are not being applied twice.
12. Pass the resulting array or tensor through the detector's documented image-input interface. Replace its webcam loader rather than disguising an SDK frame as a device index.

For packed unsigned-byte color, the expected image storage is approximately:

This calculation excludes row padding and metadata. A different bit depth or packed representation changes the calculation. Read the format metadata rather than inferring the layout from a buffer length alone.

Incorrect channel counts or element types can produce distorted images even when acquisition is healthy. The pixel format and array type mismatch checks help diagnose that boundary.

Use the camera frame memory lifetime guide when crashes, corrupted images, or intermittent results appear after queueing frames.

Preserve coordinates through resizing and padding

13. Test the detector first with the saved color image. Then feed a live frame through exactly the same preprocessing path and compare the displayed result.
14. Record the resize scale, padding, and any crop applied to each image. Use those values to map detections back to the original color frame.

Consider an illustrative source image of 1280 × 720 pixels and a square model input of 640 × 640 pixels. If preprocessing preserves aspect ratio and centers the image, the transformation is:

A model-space point at (400, 230) therefore maps to (800, 180) in the source image: 400 / 0.5 = 800 and (230 - 140) / 0.5 = 180.

Apply the inverse mapping to both box corners, then clip them to valid source-image bounds. If preprocessing stretches the image instead, use separate horizontal and vertical scale factors. If the detector already returns source-image coordinates, do not transform them again.

Verify overlays using objects near all four image edges. A centered object can hide a padding error that becomes obvious near the border.

Control latency instead of accumulating old frames

A detector can appear connected while processing increasingly stale images. Log acquisition, preprocessing, inference, and display durations separately.

At an illustrative capture rate of 30 frames per second, a new frame arrives about every 1000 / 30 = 33.3 ms. If a serial processing loop takes 50 ms per image, its maximum processing rate is approximately 1000 / 50 = 20 frames per second.

An unbounded queue then grows by roughly 30 - 20 = 10 frames per second. More queue capacity delays the failure but does not solve the throughput mismatch.

Choose the policy according to the task:

• Live display or tracking: A bounded queue that replaces an older pending frame can favor recent observations.
• Inspection requiring every acquisition: Match capture rate to sustained processing capacity, or use an appropriately designed triggered acquisition workflow.
• Offline analysis: Preserve all frames through a validated recording path and process them later.

15. Carry the source frame identifier and timestamp with every inference result. Draw each result on its associated frame, not whichever image is newest when inference finishes.

The asynchronous result and frame matching guide covers this association. Measure frame age as well as displayed frame rate; smooth video alone does not demonstrate low latency.

Add depth only after color detection works

Distance measurement adds another coordinate and timing problem. A detection box in the color image does not automatically identify the same pixels in the depth image.

16. Enable a supported depth profile only after the color-only pipeline is stable. Verify the requested stream combination before adding distance calculations.
17. Use the documented calibration and registration procedure to relate color pixels to depth samples. Preserve the frame pair and the geometry used for registration.
18. Reject invalid depth values and evaluate object pixels rather than blindly reading the center of every box.

A box center can fall on background, a hole, or a foreground obstruction. A segmentation mask or a carefully chosen interior region may provide better samples, but neither removes registration errors.

Use the raw versus rectified depth-to-color registration guide before indexing depth with color coordinates. Then apply the valid-pixel approach to object depth when selecting samples.

Also distinguish optical-axis depth from straight-line distance. The depth value versus true distance explanation describes that difference. Label the reported quantity and units explicitly.

Check hardware requirements before choosing parts

Browse 3D sensors for depth acquisition and area scan cameras when a separate image source is appropriate. The camera finder helps compare acquisition interfaces, but interface selection alone does not establish compatibility with your software.

The MV-DT01WMU is cataloged as a ToF sensing camera. Confirm its available image streams, supported SDK interfaces, and calibration outputs before specifying it for this workflow.

The MV-CU020-90GC is cataloged as a color area scan camera with a GigE interface and global shutter. It represents a separate color acquisition option, not a claim of direct compatibility with any particular depth device or detector.

A separate camera is not a drop-in replacement for an integrated color stream. If distance must correspond to its detections, plan for cross-camera calibration, overlapping views, and suitable timing. If only color detection is required, avoid adding depth hardware without a measurement requirement.

Verify the complete pipeline before deployment

Use this acceptance checklist with representative scenes:

• Acquisition: The selected device and active profile are logged, with visible timeout handling.
• Image interpretation: Saved frames have correct dimensions, colors, orientation, and row layout.
• Memory: Frame storage remains valid through preprocessing and inference.
• Detection: A saved frame and its live equivalent use the same preprocessing contract.
• Coordinates: Overlays remain correctly positioned near image edges and after resizing.
• Timing: Queue depth stays bounded, and frame identifiers connect outputs to their inputs.
• Distance, if enabled: Registration, invalid-depth handling, and measurement units are verified independently.
• Shutdown: The application stops acquisition and releases device resources through the documented API.

Test disconnects and restarts under controlled conditions appropriate to the application. A demonstration that works once is not a recovery strategy.

For hardware selection, start with the camera configurator. For help checking stream availability or integration requirements, send the active profile, pixel format, and acquisition error details through technical support.