Align depth and color frames before detection in Python

A stereo depth camera hands you two synchronized streams, and neither one is in the other's geometry. The depth sensor and the color sensor sit tens of millimetres apart behind different lenses, so the same object edge lands on different pixels in each image. Feed both straight to a detector and the bounding boxes read depth from the wrong patch of scene, or the script dies with a TypeError that looks like a camera fault. This how-to puts depth pixels into color coordinates before the detector sees them, traces the 'object is not subscriptable' crash to a variable mix-up rather than the stream, and gives you an overlay test that proves the alignment worked. The pattern applies to any stereo or structured-light 3D camera whose Python SDK delivers depth and color side by side.

Why do depth and color pixels not line up on a stereo 3D camera?

The two streams come from two separate sensors: the depth camera and the color camera sit a baseline apart — 50 mm is typical for compact stereo modules — so a point 1 m away lands about 23 pixels apart on two 640-pixel-wide frames, before any lens distortion.

Each stream has its own field of view, distortion and resolution, so there is no single shared pixel grid. The depth value measured for a table edge lives at one pixel in the depth frame; the color pixel showing the same edge sits tens of pixels away. Registration resamples one stream onto the other's pinhole geometry using the factory-measured extrinsics between the sensors and each stream's intrinsics. Why the two geometries exist at all is its own subject: depth to color registration: raw versus rectified frames.

The offset follows from pinhole geometry:

That is the misalignment the align step removes: tens of pixels at working distance, the size of a small detection box. If your depth overlay is wrong by roughly that much, alignment never ran — nothing subtler is going on.

Which stream should depth be aligned to?

Align depth to color when a detector consumes the RGB image and depth is only needed inside its boxes; align color to depth when the metric depth grid is the output you measure on; pick the stream whose pixel coordinates your algorithm indexes with.

Detection is the first case in practice. Object detection, segmentation and tracking libraries return boxes and masks in color coordinates, so the depth frame is the one that has to move. Inspection built on a height map or volume measurement is the second: the depth grid carries the millimetres, so you rectify color into it or map results back afterwards. And when you need depth at just a handful of points — a gripper target, a corner — skip whole-frame alignment and project those points individually, as in color pixel to depth projection: one point instead of a whole frame.

How do I align depth to color frames in Python?

Enable both streams, start the pipeline, create the SDK's align object once bound to the color stream, and call its process method on every frame set before converting the frames to numpy arrays — three setup lines, two lines per frame.

You need the depth camera streaming depth and color over one link, its vendor Python SDK, numpy and OpenCV, and a detector that takes RGB arrays.

1. Create the pipeline and config objects from your SDK. Name the camera side distinctly — cam_config — so the detector's settings cannot collide with it (the crash in the next section is exactly this collision).
2. Enable depth and color at resolutions the detector tolerates. Start with 640 × 480 at 30 fps for both streams; declaring the same frame rate keeps the pairs synchronized.
3. Start the pipeline and keep the profile it returns — you need its intrinsics for any manual math later.
4. Build the align object once, outside the loop, bound to the color stream.
5. In the loop, grab a synchronized frame set and pass it to the align object's process method. The returned frame set carries depth already resampled into color coordinates.
6. Convert both frames to numpy arrays and copy them: SDKs commonly reuse an internal buffer, and the next grab overwrites your image — the same trap as stereo frames saved wrong: copy Mat data from the SDK buffer.
7. Run the detector on the color array and index the aligned depth array with the detector's pixel boxes.

import numpy as np

# Illustrative names: the manual of your camera's Python SDK gives the
# exact constructor, stream names and accessor names.
pipeline = dcam.pipeline()
cam_config = dcam.config()
cam_config.enable_stream('depth', 640, 480, 30)   # depth: 640 x 480 at 30 fps
cam_config.enable_stream('color', 640, 480, 30)   # color: 640 x 480 at 30 fps
profile = pipeline.start(cam_config)

align = dcam.align('color')                        # created once, not per frame
detector = load_detector()                         # your RGB detector

while True:
    frameset = pipeline.wait_for_frames()          # one synchronized pair
    aligned = align.process(frameset)              # depth onto color geometry
    depth = np.asanyarray(aligned.depth_data()).copy()  # typically uint16 mm
    color = np.asanyarray(aligned.color_data()).copy()  # typically uint8 BGR
    boxes = detector.detect(color)                 # boxes in color coordinates
    for x1, y1, x2, y2 in boxes:
        patch_mm = depth[y1:y2, x1:x2]             # coordinates now match
        height_mm = np.median(patch_mm[patch_mm > 0])

If your wrapper exposes no align object, or you want the registration inside your own code, the projection with intrinsics and extrinsics is laid out in align depth and color without the camera SDK pipeline. From there to running inference is the subject of depth camera object detection: connect frames to inference.

Why does Python report 'object is not subscriptable' on the frame loop?

Python raises that TypeError when the code indexes an object that does not support subscript access (obj[key]), and in detection scripts the culprit is almost always a name collision: the camera pipeline's config object and the detector's settings dictionary both named config.

The error text names the type you actually put brackets on. A message like 'config' object is not subscriptable means the object is the SDK's stream-config, not a frame set and not an image array. The classic sequence: you enable streams with config, then a detection helper reads its own settings with config['model'] — and config still points at the camera. Streaming alone works fine, the crash lands at the detector call, and it looks like the camera broke. It did not; the detector read a camera object as a dictionary.

Fix it in two moves:

1. Rename the camera side: cam_config.enable_stream(...), pipeline.start(cam_config). Load detector settings under their own name, model_cfg.
2. Confirm before the failing line with print(type(obj)). A frame set prints as the SDK's frameset class; a config object prints as the config class. Only frame sets, image arrays, lists and dicts take brackets.

What does the align step cost, and when should you run it?

Alignment resamples every depth pixel once, so its cost scales with pixel count: at 640 × 480 that is 307,200 resamples per frame, and running it only for the 10 frames per second your detector consumes, not all 30 captured, cuts the work to one third.

The arithmetic: 640 × 480 = 307,200 depth pixels per frame. Capture at 30 fps, align every frame, and you resample 307,200 × 30 ≈ 9.2 million pixels per second; align only the frames the detector eats at 10 fps and it is 307,200 × 10 ≈ 3.1 million — one third. Doubling each dimension to 1280 × 960 quadruples the count to 1,228,800 per frame.

Measure on your host by wrapping the align call in a timer: at 30 fps the whole capture budget is 33 ms per frame, so an align step eating more than a third of that deserves attention. The levers, in order: run alignment only on inference frames, drop depth resolution (320 × 240 is 76,800 resamples, four times fewer), or align color into depth instead when the depth grid is the smaller one.

If the capture rate itself collapses — pairs arriving at a fraction of the declared rate — the stream setup is the suspect, as in 3D camera frame rate low on software trigger: mode and data streams.

How do I verify the alignment before trusting the detections?

Overlay the aligned depth edges on the color image and read the median depth inside a hand-drawn color box on a planar target: edges should sit on color edges within a few pixels, and a target taped at 1.0 m should read 1.0 m.

Run these checks in order:

• Blend a depth colormap with the color image at 50 percent opacity. A skipped alignment shows as two outlines of every object, about 23 pixels apart at 1 m on the setup from the worked example.
• Tape a planar target at 1.0 m, draw a box around it in color coordinates, and read np.median(depth[box]). If the median reads the background wall instead of the target, the depth frame you indexed is raw, not aligned; confirm which frame you pulled with the check in is depth image_raw aligned to RGB? Check registration.
• Slide the target from 0.5 m to 2.0 m. A residual offset that shrinks with distance is normal resampling error; a constant pixel shift at every distance is a registration offset, corrected as in depth to RGB registration offset in pixels: fix the rectified stream.
• When the offset stays more than a few pixels everywhere, calibrate the pair yourself with a checkerboard, as in depth-to-RGB alignment off? Run your own checkerboard calibration.

Read depth from the middle of each detection box, never its border: the last row of valid pixels carries mixed values, and a median over the interior ignores them.

When is the SDK alignment the wrong tool?

Skip the SDK aligner when you need measurement-grade registration, when depth edges carry the decision, or when the camera has taken a shock that may have shifted its internal geometry.

Three cases deserve more than the factory aligner:

• Edge-critical measurements. Alignment interpolates, so silhouettes get flying pixels — a halo of depths between the true surface and the background, explained in flying pixels between object and background in stereo depth edges. Measure inside boxes, not at borders.
• Metrology. Where a passed or failed part must be defensible, write your own registration from a calibration you ran yourself; the color-to-depth extrinsic errors behind misalignment are covered in stereo camera RGB calibration: fix depth-to-color alignment errors.
• Damaged or remounted hardware. The aligner trusts extrinsics measured for the camera as built. After a drop or a bracket change, re-run the overlay test before production.

The decision, by case: detection on RGB frames — SDK align to color, one align object created once, run only on inference frames. Inspection on the depth grid — keep depth native and rectify color into it. Measurement that must survive an audit — your own calibration, on raw depth geometry end to end.

To pick the camera itself, start with the 3D sensors category, filter the camera finder to stereo and structured-light types, or work forward from your part in the system configurator. The 3D topic page gathers the registration, calibration and detection guides that surround this one.