Depth frame resolution changes after alignment: read the frame size

A depth pipeline that reshaped without complaint on day one starts throwing ValueError the day you switch on depth-to-color alignment. You request a depth mode, request a color mode, pass every frame set through an alignment filter, and the depth buffer stops matching the numbers you configured. This guide is for anyone capturing RGBD data in Python. It explains why alignment changes the depth frame's resolution, gives you a three-cause test that takes a minute, shows the reshape and depth-scale fix, and works the buffer arithmetic so you can budget memory and link bandwidth with real numbers.

Alignment reprojects depth onto the color grid

Depth and color sensors sit a few centimetres apart, so one physical point lands on a different pixel in each image. Alignment removes that offset: the filter takes each depth pixel, projects it into the color camera's geometry and writes it on the color image's grid. In this SDK family the software path is an AlignFilter aligned to the color stream; the hardware path sets the pipeline config's alignment mode to its hardware value, so the same reprojection runs on the camera itself and costs the host CPU nothing.

Whichever path you choose, the output grid belongs to the color stream, and that is the whole rule: after depth-to-color alignment, the depth frame is sized to the color stream's resolution. Configure a 1280 × 800 depth mode with a 1920 × 1080 color mode and the aligned depth frame comes back 1920 × 1080. Configure both at 1280 × 800 and the sizes happen to agree, but nothing about the mechanism changes. When the color stream is finer than the depth sensor, the aligned depth is upsampled by interpolation, which fills the grid without adding depth detail; accuracy stays at the depth sensor's native pitch, as the guide on upsampled depth frames: alignment resolution vs accuracy explains.

Alignment fixes registration, not timing. If the two streams also need to be time-matched, enable the pipeline's frame sync (enable_frame_sync()) before start; sync changes nothing about resolution. And if aligned pixels still drift once the sizes are right, that is a registration problem, not a reshape problem — the guide on depth to color alignment breaking up: match stream resolution first covers that failure.

Why the color shape is right and the depth shape is not

The color image matches your configured shape because nothing moves it: the pipeline delivers the color mode you asked for and no filter reprojects it afterwards. The depth image passes through two stages — first the depth mode you requested, then the alignment filter, which rewrites it on the color grid. What you inspect after the filter is the second stage, so the depth buffer's true size is the color stream's size, whatever the config says.

Some cameras couple the two streams in the opposite direction, with the color stream's resolution following the depth setting; that variant is covered in the guide on why a depth camera's color stream resolution follows depth. Either direction, the lesson is the same: take the shape of each buffer from the frame object, not from the configuration.

Three causes, tested in order of probability

Cause 1: the aligned frame is at the color resolution. The SDK's own alignment guide says it directly: once alignment runs, the depth frame comes back at the color stream's resolution. Print the depth frame's dimensions right after the filter call and compare them with the color frame's; if they agree with each other and not with your depth request, this is your cause.

Cause 2: the buffer arrives as bytes, not 16-bit counts. A Y16 frame stores 2 bytes per pixel. If get_data() hands the buffer back as a uint8 array, the element count is twice the pixel count, and astype(np.uint16) keeps both bytes as separate elements — the reshape then fails even though the resolution is exactly what you configured. Compare raw.size against width * height and 2 * width * height, and print raw.dtype.

Cause 3: the reshape order is swapped. Writing reshape((width, height)) instead of reshape((height, width)) succeeds silently whenever the element count matches — and it always does, as the arithmetic below shows. The array comes back transposed, so overlays land on swapped coordinates. Assert depth.shape == (height, width) immediately after the reshape.

The fix: read the size back from the frame

get_width(), get_height(), get_data() and get_depth_scale() are the frame's own methods, and together they are every number the reshape needs. Read them after the alignment filter has run, on every frame whose configuration could have changed; the numpy handling around them is ordinary and yours to adapt:

import numpy as np

frames = align_filter.process(frames).as_frame_set()
depth_frame = frames.get_depth_frame()

width = depth_frame.get_width()      # columns of the aligned depth image
height = depth_frame.get_height()    # rows of the aligned depth image

raw = depth_frame.get_data()         # one-dimensional buffer from the SDK
u16 = raw if raw.dtype == np.uint16 else raw.astype(np.uint16)
if u16.size == 2 * width * height:   # Y16 arrived as uint8 byte pairs
    u16 = raw.view(np.uint16)        # reinterpret each pair as one count

depth = u16.reshape(height, width)   # rows first, columns second
depth_mm = depth.astype(np.float32) * depth_frame.get_depth_scale()

# The test that proves it
assert depth.shape == (height, width)
assert depth_mm.shape == depth.shape
print(width, 'x', height, '|', u16.size, 'counts | scale', depth_frame.get_depth_scale())
# With a 1920 x 1080 color stream: 1920 x 1080 | 2073600 counts | scale 1.0

The assert is the proof: with a 1920 × 1080 color stream the frame reports width 1920, height 1080, the buffer holds 2,073,600 counts, and depth.shape comes back (1080, 1920). For the scale, a raw count of 4250 multiplied by a reported scale of 1.0 gives 4250 mm, so that pixel sits 4.25 m from the camera. If the buffer matches neither width * height nor twice that, your resolution assumption is wrong — go back to the cause table rather than forcing the reshape.

One more trap lives after the reshape: buffers tied to a frame set that the pipeline reuses. Hold a converted array across the next capture and you read memory that has already been refilled; the guide on point clouds going null after depth-to-colour alignment walks through that lifetime problem.

Worked example: buffer arithmetic after alignment

Take these example inputs: a 1280 × 800 depth mode, a 1920 × 1080 color stream at 30 fps, Y16 output, alignment to color enabled.

Requested depth buffer: 1280 × 800 = 1,024,000 pixels; at 2 bytes per pixel that is 2,048,000 bytes, about 2.05 MB per frame.

Aligned depth buffer: 1920 × 1080 = 2,073,600 pixels, which is 4,147,200 bytes, about 4.15 MB per frame — twice the buffer you planned for.

At 30 fps the aligned depth alone moves 4,147,200 × 30 = 124,416,000 bytes per second, about 124 MB/s, before color and any point clouds. A gigabit link carries 1000 Mbps ÷ 8 = 125 MB/s of line rate and roughly 110-118 MB/s of usable throughput once protocol overhead is paid. The aligned stream therefore sits at or beyond the practical budget of a gigabit link, which is the arithmetic behind two standard decisions: step the color stream down, or let the camera align in hardware.

The transpose arithmetic closes the loop: reshape checks totals, not meaning, and the 1,024,000-count buffer reshapes equally well to (1280, 800) and to (800, 1280) because both products equal 1,024,000. With width and height different, the swapped call passes unnoticed and hands you a transposed image.

Which grid to work on, case by case

Stay off the aligned grid when the deliverable is metric geometry. A point cloud or height map built from resampled depth carries interpolated counts that add pixels without adding accuracy; build it on the depth sensor's own grid and project color into depth through the factory extrinsics instead. The guide on color pixel to depth projection shows the single-point version of that projection, and the guide on aligning depth to a third RGB camera covers the extrinsics you need when the color unit is separate. For overlays and pixel-registered detection the aligned grid is the right choice — take its size from the frame and the reshape stays correct.

If the pipeline refuses the requested depth mode outright, that is a different failure with its own fix: choose a mode from the device's listed stream profiles, as the guide on setting a listed depth mode shows.

Prevention habits

• Print both streams' width and height once at startup and again after every config change; never cache the configured numbers for post-alignment buffers.
• Assert the buffer size equals width * height before reshaping, so a wrong assumption fails in the log instead of in the overlay.
• Read the depth scale from the frame and log it beside the dimensions; it belongs to the same read-it-back habit.
• Re-run the dimension print after firmware or SDK updates, because mode lists and alignment defaults can move between versions.
• When a mode request is accepted but the buffer still disagrees, confirm the request went through the stream-profile list rather than being remapped on the way in.

Hardware and next steps

Everything above applies to an existing rig; when you are specifying depth hardware instead, check that the depth and color mode lists pair at the resolution your overlay needs, and prefer on-camera alignment when host CPU time is the constraint. These are the stereo and structured-light 3D sensors in our catalog for that class of work:

The MV-DPS200P-02 is a projected structured-light 3D camera, and the MV-DLS300P-04 pairs a laser galvanometer with stereo depth. Both sit in the 3D sensors category with the rest of the range, which you can filter in the camera finder. Once the reshape is stable, the same grid feeds masking, point clouds and height maps — the guide on turning a 2.5D height map into a 3D point cloud picks up exactly there. For a second opinion on a capture pipeline, contact us, or browse all 3D guides for the surrounding topics.