Why aligned depth to color keeps the depth resolution

You raised the color stream to 960×540, the depth stream still delivers 640×480, and the aligned topic you expected to reconcile the two publishes 640×480 as well. That is not a broken launch file; it is what alignment is built to do. This explainer is for engineers pairing a depth sensor with a color camera who need depth and color in one array. You will see why the aligned image keeps the depth grid, what each of its pixels holds, and the projection that builds the 960×540 depth array you actually wanted, with the arithmetic in the open.

Why does the aligned depth image stay 640×480 when color runs 960×540?

Because alignment changes geometry, not sampling: it reprojects every depth sample into the color camera's viewpoint while keeping one value per original depth pixel, so the message keeps the depth stream's 640×480 dimensions.

The color_width and color_height arguments in the wrapper's launch file set what the color imager streams. The depth imager keeps its own mode, and the aligned output is a depth image, so its resolution belongs to the depth side. No launch argument changes that, because there is nothing to resize — the grid holds exactly one entry per measured sample. If the color stream also resists resolution changes on your camera, that is a separate mechanism, explained in depth camera color stream resolution: why color follows depth.

What does each pixel of aligned_depth_to_color actually hold?

Each of the 307,200 entries stores one real depth sample — the surface point that depth pixel measured, re-expressed from the color camera's optical frame — at its original depth-grid index.

The message's frame identifier switches to the color camera's optical frame, which is what "aligned to color" means: the geometry has moved, the array has not. Samples whose surface point falls outside the color camera's view come back invalid, so expect thinning near the borders. For the exact byte layout and the intrinsic set that deprojects this image, read aligned depth streams: data layout and which intrinsics apply; once you convert it to a point cloud, aligned depth point cloud intrinsics: which camera to deproject with settles which camera model to use.

How do I build the 960×540 depth array I wanted?

Run the standard chain on every valid depth pixel — deproject to 3D with the depth intrinsics, transform with the extrinsics, project with the color intrinsics — and write the depth into a 960×540 buffer at the projected pixel.

1. Read the depth Z at pixel (x_d, y_d) in the depth camera's frame.
2. Deproject to the 3D point P_d with the depth intrinsics.
3. Transform to the color frame with the extrinsics R and t.
4. Project to (u_c, v_c) with the color intrinsics.
5. Write Z_c at (round(u_c), round(v_c)); drop anything outside columns 0–959 and rows 0–539, and when two samples land on one pixel keep the smaller Z_c so the front surface wins.

Example numbers for a front-facing pair — read the real ones from the camera_info messages at runtime. Depth: 640×480, fx_d = fy_d = 470 px, cx_d = 320 px, cy_d = 240 px. Color: 960×540, fx_c = fy_c = 850 px, cx_c = 480 px, cy_c = 270 px. Extrinsics: a 25 mm baseline to the right, no rotation. Take the depth pixel (320, 240) reading Z = 1000 mm:

• X = (320 − 320) × 1000 / 470 = 0 mm; Y = (240 − 240) × 1000 / 470 = 0 mm
• P_c = (25, 0, 1000) mm after the baseline translation
• u_c = 850 × 25 / 1000 + 480 = 21.25 + 480 = 501.25, so column 501
• v_c = 850 × 0 / 1000 + 270 = 270

The buffer stores 1000 mm at row 270, column 501. The 21.25 px offset is pure baseline parallax, and it shrinks as surfaces recede: the same surface at Z = 500 mm maps 42.5 px right, to column 523. That distance dependence is why no constant scale factor can align the pair.

If you need depth at only a few detected features, skip the full array and use color pixel to depth lookup for detected image features — one lookup per feature costs microseconds and invents no samples. The single-ray variant is color pixel to depth projection: one point instead of a whole frame.

Why nearest neighbor instead of interpolation?

Depth is a distance, not an intensity: interpolating across an object edge blends foreground and background into a surface that exists nowhere, so nearest neighbor is the correct write for measurement code.

Count the slots before expecting smooth coverage. The 960×540 buffer has 518,400 entries, the depth image contributes 307,200 samples, so at best 59.3% of the buffer receives a direct sample and 211,200 entries — 40.7% — start empty. Empty is honest: it means no measurement exists there. Treat the buffer as a container, fill small gaps deliberately, and read depth to color alignment without losing depth precision before reaching for any smoother.

Does the upsampled 960×540 depth measure more than 640×480?

No — resampling adds pixels, not measurements: lateral resolution stays fixed by the depth imager's grid, and the upsampled array is a registration container, not new information.

When the measurement itself needs finer lateral sampling, the fix is a finer depth sensor, not a bigger color image; the trade-offs are laid out in upsampled depth frames: alignment resolution vs accuracy. The classes worth comparing are projected structured-light cameras such as the MV-DPS200P-02, ToF cameras such as the MV-DT01WMU, and the laser profilers and stereo sensors across the 3D sensors category.

Why does the 960×540 color stream crash until you replug the camera?

Intermittent failure that only a power cycle clears points at the USB link and stuck streaming state, not at your launch file — compute the bandwidth first, then start the wrapper from a known state.

A 16-bit-per-pixel color format such as YUYV at 960×540 and 30 fps moves 960 × 540 × 2 B = 1,036,800 B per frame, which is 31.1 MB/s — about 249 Mbit/s. The wrapper's default 640×480 mode moves 614,400 B per frame, about 147 Mbit/s. A camera that enumerated at USB 2.0 High-Speed has roughly 200–320 Mbit/s usable for isochronous video after overhead, so 249 Mbit/s sits on the edge: some sessions start, others fail, and an unplug clears the half-open stream. The decision path: confirm the link reports SuperSpeed (5 Gbit/s) before touching any argument, power-cycle to clear stuck state, and request only modes the camera lists. Matching both streams to one resolution also removes a class of alignment failures, as depth to color alignment breaks up: match stream resolution first shows. For the stuck-state habit, 3D camera test flaky on leftover settings: start from a known state is the procedure.

When should I stop at the 640×480 aligned image?

Stop there whenever the downstream code consumes 3D — point clouds, plane fits, volume, picking — because those stages read the depth grid directly and gain nothing from a resampled copy at color resolution.

For the recording row, record depth and color to a bag file, not an AVI keeps both streams with their calibration for later alignment. More depth-stream questions are gathered on the 3D topic page.

What should I check before changing either stream's resolution?

Five checks, in order: the mode is in the camera's supported list, the link enumerates at SuperSpeed, the depth mode stays untouched, the bandwidth arithmetic closes, and the recording path keeps raw streams.

• Mode exists. The requested width, height, fps and format must appear in the device's mode list; the factory launch works because it asks for a supported mode.
• Link speed. A replug clears a stuck stream, but only SuperSpeed enumeration sustains the heavier color modes.
• Depth mode unchanged. The aligned output tracks the depth resolution, so leave the depth arguments alone while you move color.
• Bandwidth arithmetic. Run the bandwidth product against the link budget before the first launch.
• Record raw. Capture both native streams and align offline, so a later method change costs nothing.

When the depth grid itself limits the measurement, compare sensors in the camera finder, start a full pick in the camera configurator, or ask about a specific stream pair through the contact page.