Upsampled depth frames: alignment resolution vs accuracy

You aligned the depth stream to the color camera, saved the frame, and the file now reads 1920×1080 instead of 640×480. It looks sharper, so the doubt arrives: are those depth values still correct, or did the alignment quietly invent detail? This guide settles it for anyone pairing a depth camera with an area scan camera on the same rig. You will see exactly what the alignment step does to each value, why the upsampled canvas holds no more independent measurements than the native grid, when the values really do go wrong, and which canvas to pick for overlays, textured point clouds and dimensional work.

What happens to depth values when you align them to a 1080p color frame?

Aligning a 640×480 depth frame to a 1920×1080 color frame copies each depth value into a rectangular block of color pixels — 3 columns wide and 2 to 3 rows tall — and the copied numbers are exactly the values the depth sensor reported.

The alignment processing block in your SDK runs every depth sample through the calibrated transform between the two cameras, projects its 3D point into the color image, and finds the color pixel it lands on. Instead of writing one pixel, the implementation fills that pixel's destination rectangle, which is what keeps black gaps from appearing between mapped samples. Functionally, the rectangle fill is nearest-neighbor interpolation: no averaging, no blending, and every canvas pixel receives exactly one native sample, unchanged. Geometry moves; values do not.

Two consequences matter downstream. A statistical test on the canvas still sees the native distribution, blockiness included, because nothing was computed between samples. And the transform applied is the one your SDK calibrated; the distinction between raw versus rectified frames in depth-to-color registration decides which intrinsics that transform uses.

Does upsampling depth to the color resolution improve depth accuracy?

No. Accuracy stays where the depth sensor put it: alignment repeats every native sample across a block of canvas pixels, and a copied value carries no new information, no new lateral resolution and no lower noise than the original grid.

Separate the two accuracies. Range accuracy — how close a z value is to the true distance — is set by the sensor's optics, timing and factory calibration, and a copy operation cannot touch it. Lateral accuracy — the smallest spacing at which two different depths can be reported — is set by the native sample pitch on the part. Upsampling changes the spacing of pixels on your canvas, not the spacing of real samples on the part.

The photo analogy keeps teams out of trouble: scaling a 640×480 image to 4K in an editor adds pixels, not content. Depth is stricter still, because a blurred photo at least holds sub-pixel information for a good resampler to recover, while a block-copied depth frame holds literally nothing between blocks. Alignment done this way never loses precision and never gains any; where depth-to-color alignment keeps and loses precision shows which operations actually move it.

How many independent depth samples does the upsampled frame really hold?

A 640×480 depth frame holds 640 × 480 = 307,200 independent samples, and alignment changes nothing about that; the 1920×1080 canvas holds 2,073,600 pixels, so about one pixel in 6.75 carries a fresh value and the other 5.75 are copies.

The count is one multiplication, and the same division gives the pitch on the part:

The pitch that matters sits on the part, not the canvas. Take a depth camera with a 90° horizontal field of view and 640 native columns looking at a flat target 1.0 m away — example inputs; substitute your camera's real figures and the method holds.

The frame spans 2 × 1.0 m × tan(45°) = 2.00 m, so each native sample represents 2.00 m / 640 = 3.1 mm of target. After alignment the canvas divides the same 2.00 m into 1920 cells of 1.04 mm, but every cell is a slice of a copied block: two canvas cells 1.04 mm apart can never hold two different measurements. The smallest spacing between two independent depth readings stays 3.1 mm.

Decision from these inputs: a 3.0 mm feature spans less than one native sample at 1.0 m, so depth cannot size it there. Halve the working distance to 0.5 m and the pitch halves to 1.6 mm; choose a 1280-column depth sensor and the pitch halves at the same distance. Upsampling to 1080 columns changes neither number.

To run the same division the other way — from a feature size to the pixels you need — use the calculator below.

Why does upsampled depth look sharper on screen yet measure the same?

The sharpness is display, not data: nearest-neighbor copies give every canvas pixel a crisp value, so edges draw as hard block staircases that read as detail on screen, while the true sample pitch from the worked example above never moved.

Zoom into the saved frame and the structure shows: depth changes only every third column, and staircase edges step two to three rows at a time. A gradient filter along one row returns zero almost everywhere and jumps at block boundaries — at most 640 distinct values across 1920 pixels. The viewer paints each canvas pixel as a full square, which is why screenshots pass for high-resolution data.

When do upsampled depth values actually become wrong?

The block copy itself keeps every value valid; values go wrong when the transform's assumptions break — the cameras move relative to each other after calibration, the part moves during capture, or a depth-mixed edge pixel gets painted onto the wrong color pixel.

Mechanical: the extrinsic transform assumes both cameras sit exactly where calibration left them, and a knocked bracket shifts the mapping by a few pixels — fixing a depth-to-RGB registration offset in pixels walks through that symptom. Timing: if the part travels between the depth and color readouts, every block lands on color pixels showing a slightly earlier or later scene; the camera resolution for measurement accuracy in motion guide gives the exposure arithmetic to size that lag.

Physics: a depth pixel at an edge averages foreground and background returns into one value, and the block copy reproduces that mixture across six or seven pixels, spreading an already-mixed measurement over a wider-looking edge.

Notice that in all three cases the z numbers themselves stay honest: the error is in which color pixel each value is assigned to, or in the mixture the sensor made at an edge. The fixes are recalibration, synchronized triggering or edge masking — never a different resampling filter.

Which resolution should you request for the depth stream itself?

Ask the depth sensor for its highest native mode — 640×480 in this example — keep measurement on that grid, upsample to the color canvas only for overlays, and align color down to depth when every depth sample needs texture.

Depth cameras expose a short list of native modes, and the SDK's stream enumeration is the authority on what exists; 1080p depth is rarely on it. Pick the canvas by purpose:

For the textured cloud, align color down to depth and deproject with the depth camera's own intrinsics, as which camera to deproject an aligned depth point cloud with explains. For metrology, skip resampling and set the point cloud X Y Z resolution settings on the native grid. And if aligned output arrives black or shifted, mismatched stream resolutions are the usual cause — matching stream resolutions before alignment is the article to open next.

None of this applies to ordinary 2D color cameras, which have no depth stream to resample; it is a property of cameras that measure depth natively. The depth cameras in the shop all work this way:

A time-of-flight camera, a projected structured-light camera and a laser galvanometer stereo camera: each outputs depth on its own fixed grid, and that grid's size is the resolution you actually own.

How do you check that upsampling didn't corrupt your measurements?

Capture one flat target twice, at native depth and at the upsampled canvas: every canvas pixel inside a block must equal its native sample exactly, block-mean depths must match within the sensor's repeatability, and edge positions may move only in whole blocks.

The checks take minutes each:

• Block equality: inside every 3-wide block, all canvas pixels hold one identical value. Any difference means your pipeline blended something, and it is no longer a nearest-neighbor copy.
• Row distinctness: a 1920-pixel row of the upsampled frame contains at most 640 distinct depth values.
• Gauge comparison: measure a feature's width in both frames; the numbers must agree within the sensor's repeatability, and edges in the upsampled frame must step in block multiples, not single pixels.
• Temporal noise: the standard deviation of one pixel over 100 frames must match the native frame's value at the same spot.
• Averaging hygiene: average depth on the native grid, because averaging depth across a segmentation mask shows how mask averages go wrong when samples repeat.

When is real depth upsampling worth doing at all?

Treat any real upsampling — guided filtering and its relatives from the depth-upsampling literature, originally built for time-of-flight sensors — as reconstruction, not measurement: it pulls depth edges toward color edges, which looks convincingly right and must be validated part by part before a number from it feeds a tolerance.

For display, segmentation input or nicer visuals, those methods earn their keep. For dimensional results, the levers are physical, and all of them appeared in the worked example: working distance, because pitch scales with it — 3.1 mm at 1.0 m becomes 1.6 mm at 0.5 m — native columns, because 640 to 1280 halves the pitch at the same distance, or a different sensor class, such as a laser profiler that reports profiles at a stated resolution instead of a depth frame.

Measure first, buy second: take your smallest feature and working distance through the pitch formula above, then filter by resolution in the camera finder or start the configurator at the camera step. The 3D vision topic page collects the rest of the depth-stream guides, and free expert support with every order is there if you want a second pair of eyes on the numbers.