Aligned depth point cloud wrong scale: fix unit conversion

You align depth to the colour camera, build the registered point cloud, and every dimension in the scene is off: a box you measured with a ruler at 0.3 m reads about 2.2 m in the viewer's measure tool. The unaligned depth cloud from the same frames was correct, so the camera geometry is fine. The alignment step rescaled the depth values on the way through, and it rescaled them badly.

Below is the five-minute test that separates a per-byte conversion bug from a clean unit mismatch, the byte-level arithmetic that explains scale errors of several times, and the fix: convert each 16-bit pixel as one integer, then split it into bytes. It is written for anyone integrating a depth camera whose pipeline aligns depth to a colour frame, and for whoever signs off the result.

The symptom: only the aligned path measures wrong

Three observations together point at the alignment step rather than the camera:

• A target of known length measures wrong on the aligned or registered point cloud, and by a large factor.
• The unaligned depth cloud built from the same frames measures the same target correctly.
• Changing the reference frame, or re-running extrinsic calibration, does not move the error.

The last one matters. A calibration error shears, rotates or shifts the geometry; it does not multiply every length by a near-constant factor. The factor is not a fixed offset either: it grows with the depth value, which is what a scaling bug does and an offset bug does not.

First test: measure a known object and compute the ratio

You need a ruler and a viewer with a measure tool.

1. Pick a rigid object of trusted length: a box, a gauge block, the 300 mm edge of a calibration plate. Ruler reading: 0.3 m.
2. Measure the same edge on the aligned point cloud: 2.2 m.
3. Divide: 2.2 m ÷ 0.3 m = 7.33.

Then read the ratio. A clean power of ten — 1000×, 0.001×, 10× — means a declared-units mismatch: the frame is labelled in one unit and consumed as another. That is fixed by setting the declared scale once, and the depth scale to metres conversion for raw pixel values article walks that arithmetic. A ragged number like 7.33 means the conversion ran separately on parts of each pixel and truncated, so the surviving factor changes with each value's bit pattern.

Finish the isolation: run the same measurement on the unaligned cloud. Correct there, wrong after alignment — the alignment routine is the one stage your pixels pass through on the broken path and not on the working one, which is exactly where to look.

Likely causes, in the order to check them

1. Per-byte unit conversion in the alignment step. Ragged ratio, aligned path only. Test: the synthetic-frame test lower down. Fix: convert the full pixel once, then split the bytes.
2. Declared depth-unit mismatch. Ratio exactly 1000× or 0.001×. Test: read the scale or unit parameter your SDK exposes and check it against what the consumer assumes; the depth gray value to mm conversion for 16-bit range images article shows the arithmetic.
3. Leftover per-axis scaling downstream. A different factor per axis, so a cube comes out rectangular. Test: measure the same edge along X, Y and Z. Fix: remove the leftover factors; the depth image to point cloud: apply per-axis scale factors article shows where scaling belongs.
4. Deprojection with the wrong intrinsics. The geometry leans and shears more than it scales. Test: check whether straight edges stay straight. Fix: deproject with the depth camera's own intrinsics, as the aligned depth point cloud intrinsics: which camera to deproject with article explains.
5. A different failure with a shared symptom. Values that collapse at close range are a scale mismatch of their own; see depth scale mismatch: why your point cloud is cut off at close range.

Root cause: a unit conversion applied per byte

A 16-bit depth pixel is two bytes in memory. Read little-endian, the value recombines as low + 256 × high: a pixel holding 301 is stored as bytes 45 and 1, because 45 + 256 × 1 = 301. It has to be handled as one integer. If the code multiplies each byte separately by a fractional unit factor and stores the results back as bytes, the digits after the decimal point are lost in every byte, and the recombined value is wrong by an amount that depends on where the value's bits fall across the byte boundary.

An illustrative pass, with the arithmetic. Suppose the source frame reports depth in 0.5 mm steps and the pipeline converts to millimetres, so the factor f = 0.5:

• Pixel value 301, true depth 150.5 mm; bytes low = 45, high = 1.
• Per byte: 45 × 0.5 = 22.5, stored as 22; 1 × 0.5 = 0.5, stored as 0.
• Recombined: 22 + 256 × 0 = 22. A surface at 150.5 mm now reports 22 mm: 150.5 ÷ 22 ≈ 6.8× too close.
• Neighbouring pixel 302: 46 × 0.5 = 23 exactly, high byte again 0, output 23 against a true 151 mm — 151 ÷ 23 ≈ 6.6×.

Two neighbouring pixels, two different errors. That is why lengths come out ragged and surfaces look corrugated rather than uniformly scaled, and why no single clean multiplier fits the measurement.

When the factor runs the other way — a division that reaches a finer unit — the per-byte products no longer fit in a byte's 0-255 range, they wrap modulo 256, and pixels read too far. That is the direction field reports describe: a 0.3 m object measuring about 2.2 m, a factor of about 7.3. Either direction, the signature is the same: a value-dependent, non-clean scale that appears only on the converted path. Truncation in the multiply case always reads short, and it hits small depths hardest, which is why the damage hides until you measure something.

The fix: convert once, then split the bytes

If the conversion is your code, the whole change is to promote the two bytes to one integer, convert that value, and write the bytes back:

// Illustrative names. src and dst: uint8_t buffers holding one 16-bit depth
// image, little-endian. i: size_t pixel index.
// Source unit 0.5 mm per count, target unit 1 mm per count.
uint16_t raw = static_cast<uint16_t>(src[2 * i] | (static_cast<uint16_t>(src[2 * i + 1]) << 8));
uint16_t mm  = static_cast<uint16_t>((raw + 1u) / 2u);   // factor 0.5, round to nearest
dst[2 * i]     = static_cast<uint8_t>(mm & 0xFFu);
dst[2 * i + 1] = static_cast<uint8_t>(mm >> 8);

The test that proves it: feed raw = 301 through both versions. The per-byte version outputs 22. The full-pixel version computes (301 + 1) ÷ 2 = 151, which is the true 150.5 mm rounded to the nearest millimetre. Same input, one line moved, 129 mm of range error gone for that pixel.

Most teams meet this bug inside a camera SDK's alignment step rather than in their own code. There the practical fix is to move to a release where the conversion runs on the full pixel. Field reports describe the fix landing in one release and regressing in the next, so pin the version and keep the regression test from the next section in your build. Whatever you do, do not paper over the symptom by rescaling the cloud per axis afterwards: that route exists for documented unit scales, not for hiding a broken conversion.

Prove it: a synthetic-frame regression test

Build a depth frame in which every pixel holds the same value, 301, in 0.5 mm units, and run it through the alignment or conversion on its own.

• A correct conversion outputs (301 + 1) ÷ 2 = 151 per pixel: little-endian bytes 151 and 0.
• The per-byte bug outputs 22 per pixel, as the arithmetic above predicts.

Assert on the range too. A 16-bit frame holds 65,536 values, 0 to 65,535, so with f = 0.5 every output must lie between 0 and 32,768: 65535 × 0.5 = 32767.5, and round-to-nearest cannot exceed 32,768. Anything above that bound is a wrap or a missing conversion.

Run the test in continuous integration and again after every driver or firmware update. Then repeat the ruler test end to end: the 0.3 m box should measure 0.3 m within a few millimetres, since half a unit per pixel is 0.5 mm at this scale. With several nominally identical cameras, check each unit; scale differences between identical stereo cameras have their own causes, covered in stereo depth scale differs between identical cameras: verify and fix.

Prevent it, and when the fix is not yours to write

Four habits keep this class of bug out of a station:

• Treat arithmetic on individual bytes of a 16-bit buffer as a defect in code review, and keep unit factors as integer ratios with the ratio named in a comment.
• Assert at startup on the depth unit your pipeline assumes, so a changed driver default fails loudly instead of quietly rescaling the scene.
• Keep the synthetic-frame test and the ruler test in acceptance for every camera and driver combination, and re-run them after updates; this fix has regressed between releases before.
• Mount the acceptance target at the working distance the cell will run at; the 3D camera working distance and positioning guide covers placement. Reposition or service laser-based models according to the laser safety instructions in their own documentation.

If the pipeline is being rebuilt rather than patched, the same 16-bit rule holds for every depth camera you might specify, so make these two tests part of acceptance before the cell ships. These depth cameras cover the common jobs:

To specify the camera and the rest of the station in one pass, start from the 3D sensors category or filter the camera finder to the depth type you need:

If a conversion still refuses to add up, send the two measurements — ruler and viewer — through the contact page and we will walk the arithmetic with you.