Edit depth values in a CV_16UC1 depth buffer

You have a depth frame sitting in an OpenCV Mat — type CV_16UC1 — and you need to change what it says: clean outliers, stamp a region as invalid, patch a known-bad pixel before the inspection step runs. A 16-bit depth image is not a special format. It is a plain array of unsigned-short distance counts, and OpenCV lets you overwrite any element the way you would overwrite any pixel. What you cannot guess is the unit: the number you write is a count, not metres, and the counts-to-metres scale is a property of the camera. This guide gives you the conversion, the row-first indexing rule, the exact cast, and the read-back test that proves the edit took. The same steps work on any depth buffer from the 3D sensors range.

What a CV_16UC1 depth buffer actually holds

CV_16UC1 means one unsigned 16-bit integer per pixel. In a depth stream that integer is a distance count, not a brightness: multiply it by the camera's scale to get metres. The V4L2 Z16 pixel format defines the layout — 16 bits, one distance value per pixel — and leaves the distance unit to the device, so the unit is a property of the camera, not of the format. The scale therefore belongs to the camera, not to the buffer type.

The ×1000 you see in common examples assumes the most widespread scale: 1 count = 1 mm. Where a camera documents that scale it is exact, so 1.250 m becomes 1250. Where it documents 0.1 mm per count, the same line writes a distance ten times too short. Read the scale once per camera and reuse it everywhere — the stereo depth scale factor and focal length guide applies the same rule to stereo, and the depth gray value to mm conversion guide works the arithmetic for 16-bit range images.

Structured-light, ToF and volume depth cameras hand you exactly this kind of per-pixel depth buffer through their SDKs:

Convert metres to counts before you write

A uint16 spans 0 to 65535, so at the scale above the buffer covers 0 to 65.535 m — far beyond any indoor depth camera, which is why overflow is not the practical failure mode. The practical failure is the wrong scale. Divide, round to the nearest count, clamp to 0–65535, then cast:

uint16_t raw = static_cast<uint16_t>(std::lround(depth_m / scale));

A bare cast truncates instead of rounding, and casting metres straight in is worse: (uint16_t)1.25 stores 1 count, which reads back as 0.001 m. The depth scale to metres conversion guide covers the reverse direction on raw pixel values.

Edit depth values step by step

1. Confirm the type. depth_buffer.type() must return CV_16UC1. A CV_8UC1 Mat holds a display conversion, not depth data — the 16UC1 display fix guide explains how that swap happens.
2. Read the scale from the camera documentation or SDK property, in metres per count, and store it next to the buffer.
3. Convert and cast with the rounding line above; clamp if the distance can exceed 65535 × scale.
4. Write row-first: depth_buffer.at<uint16_t>(row, col) = raw;. The first index is the row (y). Swapping to (x, y) edits a different pixel — the row and column order guide shows the symptom.
5. Edit regions through row pointers: depth_buffer.ptr<uint16_t>(r)[c] = value;. The memory is row-major, address = y × width + x, as the raw depth frame buffer indexing guide explains.
6. Keep 0 for no return. Invalidate with 0. A small non-zero count reads as a real surface very close to the camera, which is dangerous if the map feeds collision checks.
7. Read back and compare before the frame moves on.

The edit in code, with the test that proves it:

// Illustrative values; cv::Mat, CV_16UC1, at<uint16_t> and ptr<uint16_t>
// are documented OpenCV names.
int      row     = 240;     // pixel row (y), valid range 0 .. rows-1
int      col     = 320;     // pixel column (x), valid range 0 .. cols-1
double   scale   = 0.001;   // metres per count, from the camera documentation
double   depth_m = 1.250;   // distance this pixel should report

uint16_t raw = static_cast<uint16_t>(std::lround(depth_m / scale)); // 1250
depth_buffer.at<uint16_t>(row, col) = raw;

// Test: read back and convert again.
uint16_t readback = depth_buffer.at<uint16_t>(row, col);
// readback == 1250 and 1250 * 0.001 == 1.250 m, so the write is confirmed.

The Mat usually wraps SDK capture memory — the SDK frame buffer wrap guide shows the no-copy pattern. That memory is reused every frame, so clone the Mat before writing if the edit must survive past the next capture; the frame conversion hangs guide is the ownership trap in action, and the buffer reuse guide covers the allocation side.

Worked example: writing 1.250 m into a 640 × 480 z16 frame

Inputs, each with its source:

• Frame: 640 × 480, CV_16UC1, wrapped from the SDK depth buffer — the VGA-class frame most depth SDKs start you with.
• Scale: 0.001 m per count — the scale behind the ×1000 conversion, and the value to confirm for your camera.
• Target: pixel (row 240, col 320), the frame centre, must report 1.250 m — your corrected standoff.
• Type limit: 65535 counts, from the uint16 that CV_16UC1 stores.

Rule in words: divide the target distance by the scale, round to the nearest count, clamp to 0–65535, write it at (row, col), read it back and multiply by the scale again.

Arithmetic:

• 1.250 m ÷ 0.001 m per count = 1250 counts.
• 1250 ≤ 65535, so the value fits with no clamping.
• Write depth_buffer.at<uint16_t>(240, 320) = 1250;
• Read back: 1250 counts × 0.001 m per count = 1.250 m — the write is proven.

Decision: at the 0.001 m per count scale, cast and write directly — the 65.535 m ceiling cannot be reached by an indoor depth camera, so no clamp is needed. If the camera counts 0.1 mm (0.0001 m per count), the ceiling drops to 6.5535 m, and the ×1000 shortcut writes 1250 counts, which read back as 0.125 m — ten times short. So: read the scale, divide by it, cast. The shortcut is correct only where 0.001 m per count is the documented scale.

Whole-frame edits: masks, clipping and outliers

Three operations cover most depth post-processing:

• Threshold invalidation. Set every count above a cut to 0. At that scale, a 4.000 m cut is 4000 counts:

depth_buffer.setTo(cv::Scalar(0), depth_buffer > 4000);

• Clip to the documented near and far limits so returns outside the working range cannot reach the inspection step.
• Convert for display last. Work in 16-bit, edit, then convert to 8-bit for viewing only. The JET colormap guide shows display without destroying the values, and the aligned depth display guide covers the 16-to-8 conversion.

If the SDK outputs both a raw stream and an aligned depth product, edit the one you consume; alignment resamples the frame, so a patch written upstream can land on the wrong pixel — the aligned depth point cloud wrong scale guide covers the same ordering trap for unit conversion.

Common mistakes

• (x, y) instead of (row, col): the patch lands off the diagonal. Row first, always.
• Editing a converted display Mat: one call to type() settles which buffer you hold.
• Hard-coded ×1000: exact at 0.001 m per count, ten times short at 0.0001, ten times far at 0.01.
• Truncated cast: (uint16_t)1.25 stores a single count instead of 1250.
• Writing into SDK memory: it is refilled next frame; clone or reapply per frame.
• Expecting sub-count precision: a uint16 cannot store 1250.5. Do fractional maths in a float copy, then round into the counts.

When editing the buffer is the wrong fix

A constant depth error — a mis-measured mount, a swapped lens — belongs in calibration, not in a per-frame patch; the hand-eye calibration unit mismatch guide shows how unit slips hide in exactly that path. And if the downstream step needs geometry rather than the image, convert the whole frame once instead of patching pixels: the 2.5D height map to 3D point cloud guide walks pixels to XYZ.

Picking hardware instead of patching frames: the area scan cameras finder filters by interface, resolution and shutter, and the 3D topic page collects the depth guides. To spec camera, lens and light together, start here: