Display an aligned depth image in OpenCV: 16-bit to 8-bit

Your RGB window renders fine and the aligned depth window stays black — or takes the process down with it — because depth is not a color image and never behaves like one. This guide is for engineers wiring a depth camera into an OpenCV pipeline. You will wrap the depth_aligned_to_color buffer as a CV_16UC1 Mat at the color stream's resolution, scale its millimetre values to 8-bit with cv::convertScaleAbs, prove the result against a tape-measured board, and know when cv::normalize or a JET colormap serves better. The recipe is SDK-independent: anything that publishes an aligned per-pixel distance frame works.

Why does the aligned depth window show nothing while RGB displays fine?

The aligned depth stream is a single 16-bit channel per pixel, so RGB display code fails twice: a CV_8UC3 wrapper reads three bytes per pixel from a two-byte-per-pixel frame, and a direct cv::imshow of CV_16UC1 divides every value by 256, which parks a 4,000 mm scene at gray level 15 of 255.

Alignment exists so depth and color share one pixel grid: each distance is reprojected onto the color camera's frame, which is why the aligned buffer's rows and columns follow the color stream. The CV_8UC3 wrap asks for 640 × 480 × 3 = 921,600 bytes from a frame that holds 640 × 480 × 2 = 614,400, so the last 307,200 come from memory past the buffer — that is how these windows segfault. Before the crash you see colored noise, because little-endian millimetre pairs split across the B, G and R positions. The correctly wrapped CV_16UC1 frame is intact; only the display scaling is missing. For how the aligned buffer is laid out and which intrinsics pair with it, see Aligned depth streams: data layout and which intrinsics apply.

What is in the depth_aligned_to_color buffer?

One uint16_t value per color pixel: the aligned frame carries the color stream's row and column count, every value is a distance in the SDK's depth units, and zero means no measurement came back, not zero distance.

A 640 × 480 color mode therefore yields a 640 × 480 depth frame whatever the depth sensor's native resolution. Values are distances in the SDK's depth scale; most cameras report integer millimetres and the worked example below assumes 1 mm per unit, so read the depth-scale setting in your SDK before trusting any threshold. At that scale the 16-bit span runs from 0 to 65,535 mm — 65.5 m — so an indoor scene never clips the range; contrast, not range, is the display problem. The zeros matter at scaling time, and byte order explains the colored noise above: 1,000 mm stores as the little-endian pair E8 03, which an 8-bit color reader splits into blue 232, green 3 and whatever follows as red.

Brightness wrong but geometry right is a display problem. Geometry wrong is a stream problem: color and depth running at different resolutions shifts the overlay, which Depth to color alignment breaks up: match stream resolution first sorts out, and Depth to color registration: raw versus rectified frames explains which frame carries which grid.

How do you wrap the buffer into a cv::Mat?

Wrap the pointer your SDK returns for the depth_aligned_to_color frame as cv::Mat depth16(rows, cols, CV_16UC1, ptr), with rows and cols taken from the color stream — for a 640 × 480 color mode that is cv::Mat(480, 640, CV_16UC1, ptr).

1. Read rows and cols from the color stream configuration, never from the depth sensor's native mode — alignment already did that mapping.
2. Wrap without copying. If the SDK documents a row stride larger than cols × 2 bytes, pass it as the fifth constructor argument instead of assuming packed rows.
3. Assert the result: depth16.type() == CV_16UC1 and depth16.total() == 307,200 pixels.
4. Clone before you compute on values: the Mat is a view of the SDK's buffer, and the next frame overwrites it mid-loop — the same trap behind torn frames in a camera callback.

// rows and cols come from the color stream configuration, not the depth stream
const int rows = 480;                 // pixels
const int cols = 640;                 // pixels
// depth_ptr: the frame pointer your SDK returns for depth_aligned_to_color
cv::Mat depth16(rows, cols, CV_16UC1, depth_ptr);   // wrap only, no copy
CV_Assert(depth16.type() == CV_16UC1);              // single 16-bit channel
CV_Assert(depth16.total() == 307200);               // 480 x 640 pixels

A frame arriving at half the expected byte count usually means an 8-bit preview got wrapped instead of the 16-bit depth data; Why your 16-bit image buffer has half the expected pixels covers that variant.

How do you convert 16-bit depth to 8-bit for cv::imshow?

Map a chosen millimetre range onto 0-255 with cv::convertScaleAbs: for a 500-4000 mm scene, alpha = 255/(4000 - 500) ≈ 0.0729 per mm and beta = -0.0729 × 500 ≈ -36.4, so a board at 1,000 mm lands at gray 37 and the 4,000 mm back wall saturates at 255.

1. Pick z_near and z_far in millimetres — here 500, the closest this rig can see, and 4000, the back wall.
2. Compute alpha = 255/3500 = 0.0729 per mm and beta = -36.4.
3. Clone the frame and park no-return zeros at z_near, so holes render black instead of gray 36.
4. Convert and show:

const double z_near = 500.0;             // mm, closest distance worth showing
const double z_far  = 4000.0;            // mm, farthest distance worth showing
const double alpha  = 255.0 / (z_far - z_near);   // 0.0729 per mm
const double beta   = -alpha * z_near;            // -36.4

cv::Mat show = depth16.clone();                      // display copy, stays 16-bit
show.setTo(cv::Scalar(uint16_t(z_near)), show == 0); // no-return pixels -> black
cv::Mat depth8;
cv::convertScaleAbs(show, depth8, alpha, beta);      // 500-4000 mm -> gray 0-255
cv::imshow("Aligned depth", depth8);

Arithmetic check: 500 mm -> |36.45 - 36.4| ≈ 0; 1,000 mm -> |72.9 - 36.4| ≈ 37; 4,000 mm -> |291.6 - 36.4| = 255.2, which saturates to 255.

cv::normalize(depth16, depth8, 0, 255, cv::NORM_MINMAX, CV_8U) is the one-liner for a saved snapshot: it stretches the frame's own minimum and maximum to 0-255. Live it flickers, because every frame rescales, and unmasked zeros drag the minimum down, landing 500 mm at 500/4000 × 255 ≈ 32 instead of 0. Fixed-range convertScaleAbs is the live-window choice; normalize is fine for files.

Why does cvtColor fail on the depth frame?

Color conversion codes are defined for multi-channel color formats, so cvtColor rejects a single-channel CV_16UC1 Mat or converts nonsense — run it only on the RGB frame, exactly where your working CV_BGR2RGB line sits today.

There is no cvtColor code that means '16-bit distances to viewable gray'; that job belongs to convertScaleAbs. Keep the paths separate: the color line — cv::Mat(480, 640, CV_8UC3, ptr) followed by cvtColor with CV_BGR2RGB, spelled COLOR_BGR2RGB in current OpenCV — stays untouched, and depth never enters it. The full list of 16UC1-versus-conversion failures and their fixes lives in Depth image 16UC1 display: fix color conversion errors.

How do you keep millimetre values while showing a picture?

Measure on the 16-bit Mat and display only an 8-bit copy: thresholds and statistics run in millimetres on depth16, while depth8 exists solely for cv::imshow, made fresh each frame so the SDK's buffer reuse never corrupts the view.

A distance threshold is a one-line expression: depth16 <= 1000 builds an 8-bit mask of everything within 1 m and never touches the display scaling. The 8-bit view itself quantizes distance to 3,500/255 ≈ 13.7 mm per gray level in the example range — fine for judging an image, far too coarse for measuring one.

For color, colormap the display copy: cv::applyColorMap(depth8, colored, cv::COLORMAP_JET) puts near parts at the blue end and far parts at red, while depth16 keeps every millimetre. Normalize a depth map for JET colormap display, keep the values shows the variant that preserves raw values through colorization.

Which checks prove the display before you trust it?

Three checks catch every wrong-wrap case: depth16.type() must equal CV_16UC1, the valid-pixel share comes from cv::countNonZero, and a taped target settles whether the values are distances at all — about 1,015 mm expected for a board taped at 1,000 mm and sitting 10° off square.

• Type and size first: depth16.type() == CV_16UC1 and depth16.total() == 307,200. Every blank-window case above fails one of these two.
• Valid-pixel fraction: cv::countNonZero(depth16) divided by depth16.total(). A share of zero means the sensor returned nothing at all, which is a capture problem, not a display problem.
• The tape-measure test: read depth16.at<uint16_t>(y, x) near the center of a flat white board square to the camera at a taped 1,000 mm; the value should print about 1000. Tilt the board 10° and the reading climbs to 1000/cos(10°) ≈ 1015 mm, because the camera measures along the ray, not the board normal.
• Ramp sanity: after scaling, the 1 m board sits at gray ≈ 37, the wall at 255, holes at 0.
• If the color view is the odd one out — black-and-white where you expected color — you grabbed the IR stream instead of RGB; Depth camera RGB shows black and white: you are on the IR stream is the fix.

Which depth cameras give you a stream worth displaying?

The display code above is camera-independent: any depth camera whose SDK publishes an aligned per-pixel distance stream feeds it unchanged, and only the depth scale and the range in the alpha calculation move. Structured-light projectors suit close-range, high-detail benches; ToF sensing covers larger volumes; laser profilers scan a line across moving parts. Examples from the shop's depth-camera range that feed this workflow:

The MV-DPS200P-02 is the projected structured-light option for bench cells, and the MV-DT01WMU is the ToF sensing option for larger volumes. The 3D sensors category holds the full range, and the camera finder filters it by type and resolution. To match a depth camera with the rest of the chain in one pass, start the vision configurator at the camera step.

If the window is still black after all of this, /contact/ reaches the applications team — send your Mat type, both stream resolutions and one saved frame. The 3D vision guides collect the rest of this topic, from data layout to point clouds.