Depth image at the wrong pixel: row and column order

You subscribe to an aligned depth topic, copy each message into a cv::Mat, read one pixel and multiply by the depth scale — and the distance comes back wrong. Sometimes it is a few tenths of a metre off on a static scene, sometimes complete garbage, sometimes a crash for pixels on the right of the frame. The cause is almost never the camera: OpenCV's at() takes its arguments as (row, column) while pixel coordinates are written (x, y), and x is the column. This guide runs the failure from symptom to fix — the argument order, the colorizer filter that quietly replaces 16-bit depth with RGB, the scale factor — and works an 848 × 480 example you can check by hand.

Symptom: one pixel's depth is wrong

The failing pattern, trimmed to the lines that matter:

ros::Subscriber aligned_depth_sub =
    nh.subscribe("/camera/aligned_depth_to_color/image_raw", 1, &depth_callback);

int   pixel_x = 200;         // illustrative: pixel taken from the color frame
int   pixel_y = 300;         // illustrative
float depth_scale = 0.001f;  // metres per raw step, read from the camera

Mat depth_image = Mat::zeros(Size(848, 480), CV_16UC1);

void depth_callback(const sensor_msgs::ImageConstPtr &msg)
{
    depth_image = cv_bridge::toCvCopy(msg, sensor_msgs::image_encodings::TYPE_16UC1)->image;
}

float real_z = depth_scale * depth_image.at<uint16_t>(pixel_x, pixel_y);  // wrong order

Two further clues: values at some pixels look believable, and failures cluster where pixel_x is large. Both point at the argument order inside at() — the most frequent cause of this symptom, ahead of the colorizer trap and the scale factor.

Cause 1: at() takes row then column; pixels are written x then y

An OpenCV Mat stores pixels row by row, and at() indexes row first, then column. Pixel coordinates run the other way: x is the column, y is the row. Put together:

Pass (pixel_x, pixel_y) and you read the pixel mirrored across the frame diagonal — (x = pixel_y, y = pixel_x). On a square frame that is a silent transpose; on a wide frame the same mistake also walks off the buffer.

Prove the order to yourself in three lines:

Mat probe = Mat::zeros(Size(4, 2), CV_16UC1);  // 4 wide, 2 high
probe.at<uint16_t>(1, 3) = 7;                  // row 1, column 3 = pixel (x=3, y=1)
bool ok = (probe.at<uint16_t>(1, 3) == 7);     // true: the value sits at (row, col)

probe.at<uint16_t>(3, 1) on this matrix addresses row 3 of a two-row image — exactly the out-of-range access a swapped call makes on your depth frame.

Cause 2: the colorizer filter replaces the depth topic with RGB

Many ROS depth-camera wrappers ship a colorizer filter, enabled with a launch option such as filter:=colorizer. Its documented effect: the depth topic then publishes a colorized RGB image instead of the 16-bit depth values. Your 16-bit read now interprets display bytes, and no scale factor rescues that.

The test is the message encoding. The raw depth stream arrives as a single-channel 16-bit image; the colorized stream arrives as three-channel RGB. Ask the converter for TYPE_16UC1 and treat anything three-channel as a display stream, not a measurement source.

The fix separates the two jobs: measure from the raw 16-bit topic, and build the colored view in your own node from that same frame. The 16-bit-to-8-bit conversion step is what displaying an aligned depth image in OpenCV walks through; if the raw frame shows up almost black in a viewer, that is the display path dividing 16-bit values by 256, not missing data. Colour-conversion failures on a 16UC1 frame have their own guide on 16UC1 display errors.

Cause 3: the scale factor applied never, twice, or in the wrong direction

A 16-bit depth frame holds raw steps, not metres. Multiply by the camera's depth scale — metres per step — read from the camera rather than hard-coded. A 1 mm step (0.001 m) is the common default for stereo depth cameras; read yours anyway, because the scale factor and focal length belong to each camera. The conversion from raw pixel values and the gray-value-to-mm mapping are covered in their own guides.

The test: place a flat target at a tape-measured 1.000 m and read the raw value. At a 1 mm step it should land near 1000. A converted distance near 1230 m means the scale was never applied; near 1.23 mm means it was applied upside down.

One further trap: the depth value is Z along the optical axis. The straight-line distance to an off-axis pixel is longer, so convert a pixel's Z to a true distance before comparing against a tape measure far from the frame centre.

Worked example: pixel (200, 300) on an 848 × 480 aligned depth frame

Inputs, each with its source:

• Frame: 848 columns × 480 rows, the aligned-depth-to-color resolution of the set-up above.
• Pixel: x = 200, y = 300, taken from the color frame.
• Depth scale: 0.001 m per step — the typical 1 mm stereo default, read from the camera.
• Raw values: 1230 at pixel (200, 300), a wall; 856 at pixel (300, 200), a nearer box. The raw numbers are example inputs; the arithmetic is what transfers.

Rule: row = y, column = x, so the read is at<uint16_t>(300, 200).

The arithmetic:

• Correct: at<uint16_t>(300, 200) returns raw 1230, and 1230 × 0.001 m = 1.230 m — the wall.
• Swapped: at<uint16_t>(200, 300) addresses row 200, column 300, which is pixel (300, 200), raw 856, and 856 × 0.001 m = 0.856 m.
• Error: 1.230 m − 0.856 m = 0.374 m, silent because both results are plausible distances.

The crash case: pixel (640, 200). Swapped, it becomes at<uint16_t>(640, 200) — row 640 of a 480-row frame. Columns x = 480 through 847, that is 848 − 480 = 368 of 848 columns, about 43% of the width, index memory outside the image.

The decision: on any frame wider than it is high — 848 × 480 here — the swapped call is out of bounds for 368 columns, and a release build returns foreign memory instead of throwing. Fix the order; never calibrate around a read from the wrong pixel.

Corrected callback, end to end

1. Subscribe to the raw or aligned depth topic and copy each message with TYPE_16UC1.
2. Accept only the single-channel 16-bit stream; a three-channel frame is the colorized image.
3. Read with at<uint16_t>(pixel_y, pixel_x) and multiply by a scale read once from the camera.
4. Treat raw 0 as invalid depth and skip it.
5. Derive any display image from the same 16-bit frame you measure.

Mat   depth_image  = Mat::zeros(Size(848, 480), CV_16UC1);  // placeholder sized to the stream
float depth_scale  = 0.001f;                                // metres per step, read from the camera

void depth_callback(const sensor_msgs::ImageConstPtr &msg)
{
    depth_image = cv_bridge::toCvCopy(msg, sensor_msgs::image_encodings::TYPE_16UC1)->image;
}

float depth_m(int pixel_x, int pixel_y)          // caller keeps x < width and y < height
{
    uint16_t raw = depth_image.at<uint16_t>(pixel_y, pixel_x);  // row first, then column
    if (raw == 0) return -1.0f;                  // invalid depth
    return depth_scale * (float)raw;
}

Until the first callback fires, the placeholder returns 0, which the invalid check absorbs: the read cannot crash, it can only report that no frame has arrived.

When the order is fixed and the value is still wrong

• The pixel came from the color frame but you read the raw depth topic: map the coordinates, or read the aligned stream and match the stream resolutions first.
• A ROI or binning setting on either stream changes which pixel is which — scale camera parameters after binning before reusing coordinates captured earlier.
• The surface genuinely returns no depth: dark, glossy or out-of-range areas produce invalid pixels, so mask them before any average or fit.

Checks before you trust the number

• The subscribed topic carries the 16-bit depth stream, not the colorized RGB image.
• The call reads at<uint16_t>(y, x), with x below the frame width and y below the height.
• The scale comes from the camera and is applied exactly once.
• One flat target at a tape-measured distance reproduces within a few steps.
• Raw zeros are masked as invalid before any average or fit.
• Display images are derived from the measured 16-bit frame, never measured themselves.

Depth cameras that stream 16-bit frames

The bug lives in the accessor, not the sensor: every stereo, structured-light and Time-of-Flight camera that publishes a 16-bit depth stream behaves identically. For hardware, the 3D sensors range covers the depth and stereo types, area scan cameras supply the color frame of an aligned pair, the camera finder filters by type and interface, and the camera step of the configurator walks the whole selection.

Time-of-flight sensing in a standalone unit:

Projected structured light and laser-galvanometer stereo types:

Laser-based 3D sensors project laser light: respect the laser class printed on the housing and never look into the emitter at close range. For questions about a specific stream or camera, ask the applications team.