Body index mask misaligned on colour image: fix the 4-byte stride

You transform the body index map into the colour camera's geometry, overlay it on the colour frame, and the mask sits squeezed toward the left of the person, drifting further off towards the right edge. It looks exactly like a bad depth-to-colour transform, so you re-read the transformation call and find nothing wrong.

The transform is usually innocent. The overlay loop is the culprit in almost every case of this kind: it walks the colour image three bytes per pixel while a BGRA32 frame stores four, so each write lands progressively further left of the pixel it meant to reach. You can confirm it with row, column and element-size arithmetic alone, and fix it by matching the pointer's byte width to the colour image's element size.

The symptom signature

Show the mask Mat on its own and it looks right: a clean silhouette against a 255 background, already in the colour camera's geometry. Everything goes wrong at draw time. The left edge of the overlay sits on the body, the error grows column by column towards the right, and the last columns of the row receive nothing at all. Any semi-transparent blend adds strips of scrambled colour, because three bytes of a four-byte pixel were written and the fourth was left stale. That is the signature of a byte-stride bug, not of a calibration fault.

The configuration this usually hides in: colour captured as K4A_IMAGE_FORMAT_COLOR_BGRA32 at K4A_COLOR_RESOLUTION_1080P, depth in K4A_DEPTH_MODE_NFOV_UNBINNED, mask converted to CV_8UC1 for drawing. If your setup matches, suspect the stride before you suspect the transform.

Clear the transform first: two checks

1. Show the mask buffer by itself. A complete silhouette with background at 255 means the depth-to-colour transform delivered sane data, and the fault lives downstream.
2. Print the geometry before the loop: indexMap.rows == colour.rows, indexMap.cols == colour.cols, colour.type() == CV_8UC4 and colour.elemSize() == 4. Matching sizes plus a four-byte element leave the write loop as the only suspect.

If the first check fails, the mask itself is broken and the transform needs attention. If rows or columns differ, a resize between capture and draw is bending the overlay; that case appears below. A clean, uniform offset with everything matching belongs to the transform, and that path starts with the alignment checks in Stereo camera RGB calibration: fix depth-to-color alignment errors.

The arithmetic on a 1920 × 1080 BGRA32 frame

The inputs, as captured:

• Colour frame: 1920 × 1080 pixels, CV_8UC4 (BGRA32), 4 bytes per pixel.
• Mask: 1920 × 1080, CV_8UC1, one byte per pixel, straight from the transform.
• The buggy loop: colour.ptr<cv::Vec3b>(row)[col], which advances 3 bytes per column.

A typed ptr<T> call computes each write's byte address as row × stride + col × sizeof(T). The stride is right; only the per-column term is wrong, but that is enough to move every write after the first.

Worked through:

• Row stride: 1920 px × 4 B/px = 7680 B.
• Column 1919 should land at 1919 × 4 = 7676 B, inside pixel 1919.
• The cv::Vec3b loop writes it at 1919 × 3 = 5757 B; 5757 / 4 = 1439.25, inside pixel 1439.
• Drift at the right edge: 1919 - 1439 = 480 px, one quarter of the row.
• Writes per row at 3 B/px: 7680 / 3 = 2560. The loop stops at column 1919, so columns 1440-1919 never receive a write: the same 480 columns.

Net effect: the mask is squeezed to 1440 of 1920 columns, 75 % of the frame width, with drift proportional to column index, and every write straddles two pixels, corrupting a neighbour's green, red or alpha byte. A mask that lands 75 % wide with left-to-right drift is a stride bug. Match the pointer to elemSize() and it disappears; only a uniform offset left over after that fix justifies touching the transform.

The fix: address the colour image four bytes at a time

The mask keeps its one-byte pointer; the colour image gets a cv::Vec4b pointer, so one increment moves exactly one pixel:

// colour: CV_8UC4 BGRA32 frame; indexMap: CV_8UC1 mask, same rows and cols
for (int row = 0; row < indexMap.rows; ++row) {
    const uchar* m = indexMap.ptr<uchar>(row);   // 1 byte per pixel
    cv::Vec4b*   p = colour.ptr<cv::Vec4b>(row); // 4 bytes per pixel
    for (int col = 0; col < indexMap.cols; ++col) {
        if (m[col] == 255) continue;             // 255 = background
        p[col] = cv::Vec4b(0, 255, 0, 255);      // B, G, R, A: solid green mark
    }
}

Writing all four bytes also repairs transparency blends; the strip artifact came from the stale alpha byte. If you do not need alpha, convert once with cv::cvtColor(colour, bgr, cv::COLOR_BGRA2BGR) and draw on three channels instead. Either variant is correct as long as the pointer's byte width equals elemSize().

One caveat: cols × elemSize() equals the stride only for a continuous Mat. The 1080p BGRA32 frame here is continuous, so its stride is 7680 B. A ROI sub-matrix can carry row padding, so read step[0] from the Mat rather than recomputing it once you crop.

If the stride was right: the two remaining causes

Ordered by how often each turns up:

1. A resized copy got drawn on. Display code scales the frame, the mask is drawn at capture resolution onto the scaled image, and the offset appears in both axes. Test: draw before any resize, or scale the mask with the frame, and confirm rows and cols still match at draw time.
2. The transform really is off. The overlay stays uniformly displaced, or the error concentrates where the mask holds few valid pixels, near the depth camera's field edge. Test: map the depth edges into the colour frame and overlay them, as Depth image edge overlay from a single acquisition stream shows from one acquisition, masking invalid pixels first as the confidence image bitflags guide describes. Fix: re-run the depth-to-colour extrinsic calibration, then re-check alignment.

Verify the overlay before you trust it

1. Assert types at function entry: CV_Assert(colour.type() == CV_8UC4 && indexMap.type() == CV_8UC1); — one line that would have caught the original bug on the first call.
2. Check the stride once: on the continuous 1080p BGRA32 frame, colour.step[0] reads 7680, which is 1920 × 4.
3. Draw a probe band: fill rows 100-109 across every column in green. Correct addressing hits both edges; a cv::Vec3b loop leaves the right 480 columns untouched.
4. Walk a body across the full frame, out to x near 1919. The mask must stay on the body at the right edge, where the old drift peaked at 480 px.
5. For pixel-level evidence beyond the mask, project the point cloud back into the image as Overlay 2D image on depth map describes, and keep residual flicker under control with the habits in Why pixel shifts change detection results.

Keep it fixed

• Never call ptr<uchar> or ptr<cv::Vec3b> on a Mat whose type() you have not checked; assert it at the top of every drawing function.
• Read cols, elemSize() and step from the Mat; derive nothing from a hardcoded 1920.
• Keep one helper for BGRA overlays and one for BGR, so the byte width is decided in exactly one place.

Decision, restated: shear that grows to the right is the stride; a constant offset is the transform; a stretch in both axes is a resize. Work through them in that order and the mask lands on the body at every column. If you are standing up the depth station this code runs on, start at the 3D sensors category, narrow the options with the camera finder, and browse the other SDK guides on the 3D topic page; contact the applications team when a configuration question outlasts the guides.