Skeleton overlay on RGB images: align depth camera joints

A depth-based tracking application can show a skeleton correctly while still displaying the wrong background. Changing the depth palette does not produce a color-camera view. You need the captured RGB image, joints expressed in a known coordinate system, and calibration that maps those joints into color-image pixels.

This workflow is for developers building a live operator preview or recording an annotated image. You will separate acquisition, geometric projection, and drawing so that timing errors do not get mistaken for calibration errors. The result is an RGB frame with joint markers and bone connections, not merely a recolored depth image.

Choose point projection, not depth colorization

Start by identifying the output you actually need. These operations solve different problems:

For a skeleton, you normally project only the joint positions. You do not need to register every depth pixel first.

Use the raw-versus-rectified registration guide to establish which image geometry your application uses. A correctly projected joint can still appear misplaced when drawn on an image with different distortion correction or cropping.

Follow these steps in order, using the detailed sections below:

1. Associate each tracking result with its source capture and corresponding color frame.
2. Establish coordinate units, axes, transform direction, and matching image calibration.
3. Transform joint positions into the color-camera coordinate system.
4. Project valid joints into the exact RGB image geometry.
5. Draw joint markers and bones, applying display transformations consistently.
6. Verify stationary and moving alignment, pixel format, stride, and buffer ownership.

1. Keep tracking results with their source capture

Enable both depth and color acquisition through the device's documented configuration. Confirm that color frames actually arrive before changing the renderer. A tracking viewer may continue displaying depth even when a color stream is available elsewhere in the application.

Submit the depth capture to the tracker, then associate each tracking result with the capture that produced it. Tracking can finish several frames after acquisition. Drawing that result over the newest RGB frame introduces motion-dependent displacement even when calibration is correct.

Store a small record containing:

• The tracking result and its source capture identifier.
• Depth and color exposure timestamps, where available.
• A retained color frame or an application-owned copy.
• Image dimensions, pixel format, and row stride.
• The calibration and stream configuration used for that capture.

Do not assume timestamps from different devices share a clock. Establish their relationship before comparing them. For integrated streams, follow the documented synchronization behavior rather than assuming that equal frame rates imply simultaneous exposure.

The depth and color timestamp guide explains how to distinguish capture timing from processing latency.

For scale, suppose a visible feature moves laterally at 1 m/s. A 20 ms mismatch represents approximately:

That displacement can make a correct calibration look inaccurate. Record timestamps alongside the preview before adjusting geometry.

2. Establish coordinate and calibration conventions

Determine where the tracker expresses its joint positions. Common possibilities include the depth-camera frame, a tracking reference frame, or an application-defined world frame. Do not infer the coordinate system from the appearance of a rendered skeleton.

Record these conventions explicitly:

• Axis directions and handedness.
• Position units.
• The direction of the depth-to-color rigid transform.
• Color intrinsics for the active resolution and image geometry.
• Whether color pixels are raw, rectified, cropped, resized, or mirrored.

If joints are already in the color-camera coordinate system, skip the depth-to-color rigid transform. If a documented projection helper accepts depth-space points and returns color pixels, it may perform both transformation and projection internally. Applying the transform beforehand would then apply it twice.

Use calibration associated with the active stream configuration. A principal point from a different crop or resolution can create systematic offsets. The intrinsic parameter conversion guide covers the relationship between image dimensions and camera parameters.

For raw images with lens distortion, prefer the manufacturer's documented projection function. The simple pinhole equations below apply to an appropriate undistorted or rectified image, using its corresponding intrinsics.

3. Transform joints into the color-camera frame

When joints are in depth-camera coordinates, transform each position into color-camera coordinates before projecting it.

The rotation matrix is dimensionless. Translation and joint positions must use the same length unit. Mixing positions in meters with a translation expressed in millimeters produces a large error that drawing adjustments cannot fix.

If calibration supplies the opposite transform, invert the rigid transform correctly:

Simply negating the supplied translation is insufficient when the rotation is not identity.

Worked geometric example

Assume a hypothetical joint position of (100, 50, 2000) mm in the depth frame. For this example only, let rotation be identity and let depth-to-color translation be (30, 0, 0) mm.

These numbers illustrate the calculation; they are not calibration values for a particular device. Use the calibration supplied or measured for your actual camera arrangement.

Reject nonfinite coordinates. For the conventional pinhole model used here, also reject points whose color-camera Z coordinate is zero or negative. The near-zero rejection threshold should reflect the application's units and supported geometry.

4. Project into the exact RGB image geometry

For an undistorted pinhole image with negligible skew, project the transformed point using the color-camera intrinsics:

Here, fx and fy are focal lengths in pixels. The principal point is (cx, cy), also in pixels. Position units cancel through the coordinate ratios, provided those position coordinates use the same unit.

Continue the example with an image measuring 1280 × 720 pixels, focal lengths of 900 pixels, and a principal point at (640, 360) pixels:

Keep floating-point coordinates until rasterization. Repeated rounding during transformation, projection, and scaling can introduce avoidable jitter.

If the original joint coordinates were incorrectly treated as color-camera coordinates, the horizontal result would be 685 pixels. The missing translation therefore causes a 13.5-pixel error in this example.

After projection, check the returned validity flag if you used a calibration helper. Separately check image bounds:

These checks establish whether a marker can be drawn inside the image. They do not prove that a joint is visible: a tracked joint may be behind another person or an object.

5. Draw the skeleton without changing its geometry

Use the RGB image as the background. Draw accepted joint markers, then connect joints according to the tracker's documented skeleton topology. Maintain a separate joint collection for each tracked body identifier so that bones never connect different people.

The following is SDK-neutral pseudocode. Its operations describe the pipeline rather than names of callable library methods:

for each tracking result:
    capture = retained source capture for this result
    canvas = writable copy of capture.color_image

    for each body in tracking result:
        projected = empty joint lookup

        for each joint in body:
            if joint confidence is below application threshold:
                continue

            point_color = transform joint into color-camera coordinates
            if point_color is nonfinite or behind the camera:
                continue

            pixel, valid = project using matching color calibration
            if not valid or pixel is outside canvas:
                continue

            projected[joint.identifier] = pixel
            draw marker on canvas at pixel

        for each bone in documented skeleton topology:
            if both bone endpoints exist in projected:
                draw line between the projected endpoints

    publish canvas with source capture identifier and timestamp

Omitting a bone whenever an endpoint is rejected is a conservative default. More elaborate renderers can clip segments at image boundaries, but should not replace missing endpoints with (0, 0).

Confidence is not the same as visibility. A tracker may estimate an occluded joint confidently. If you need occlusion-aware rendering, compare against a registered scene-depth representation using a defined tolerance and visibility policy. Sparse skeleton projection alone cannot provide that test.

Apply display scaling once

Draw in original RGB pixel coordinates before resizing, or apply the exact display transform to every projected point. Do not mix both approaches.

For a letterboxed display:

A 1280 × 720 image fitted inside a 960 × 600 display uses a scale of 0.75. Its displayed height is 540 pixels, leaving 30 pixels of padding above and below. The example joint appears at approximately (524, 317) display pixels.

Mirror the background and annotations together. The preview-overlay debugging guide provides a useful separation between image coordinates and display coordinates.

6. Verify alignment and bitmap handling

Validate one stationary subject before adding multiple bodies or fast movement. Test near the image center and toward the edges, then repeat at different distances within the supported tracking range.

Use the error pattern to choose the next test:

For bitmap output, verify the actual source pixel format. An RGB stream may arrive in another channel arrangement or as encoded data. Decode or convert it before drawing, using the acquisition interface's documented format description.

Respect row stride rather than assuming tightly packed rows. For example, a packed 1280-pixel-wide image with four bytes per pixel requires 5120 bytes of image data per row; the actual buffer may include padding.

Retain or copy buffers before their acquisition owner releases them. Publish a completed frame rather than allowing acquisition and display threads to modify the same pixels concurrently. If you need clean recordings, keep the original image separate from the annotated bitmap.

Register masks separately when the task changes

A body-label mask contains pixel labels, not a small list of joint positions. Registering it to RGB requires the corresponding depth and camera geometry. Resizing the mask to color-image dimensions does not account for parallax.

Use the documented depth-aware label transformation if the acquisition interface provides one. Otherwise, the registration pipeline must handle projection, overlapping surfaces, and unmapped output pixels explicitly. Preserve categorical labels with nearest-neighbor handling or an equivalent label-safe operation; ordinary interpolation can create values that represent no body.

Mask registration also has visibility limits. Regions visible to the color camera may be hidden from the depth camera, leaving genuine gaps. The image overlay and mask guide covers compositing after registration; transparency does not correct misalignment.

Select hardware without assuming tracking compatibility

For an existing integrated depth-and-color device, correct the software pipeline before replacing hardware. Adding a separate camera creates additional requirements for extrinsic calibration, synchronization, mounting stability, and capture association.

The 3D sensors category includes several sensing approaches. The MV-DPS200P-02 is a projected structured-light 3D camera. Its catalog identification alone does not establish compatibility with a skeletal tracking library.

For a separate color stream, the area scan cameras category provides camera options. The MV-CU020-90GC is a color, global-shutter GigE area scan camera. That description does not establish synchronized operation with the depth camera or support in your application.

Use the camera finder to narrow imaging requirements. Before ordering, confirm acquisition support, documented timing interfaces, calibration access, working range, and explicit tracking-library compatibility. Neither example above is presented as a turnkey body-tracking system.

Final checklist and next step

• Confirm the preview background is a decoded color frame, not a depth palette.
• Associate each skeleton with its source capture and corresponding color exposure.
• Verify coordinate units, axes, and transform direction.
• Use calibration for the active image mode and distortion state.
• Reject invalid, low-confidence, and out-of-bounds joints before drawing.
• Keep body identifiers and bone connections separate.
• Apply resizing, padding, cropping, and mirroring consistently.
• Verify bitmap channel order, row stride, and buffer ownership.
• Test stationary alignment before evaluating moving subjects.

If you are selecting a new acquisition system, start with the camera configuration step. For integration questions, contact the applications team with your image modes, coordinate conventions, timing records, and a paired RGB-and-joint capture.