Normalize a depth map for JET colormap display, keep the values

A depth map that renders as one flat color tells you nothing about the scene. The camera is returning good values; the display pipeline is throwing them away. The fix is a three-step display routine: a min-max stretch, a cast to 8-bit, then the JET colormap, all run on a copy of the array while the original stays untouched for measurement. Do it the other way around and the array you inspect, deproject or hand to the robot is quietly degraded. This guide gives the formula, shows why coarse integer disparity gains the most from the stretch, works a 96-step example end to end, and fixes the mirrored rectified output that makes detection overlays drift off their objects. The same routine covers stereo, structured-light and time-of-flight depth output alike.

What the min-max stretch actually does

cv2.applyColorMap assigns one color per 8-bit input value: index 0 takes the first entry of the ramp, index 255 the last. Depth arrays rarely fit that input. Stereo pipelines return disparity, and most depth cameras return 16-bit millimetre integers or floating-point values. Feed such an array to the colormap unchanged and you get an error, or a picture that is nearly one color.

cv2.normalize with cv2.NORM_MINMAX fixes the scale, not the values:

The frame minimum becomes 0, the frame maximum becomes 255, and everything between moves linearly. Three details decide whether that helps or hurts:

• Run it on a copy. Passing None as the destination sends the result to a new array; pass your source array instead and the measurement data is overwritten.
• Set the output type. Left unset, the result keeps the input type, so the 0-to-255 values sit in a 16-bit container. Pass dtype=cv2.CV_8U, or cast afterwards, so the colormap stage receives plain 8-bit values.
• Keep the original within reach. Every millimetre you report comes from the untouched array, never from the display copy.

depth_u16 = depth_raw                            # uint16, shape (480, 640), millimetres
disp = cv2.normalize(depth_u16, None, 0, 255,
                     norm_type=cv2.NORM_MINMAX, dtype=cv2.CV_8U)
disp = cv2.flip(disp, 1)                         # match the video orientation, next section
jet = cv2.applyColorMap(disp, cv2.COLORMAP_JET)  # uint8, shape (480, 640, 3)
# test: depth_u16.dtype is still uint16 and depth_u16[240, 320] returns the same
# millimetre value before and after these calls; jet.shape == (480, 640, 3)

A black image or a conversion error instead means the array type is the problem; the depth image 16UC1 display fix walks through those conversion errors. The same copy-first discipline covers any visual change, including inverting the view: inverting a depth image display without changing measurements applies it step by step.

Fixed range or per-frame range

The minimum and maximum can come from each frame or from fixed constants, and that choice decides what the colors mean.

Per-frame normalization gives every snapshot full contrast, but the mapping moves. The same wall at 3 m turns green, then yellow, whenever something nearer enters the frame, so a recording looks like the scene is pulsing. Use it for single captures and quick notebooks.

A fixed range maps one chosen span to 0-to-255 as constants, clipping values outside it to the ends. Colors become a distance legend: every frame and every session comparable. Contrast is lower when the scene fills only part of the span, which is the trade-off for stability. Live video, recordings and anything compared across time want the fixed map.

If you feed colorized frames to change detection, you are diffing mapping artifacts along with the scene; the depth map change detection method diffs raw depth values instead.

A fixed map also reads as a legend without extra work: a time-of-flight sensing camera streams depth as a dense millimetre array, and each color band becomes a distance band on the wall of the cell.

Why coarse disparity gains the most

Disparity is the horizontal displacement, in pixels, between the same feature in the two rectified images; larger disparity means a nearer point, and depth follows inversely. With subpixel disabled, d moves in whole pixels, so Δz is the staircase between neighbouring depth values.

Here f is the focal length in pixels, B the baseline in millimetres and d the disparity in pixels; the stereo disparity scale and calibration guide works the scale factor through in detail.

Integer disparity also starves the colormap. A pipeline with 96 integer steps puts at most 96 distinct values into the 256 available indices, so without a stretch most of the JET ramp never appears and adjacent steps share a color. Normalization spreads those steps across the full ramp and separates them.

A stereo depth camera delivers exactly this kind of disparity array; the MV-DLS300P-04 pairs its stereo reconstruction with a galvanometer-projected laser line for profile work.

Worked example: a 96-step disparity map covering 0.53 to 10 m

Inputs:

• Stereo pipeline with subpixel disabled: integer disparity, 96 quantization steps, values 0 to 95.
• Optics for the arithmetic: f = 500 px, B = 100 mm, so f × B = 50,000 mm·px.
• Frame content: disparity from 5 at the far wall to 95 in the near field.

The rule in words: the colormap shows one color band per 8-bit value, so normalization trades the frame's disparity span against the full ramp, and the depth staircase behind the colors follows Δz ≈ f × B / d².

1. Depth span: z(95) = 50,000 / 95 = 526 mm and z(5) = 50,000 / 5 = 10,000 mm, so the frame covers 0.53 to 10 m.
2. Display levels per disparity step after the stretch: 255 / (95 − 5) = 255 / 90 ≈ 2.83. Every quantization step becomes roughly 2.8 color levels, a visible band of its own.
3. Without the stretch: values 5 to 95 occupy 91 of the 256 colormap indices, about 36 percent of the ramp. Far objects sit near index 5 and the rest of JET never shows.
4. Depth staircase: from d = 10 to d = 11, depth moves from 50,000 / 10 = 5,000 mm to 50,000 / 11 = 4,545 mm, a jump of 455 mm or 9.1 percent of the reading. At d = 50 the step is 1,000 − 980 = 20 mm, about 2.0 percent. At d = 95 it is 526 − 521 = 5.5 mm, about 1 percent.
5. With 1/32 subpixel enabled, the same pipeline returns 96 × 32 = 3,072 levels; the near-range step drops to 455 / 32 ≈ 14 mm, and the 256-level display ramp becomes the coarser of the two.

Result: the stretch turns a picture that used about 36 percent of its colors into one with 2.83 display levels per disparity step, while the staircase behind it runs from 455 mm at 5 m down to 5.5 mm at 0.53 m.

Decision: normalize the display copy, because the raw values hide real steps and the stretch shows all of them; keep the measurement on the original disparity array, because the colormap adds no precision. If close-range terraces matter to a gripper, enable subpixel and set the depth resolution at capture time, as the change 3D camera depth resolution guide covers, rather than reading finer bands off the image.

The flip and overlays that follow the object

Some stereo pipelines deliver the rectified depth stream mirrored left-right relative to the raw camera view. The detector runs on the raw mono frame, so its boxes land in one orientation while the depth picture lives in the other; the classic symptom is an overlay that follows for a moment and then drifts off the object.

Fix it once, on the display copy: cv2.flip(disp, 1) mirrors the array about the vertical axis, flip code 1. Apply it before you draw anything, then keep one orientation for everything. Boxes, labels and the image all live in the flipped frame, or all in the unflipped one; flipping the picture but not the boxes recreates the drift.

Detection geometry needs the same care. If the network consumed a square, non-uniformly scaled frame, its boxes are in network coordinates, and mapping them back takes separate axes:

With a stretched input the two ratios differ, and a single shared scale factor drifts boxes progressively toward the frame edges. If the input was letterboxed instead of stretched, add the letterbox offsets to the same two lines.

Rectification quirks beyond the mirror are covered in the stereo disparity map troubleshooting guide. When the overlay must sit on the color frame instead, misalignment there comes from depth-to-color calibration, which the stereo RGB calibration guide treats in detail. The measurement itself, such as extents read from the depth under each box, belongs to the original array, as the 2D bounding box to 3D guide shows.

Invalid pixels and outliers that flatten the picture

Per-frame min-max is only as good as its extremes, and two common array contents corrupt them.

Invalid returns: many depth cameras write 0 where nothing came back. Treat 0 as a real value and the frame minimum drops to it, pushing the nearest real surface up the ramp and the color legend off its distances. Compute the minimum over valid pixels only, excluding 0 and whatever invalid marker your pipeline uses.

Outliers: one bad stereo match at 60,000 mm, in an array that tops out at 65,535, stretches the span so far that the real scene collapses. A scene spanning 600 to 10,000 mm with a single 60,000 mm stray pixel gets (10,000 − 600) / (60,000 − 600) = 15.8 percent of the ramp, about 40 of 255 color levels for everything you care about. Clip the range at the 1st and 99th percentiles of the frame instead of the hard minimum and maximum, and one stray pixel stops owning the picture.

Point clouds, offsets and extents all start from the original array: the depth map to point cloud deprojection guide builds the cloud from raw depth values, and feeding it the 8-bit display copy would quantize every surface into 256 shelves.

Checks before you ship the visualizer

• Measure on the original array only; the display copy holds 256 levels across the chosen span.
• Read depth_u16[y, x] before and after the display calls; identical values prove the copy stayed a copy.
• Flip once, on the display copy, before overlays, and confirm the orientation against an asymmetric scene.
• Map detection boxes back with separate horizontal and vertical scales whenever the network input was stretched.
• Fix the range for live video and recordings; keep per-frame min-max for single snapshots.
• Exclude invalid pixels and clip to percentiles before computing the frame minimum and maximum.
• Treat display problems with display changes and depth problems with subpixel or calibration.

The decision and the hardware

Decision in one pass: normalize a copy with a min-max stretch, cast to 8-bit, apply the JET colormap; fix the range to the device's operating span for anything live; flip the copy once to match the video orientation; read every millimetre from the original array.

Every sensor type in this guide is listed under the 3D sensors category: stereo and structured-light cameras for disparity-style arrays, time-of-flight cameras for dense millimetre depth. A projected structured-light camera such as the MV-DPS200P-02 reconstructs depth from a projected pattern and feeds the same display routine unchanged.

The camera finder narrows the depth camera range, the vision system configurator pairs the camera with lens, light and controller, and the 3D vision topic page gathers the related guides. When the choice between disparity, structured light and time-of-flight is still open for the cell, the contact page reaches the applications team.