3D camera SDK returns an empty depth image buffer: set the schema mask

You request the depth image the same way you requested the amplitude image, and OpenCV throws an assertion instead of showing data. The pattern is common to time-of-flight and structured-light SDKs that push several data types through one frame-grabber class. The camera delivers only what the acquisition schema asks for, and the amplitude image happens to sit in the default set while the distance image does not. This guide is for engineers wiring a 3D camera into OpenCV through an SDK buffer. You will set the schema mask so both planes arrive, prove the buffer is populated before you convert it, scale 32-bit float millimetre values for display without destroying the measurement, and size the bandwidth each stream costs. Every step runs against a live camera in minutes, and none of it needs new hardware.

Why does the amplitude image work but the distance image fail with an assertion?

The amplitude accessor works because the frame grabber's default schema mask already streams the amplitude image and the Cartesian point cloud; the distance accessor finds nothing populated and returns an empty matrix, and OpenCV asserts at the first call that needs pixels.

The wait itself is not the problem. Frames arrive on time and your timeout of 1000 ms never fires, because the camera happily streams its default set. The empty matrix then passes through convertTo without complaint. The assertion fires where pixels are actually needed: imshow checks that the matrix has non-zero rows and columns and aborts. The error text points at OpenCV, but the decision was made at frame-grabber construction.

Name the asymmetry for what it is: amplitude worked by accident of the default, not by symmetry of the API. Any accessor whose stream is missing from the mask returns empty, every time. The amplitude accessor would fail in exactly the same way if you requested only the distance stream.

What is a schema mask, and what does the default contain?

A schema mask is a bitmask you pass to the frame grabber constructor, OR-ing one flag per data stream, and the camera transfers exactly the selected set. The default mask in this SDK family, DEFAULT_SCHEMA_MASK, carries the normalized amplitude image plus the Cartesian cloud and nothing more.

Each flag is a switch for one stream. Typical members of the set are the amplitude image, the radial distance image, the Cartesian x, y and z planes, and a per-pixel confidence value, the one the confidence image bitflag guide decodes. A ToF camera measures distance from the phase shift of modulated infrared light and computes these planes internally either way. The mask decides which ones leave the camera and get populated in your buffer.

That is why the fix lives at construction rather than at the accessor. Declare the union of what your pipeline consumes and nothing more, because every extra plane is payload you pay for twice: once in link bandwidth and once in host memory and copies.

How do you request the depth image and keep the amplitude image?

OR the amplitude and radial-distance flags into the mask at construction — IMG_AMP | IMG_RDIS in this SDK family's naming — and both accessors populate on every frame. The change is one line, and the .empty() check proves it on the first captured frame.

Build the mask from the two flags and pass it to the frame-grabber constructor. The loop below keeps the shape of the failing version; only the construction line changes:

uint32_t mask = IMG_AMP | IMG_RDIS;               // was: DEFAULT_SCHEMA_MASK
auto fg = std::make_shared<FrameGrabber>(camera, mask);

ImageBuffer buf;
if (!fg->WaitForFrame(&buf, 1000)) {              // false = no frame within 1000 ms
    std::cerr << "no frame: check link and schema\n";
    return -1;
}
cv::Mat dist = buf.DistanceImage();               // CV_32F, millimetres
if (dist.empty()) {                               // the test that proves the fix
    std::cerr << "distance stream missing from the mask\n";
    return -1;
}
cv::Mat dist8;
dist.convertTo(dist8, CV_8U, 255.0 / 4000.0, 0);  // alpha = 255 / range_mm
cv::imshow("distance", dist8);

The proof is one print. With the default mask, dist.empty() returns true and the log shows the mask message. With IMG_AMP | IMG_RDIS it returns false on the first frame and the window shows the range image. The rule is symmetrical: request only IMG_RDIS and the amplitude accessor goes empty instead. When the buffer and the Mat must share memory rather than copy, the zero-copy SDK-to-cv::Mat wrap is the pattern that keeps it.

Why does the converted depth image come out flat white?

A direct CV_32F-to-CV_8U cast saturates: convertTo without a scale factor maps every value above 255 to 255, and millimetre distances live far above that, so nearly the whole image clips to white. The fix is one multiplication, alpha = 255 divided by the working range in millimetres.

OpenCV's convertTo computes alpha × src + beta and saturate-casts the result. With no scale arguments, alpha is 1 and beta is 0. The cast maps millimetres one-to-one onto byte counts and pins everything at or above 255 to white.

Worked example: a head with a 4000 mm working range, a scene between 500 and 4000 mm, and one return at 1500 mm.

• Direct cast: meaning survives only below 255 mm — 255 / 4000 ≈ 6.4% of the span — so the display is a white sheet with a thin dark fringe.
• Full-range stretch: alpha = 255 / 4000 = 0.0638 counts per mm, so the 1500 mm return lands at 1500 × 0.0638 = 95.6, printed as 96 counts.
• Scene stretch across 500–4000 mm: alpha = 255 / 3500 = 0.0729 and beta = −0.0729 × 500 = −36.4, so the same return lands at 0.0729 × 1500 − 36.4 = 72.9, printed as 73 counts and clearly separated from the near and far ends.

Keep the CV_32F plane for every measurement and convert only a display copy. A threshold at 1200 mm applied to the float is exact. The same threshold applied to stretched bytes moves with alpha and beta and drifts the moment you change the window. Unit handling for the raw values is its own subject: the depth scale to meters conversion for raw pixel values and depth gray value to mm conversion for 16-bit range images guides cover the constants, and reading a distance value from a depth image pixel without an SDK shows the indexing.

How much bandwidth does each extra stream cost?

Radial distance costs twice the amplitude image per pixel and the Cartesian cloud three times more, because they are 32-bit floats against a 16-bit integer. On a 320 × 240 array that is about 307 kB, 154 kB and 922 kB per frame respectively.

The table assumes a QVGA-class array of 320 × 240 pixels, amplitude at 16 bits and distance and Cartesian planes at 32-bit floats, streamed at 30 fps.

Add up only what you request. The default set, amplitude plus cloud, is 153,600 + 921,600 = 1,075,200 B per frame, or 32.3 MB/s ≈ 258 Mb/s at 30 fps. Amplitude plus radial distance is 153,600 + 307,200 = 460,800 B per frame, or 13.8 MB/s ≈ 111 Mb/s. The cloud alone accounts for 921,600 of the 1,382,400 B three-stream total, two thirds of the payload.

The decision follows the link. A gigabit link moves roughly 115 MB/s of payload after overhead, so even all three streams at 41.5 MB/s fit with margin. A 100 Mb/s link moves about 11 MB/s: amplitude alone at 4.6 MB/s fits, distance alone at 9.2 MB/s fits, amplitude plus distance at 13.8 MB/s does not. The structural win is dropping the Cartesian planes and rebuilding x, y, z from the distance image and the camera model, as the 2.5D height map to 3D point cloud walkthrough shows; that one change cuts the payload by two thirds.

Oversized masks show up later as frame-rate problems. Low frame rate on software trigger: mode and data streams sorts the stream side, and acquisition faster than processing: fix the frame rate gap sorts the downstream side.

If the head itself is also on the table, our 3D sensors line splits by depth technology and the camera finder filters it. The MV-DT01WMU is a ToF sensing camera; the MV-DPS200P-02 plays the same role with projected structured light.

How do you tell an empty buffer from a camera that is not streaming at all?

Test the matrix, not the wait. WaitForFrame returning false means no frame at all arrived within its timeout — 1000 ms in the sketch above — while a true return with dist.empty() still true means frames flow and the mask alone is wrong.

Three states, three causes. A false return from the wait is transport: check link, subnet and power first, the path 3D camera not pingable on a Linux controller: fix subnet and link walks. A true return with an empty distance accessor is the mask; apply the one-line fix above. A true return with populated but implausible values — a hillside where a flat plate sits — is conversion or units: re-check the scaling above and make sure planes were copied out before the SDK reused the buffer, the failure behind stereo frames saved wrong: copy Mat data from the SDK buffer.

Write that three-way check into your first-frame handling once, and every later failure announces its own layer.

Which settings silently override the mask you set?

Configuration that re-opens the camera after your code does will win. A launch file or framework that rebuilds the frame grabber with its own schema replaces your flags wholesale, so construct the grabber once per session and prove the first frame with .empty() before you trust the pipeline.

This exact failure has a walkthrough: ROS launch overrides your camera config: the schema mask wins. The habit to adopt comes from the same family of bugs, flaky test on leftover settings: start from a known state: construct, verify the first frame, then keep the instance until the session ends.

Order matters elsewhere too. Some SDKs lock the pixel format until the scan type is set, which surfaces as another read-only surprise; pixel format locked read-only: set the scan type first covers that sequence, and a resolution request that returns zeros instead of buffers is the sibling case in camera fx returns 0: fix the null resolution request.

What should you check before blaming the camera?

Work the five checks in order — link up, mask declared, accessor matches mask, buffer copied before conversion, display scaled separately — and each failure isolates to one layer in under a minute on a live system.

• Link up: the SDK opens the camera and WaitForFrame returns true; a false is transport, not code.
• Mask declared: the constructor line names every stream your accessors read, amplitude and radial distance at minimum.
• Accessor matches mask: one empty() check per accessor on the first frame; any true names the missing stream.
• Buffer copied: planes leave the SDK buffer before the next WaitForFrame reuses it.
• Display scaled: 8-bit copies carry alpha and beta; the float planes carry the millimetres.

When mask, scaling and buffer handling are settled and you are sizing hardware instead of code, the system configurator steps through camera, lens, light and controller in order, and the 3D vision topic page gathers the other depth-pipeline guides. If a failure does not match the three states above, contact us with the mask line and the first frame's empty() results.