Point cloud null after depth-to-colour alignment: fix frame lifetime

Your script enables colour and depth at 1280×720, aligns depth to the colour image, calculates the points object and calls get_vertices — and every so often the run stops with null pointer passed for argument "frame". Run it again and it works; change the camera resolution and the failures crowd in. The fault is not in get_vertices and not in the alignment maths. It is a frame-lifetime problem, and this guide gives you the loop structure that fixes it, the arithmetic that explains why resolution changes make it worse, and the tests that separate it from a failing USB link.

Why does the point cloud come back null after aligning depth to colour?

The null pointer almost always means the native frame behind the points object was freed before get_vertices ran: MATLAB collected the aligned frameset or the depth frame while the points object still pointed into it, so the wrapper passed a null frame argument.

The error surfaces at get_vertices because that is the first call to touch the dead memory: the wrapper hands the points object's handle to its native layer, and the layer rejects a null frame rather than reading through it. Two objects therefore have to outlive the whole call chain: the frameset the aligner's process method returns, and the depth frame pulled from it. If either is a temporary — created inline, never assigned to a named variable — MATLAB may free it the moment the statement ends, and the null arrives with no hint about which object died.

Why does it work without the align step and fail only sometimes with it?

Without alignment the depth frame lives inside the pipeline's frameset variable, which your code holds for the whole iteration; with alignment the frameset that process returns is a new native object, and if you never store it, MATLAB frees it at the end of the statement.

That is why the unaligned path never fails: get_depth_frame returns a child of a frameset you already own, so the depth frame lives exactly as long as your variable does. Alignment inserts one native object with no owner unless you give it one, and a chained call such as pc.calculate(alignTo.process(frames).get_depth_frame()) is the classic shape of the bug — by the time calculate executes, the frameset temporary is already eligible for collection.

Collection timing is also why the fault is intermittent instead of constant. MATLAB collects native wrappers opportunistically: on a quiet session a temporary can survive longer than its own statement, which is why the script sometimes runs clean for a whole afternoon. Treat any unassigned intermediate in this chain as a dangling handle, not as working code that happens to pass.

What loop structure keeps every handle alive?

Create the aligner and the points calculator once, store every intermediate frame in a named variable in the same scope, and call get_vertices before any of those variables is reassigned; the method names — wait_for_frames, process, get_depth_frame, calculate, get_vertices — stay exactly as your script already has them.

% pipe, cfg, alignTo and pc are created once, before the loop,
% exactly as in your current script - those lines do not change.
expected = 1280 * 720;                    % 921600 points at this resolution
for i = 1:1000
    frames   = pipe.wait_for_frames();    % frameset: colour + depth
    aligned  = alignTo.process(frames);   % 1 keep the aligned frameset
    depth    = aligned.get_depth_frame(); % 2 keep the aligned depth frame
    pnts     = pc.calculate(depth);       % 3 points object for this frame
    vertices = pnts.get_vertices();       % 4 921600x3 xyz matrix
    assert(size(vertices, 1) == expected, 'short vertex buffer');
end

The method calls are the wrapper's, in the same order the failing script uses them; the variable names are this guide's own. Reuse one points calculator across the loop instead of constructing a new one per capture — it recalculates from the frame you pass it and avoids a native allocation on every iteration. What calculate does between the depth frame and the vertices is the deprojection step described in depth map to point cloud: the deprojection calculation explained, and which intrinsics govern the aligned output is set out in aligned depth streams: data layout and which intrinsics apply.

Run this loop as the regression test: expect the unfixed chain to stop inside a few hundred iterations at get_vertices, and the restructured form to complete all 1000 captures with zero exceptions.

Why does changing resolution bring the fault back?

A resolution change renegotiates both streams: the first framesets after the restart can arrive with no usable depth frame, and the points buffer must grow from 307200 points at 640×480 to 921600 points at 1280×720 — exactly when stale handles and reallocation hit hardest.

Until the camera settles, an empty depth frame fed to calculate produces a points object with nothing behind it, and get_vertices throws the same null. So stop the pipeline, rebuild the configuration with the new resolutions, restart, and discard the first few framesets before resuming the capture loop. Reconfiguration also churns memory, which makes the collector run more often: intermittency worsens right after every change even when the lifetime bug has been dormant for hours. A cloud that arrives but shows holes where surfaces should be is a different fault, covered in missing points in a 3D camera point cloud: fix the depth holes.

The vertex buffer scales with the pixel count, so budget for it:

Which failures are the SDK's rather than yours?

Run the vendor's current SDK release before rewriting anything: the identical script that failed on an older wrapper stopped reproducing on a newer one, and a repro that a version bump removes is wrapper-bug evidence, not a flaw in your loop.

Pin the version that passes the regression test, record it in the project file, and retest after every SDK update — a fix should never rest on an untracked upgrade. A null that appears when you query intrinsics instead of vertices, with the resolution request coming back as zero, is the sibling fault described in camera fx returns 0: fix the null resolution request. A crash inside the alignment stage itself, rather than at get_vertices, belongs to undistortion pipeline crash when aligning depth to color.

When is a marginal USB link the real cause?

A link that drops framesets fails earlier in the chain than get_vertices: wait_for_frames throws a timeout, or the frameset arrives with no depth frame, so the empty-frame path — not the lifetime path — is your USB signature; shorten the cable, move to another controller and retest.

The stream arithmetic tells you why the link is stressed. At 1280×720 and 30 fps, 16-bit depth is 1280 × 720 × 2 = 1843200 bytes per frame, or 1843200 × 30 ≈ 55.3 MB/s; 8-bit RGB colour is 1280 × 720 × 3 = 2764800 bytes per frame, or ≈ 82.9 MB/s. Together that is about 138 MB/s, which no USB 2 link carries in practice — this class of camera belongs on USB 3 with a short, shielded cable. What a link must carry, and where extension cables start to fail, is set out in USB cable length limits for 3D cameras: keeping the data link alive.

How do you verify the fix holds frame after frame?

Instrument the loop to run 1000 consecutive captures at production resolution and assert on every iteration that the vertex array has width × height rows — 921600 at 1280×720 — with zero exceptions across the run as the pass criterion.

Prevention stays short:

• Keep the aligned frameset, the depth frame and the points object in named variables for the whole iteration.
• Reuse one points calculator; rebuild it only when you rebuild the stream configuration.
• After any resolution change, discard the first few framesets before resuming.
• Guard the loop with a catch that logs the stage and the frame index, so a future failure names itself.
• Pin the SDK version that passes the regression test and retest after every update.
• Assert that size(vertices, 1) equals width × height on every capture.

The decision, case by case: failures at get_vertices with the align step in place are the lifetime bug — restructure the loop and they stop. Failures at wait_for_frames or empty depth frames are delivery problems — repair the link and discard post-change framesets. A repro that a newer SDK build removes is a wrapper bug — pin the passing version. Vertices that arrive but land offset from their colour are geometry, not lifetime — that is calibration work, covered in stereo camera RGB calibration: fix depth-to-color alignment errors; when the RGB comes from a separate camera, the extrinsic set-up is covered in align depth to a third RGB camera: fix missing extrinsics.

If the fault belongs to a project you are specifying now rather than a camera already on the bench, shortlist the hardware first and keep the loop discipline above. The range below spans projected structured light, laser galvanometer stereo and ToF sensing:

To match a camera to your resolution, frame rate and link, start at the 3D sensors category or the camera finder, or step through the configurator from the camera stage:

The rest of the depth-camera fixes sit on the 3D vision topic page. If the null survives the restructured loop, the discarded framesets and a current SDK, send the loop, the resolution and the SDK version through the contact page — free expert support with every order.