RGBD stream RAM usage grows until the node stops: find the leak

The stream runs clean all morning. Then, roughly an hour in, the point-cloud topic goes quiet and the launch log closes with a process-has-died entry reporting exit code -9. Nothing in your code called exit: a negative code is a signal, and 9 is SIGKILL, delivered by the kernel's out-of-memory killer after RAM usage climbed frame by frame to the top of the machine. This page is for anyone whose depth node publishes a registered RGBD point cloud, a topic such as /camera/depth_registered/points, in ROS or in a hand-rolled pipeline. You will measure the leak in retained bytes per frame, use that number to name the guilty stage, apply the matching fix, and prove the result with a soak test that has a numeric pass line. The camera is rarely the culprit; the retention is.

What exit code -9 actually tells you

The negative code narrows the problem before you open an editor. The kernel chose your process, sent SIGKILL, and the launcher reports signal 9 as exit code -9; your own code cannot produce it. Confirm the mechanism with two checks. Run dmesg and look for the out-of-memory entry: it names the killed process and prints its anon-rss at the moment of death, which is exactly how full the machine was. Then check where the process was when it died: a node killed while parked in a wait-for-frames call (rs_wait_for_frames in one SDK's API) was idle at the moment of the kill, so the leak lives in the loop around it, in buffers queued, stored or copied between frames.

Separate a leak from churn before changing anything. Allocator warm-up and caches raise resident memory for the first minutes, then it plateaus, and that is healthy. A leak is linear: about the same megabytes every 10,000 frames, hour after hour, until the kill.

Measure the leak rate first

You need one number: retained bytes per frame. Record the node's resident set size (VmRSS in the kernel's per-process status file, or the RES column in htop) against the frame counter every 10,000 frames. Two points give a slope; five give confidence. Count frames in one place consistently, in your callback or from the driver's counter, and keep the timestamp in the same log line.

With RSS in bytes, leak_per_frame comes out in bytes per frame, and time_to_kill predicts when the next crash lands. Measure any downstream consumer too: a viewer such as rviz keeps its own display history and can be the hog you are blaming on the node. If the stream stops while VmRSS sits flat, that is a different fault: pipeline load or a driver problem, covered in when a point cloud stops publishing under load and finding where depth frames stop on Linux.

Worked example: 29.4 GB gone in 100,000 frames on a 32 GB host

The inputs, as measured on a machine showing the failure: 32 GB of RAM; the process was killed at 95% of that, so about 30.4 GB resident; the frame counter passed 100,000 clouds before death; startup VmRSS assumed at 1 GB, so record yours; frame rate 30 Hz, the common depth rate, so record yours if you throttle.

The rule in words: retained bytes per frame equals memory growth divided by frames processed. The arithmetic:

• Growth: 30.4 GB - 1.0 GB = 29.4 GB = 29.4 × 10^9 bytes
• Retained per frame: 29.4 × 10^9 / 100,000 = 294,000 bytes ≈ 0.3 MB per frame
• One 640 × 480 XYZRGB cloud: 640 × 480 × 32 bytes = 9.83 MB
• A whole cloud leaked per frame would die after 29.4 × 10^9 / 9.83 × 10^6 ≈ 2,990 frames, about 100 s at 30 Hz
• Observed survival: 100,000 / 30 ≈ 3,333 s ≈ 56 min, the dies-after-an-hour signature
• Cross-check with the second formula: (32 - 1) × 10^9 / (294,000 × 30) ≈ 3,515 s ≈ 59 min

The retained 294 kB is 294 / 9,830 ≈ 3% of one cloud, and that ratio is the decision:

• A few kB per frame: allocator churn, not a leak; monitor only.
• A few percent of a cloud: small objects leak, such as queue bookkeeping, per-callback copies and metadata containers.
• About one cloud per frame: an unbounded queue of clouds.
• Several clouds per frame: publisher buffering for a subscriber that cannot keep up.

Pass line for the fixed node: growth under 1% of the budget per 100,000 frames, here under 0.304 GB ≈ 304 MB, which is under 3.0 kB per frame.

Likely causes, most probable first

1. An unbounded queue behind a consumer that cannot keep up. Time one callback or processing step. At 30 Hz the budget is 33.3 ms per frame; if your average exceeds it, clouds arrive faster than you drain them and the transport holds the difference. Set queue depth to 1 on the publisher and subscriber sides so the oldest cloud is dropped, and process every Nth frame if the budget cannot be met. The code-level pattern is the same as in bound the frame queue and reuse frames.

2. A container that grows once per frame. Search the node for vectors, maps or lists appended in the frame loop: result caches, pose histories, debug overlays. Print their length at shutdown; anything proportional to frames processed is the leak. Cap them or replace them with a fixed-size ring. The video-loop variant is the per-frame image leak, and the inspection-loop variant is inspection-loop memory growth.

3. Buffers referenced after use. Run a heap profiler for ten minutes, Massif from valgrind or heaptrack, and read the allocation stacks. A leak shows one stack whose live total grows in lockstep with the frame counter. Reset the shared pointer once the cloud is published, and reuse image buffers instead of allocating per frame.

4. A link that resets and a driver that rebuilds. Count device resets in dmesg over an hour. Every teardown and re-initialization allocates, and some stacks never return all of it, so a reconnect every few minutes looks exactly like a slow leak. Fix the physical link first; the next section covers it.

5. The SDK or driver build. Run the vendor's capture sample alone, publishing clouds with none of your downstream code, and watch VmRSS for an hour. If it still climbs, the defect is below your code: update the SDK and firmware, and rebuild the driver stack as in rebuilding the driver stack after a firmware update. The cross-check is decisive: the identical node with a flat slope on a different host build indicts the build, not your logic.

Fix the physical link before blaming the code

A USB3 link on a flexing, marginal or over-long cable drops and renegotiates, and dmesg logs each reset. Every reset makes the driver destroy and rebuild the device context, and the allocations from those cycles pile up. The pattern to watch for is resets every few minutes plus a leak slope that no code change flattens. Replace the run with cable built for the motion: the camera cables range includes drag-chain and high-flex builds, and for a moving run the LS-CBL-0219 is a 3 m high-flex USB3.0 camera cable made for exactly this duty. Add strain relief at both ends, then re-measure the slope. With the resets gone, a healthy pipeline returns to churn-level growth.

Depth cameras that publish these streams

If the rig is being rebuilt anyway, registered RGBD point clouds come from two camera families. Projected structured-light cameras such as the MV-DPS200P-02 pattern the scene and triangulate depth; time-of-flight cameras such as the MV-DT01WMU measure the return delay of emitted light. Both feed the same host-side pipeline, and the memory discipline above applies to either. The 3D sensors category holds the full range, and the camera finder filters it by type and interface.

Prove the fix with a soak test

Re-run the measurement with the fix in place and judge it against the pass line from the worked example.

1. Record VmRSS at startup.
2. Run at production rate for at least one shift: 8 hours at 30 Hz is 864,000 frames.
3. Record VmRSS at the end and scan dmesg for out-of-memory entries.
4. Pass: growth under about 304 MB per 100,000 frames, under 3.0 kB per frame on the 32 GB host, with a clean dmesg. Fail: the slope survives, and the per-frame arithmetic names the remaining stage.

Watch frame rate as well. As RAM fills, the node slows before the kill, so a rate that holds flat through the soak is supporting evidence. Alerting helps between releases: a VmRSS alarm at 70% of RAM, which is 22.4 GB on the worked example's host, fires about 20 minutes before the kill at that slope.

Prevention checklist

• Queue depth 1 everywhere. A deeper queue stores clouds no inspection or guidance loop can act on; it is deferred waste.
• No appends without eviction. Any container touched in the frame loop gets a cap or a ring; histories live a fixed number of frames.
• Buffers reused, not re-allocated. Preallocate at startup; per-frame allocation survives only where a profiler shows it is clean.
• VmRSS alarm at 70% of RAM. Memory pressure announces itself long before the OOM killer does.
• A soak gate on releases. One shift at production rate, with the slope number recorded in the release notes.
• SDK kept current. A defect below your code cannot be fixed above it; a vendor sample that leaks convicts the build.

More depth-pipeline fault-finding lives on the 3D topic page. Building or refreshing the rig? The camera configurator pairs the camera with the rest of the chain, and if a slope survives every test above, bring the slope and the profiler output to the contact page: those two artifacts usually end the diagnosis in one reply.