Pointcloud filter slows depth camera FPS: fix the CPU load

Your depth camera streams 30 fps clean. Add the pointcloud filter and the depth topic sinks to 8 fps, the color topic vanishes, and the cloud never appears. On an embedded ARM board this is not a mystery: alignment, deprojection and color mapping are per-pixel CPU work, and the whole frame budget at 30 fps is 33 ms. Here you will measure which stage eats that budget, see the arithmetic that predicts what each fix buys, and finish with a decision per case: cut pixels, build the SDK with GPU acceleration, or move the pipeline to a vision controller.

The symptom: 30 fps depth, 8 fps with the pointcloud on

Launch the driver on its own and both base topics behave: /camera/color/image_raw and /camera/depth/image_rect_raw stream at the rated 30 fps in rviz. Add the pointcloud filter to the same launch command (filters:=pointcloud) and three things happen at once: the depth topic drops to roughly 8 fps, the color topic stops publishing, and the cloud topic /camera/depth/color/points never arrives. Route the camera through the wrapper's separate RGB-D registration launch instead, and the registered cloud /camera/depth_registered/points limps below 1 Hz, the viewer often reporting plain "no new messages".

Read that pattern before you change anything. A topic that publishes slowly is a CPU-bound pipeline. A topic that publishes nothing, or reports no new messages, is a node too starved to service its callbacks. Both symptoms share one cause here: the per-pixel math of alignment, deprojection and color mapping running on an embedded ARM core with no GPU help. A pure stall with a comfortable CPU is a different failure — that one starts at Depth camera pointcloud stops publishing: diagnose the pipeline load.

In order of probability for this symptom:

1. Alignment, deprojection and color mapping on the CPU, with no GPU acceleration in the SDK build.
2. A binary wrapper package that was built without the vendor's GPU acceleration for your board.
3. An SDK and wrapper pair that do not match, which stalls streams entirely.
4. Transport bandwidth, which slows the base stream too — if the first test below already reads low, start at frame drops on embedded boards.

What the pointcloud stage costs

Three budgets fall out of the first line: 30 fps = 33.3 ms, 15 fps = 66.7 ms, 8 fps = 125 ms per frame.

Why the cloud is the heavy stage: every depth pixel becomes a vertex with x, y, z float math; an aligned depth frame adds one resample per pixel; every vertex then takes a color lookup. Typical packing is 16-32 bytes per vertex — x, y, z as float32, packed RGB, padding — so plan on 20 bytes. All of this scales with pixel count, close to linearly, plus a small fixed overhead per frame.

Worked example: a 92 ms stage at 640×480

Take the 640×480 profile on both sensors at a requested 30 fps, and measure twice.

• Depth alone holds 30 fps: 1000 / 30 = 33.3 ms per frame.
• With align_depth:=true and filters:=pointcloud, depth measures 8 fps: 1000 / 8 = 125 ms.
• Added stage cost: 125 − 33.3 ≈ 92 ms per frame, which is 92,000 µs ÷ 307,200 px ≈ 0.30 µs per pixel on this board and build.
• Cloud payload: 640 × 480 = 307,200 vertices; × 20 B ≈ 6.1 MB per frame; × 30 fps ≈ 184 MB/s before serialization, and middleware copies per subscriber push real traffic toward ~550 MB/s.
• The image streams are light by comparison: aligned depth 640×480 × 2 B ≈ 0.61 MB, RGB 640×480 × 3 B ≈ 0.92 MB, about 46 MB/s together at 30 fps.

That 0.30 µs per pixel is the planning constant. To hold 15 fps, the stage must fit inside 66.7 − 33.3 ≈ 33 ms, which at 0.30 µs per pixel allows about 110,000 pixels. At 320×240 = 76,800 px the stage costs ≈ 23 ms, the frame ≈ 56 ms, and the pipeline sustains ≈ 18 fps. Commit: a CPU-only ARM path at 640×480 with a colorized cloud tops out at the measured 8 fps; 30 fps does not arrive without the GPU.

These are estimates for one board; run the two-rate measurement on yours and reuse the same arithmetic. Record wall-clock rate, not CPU time — CPU time vs wall-clock time in vision pipeline benchmarking explains the difference that matters here.

Low-cost wins first: pixels, rate, and the color path

• Cut pixels before anything else. 1280×720 → 640×480 is 921,600 → 307,200 px, exactly 3× less stage cost; 640×480 → 320×240 is another 4×. Use the smallest depth resolution the task tolerates, and if the delivered depth image comes out smaller than requested, check decimation first — depth image resolution smaller than expected. Where the task allows it, crop to the region of interest at the source rather than downstream — camera ROI cropping at the source.
• Lower fps to cut load, not to rescue a stage that already misses its budget. At 15 fps the budget is 66.7 ms and a 92 ms stage still misses. Rate changes cut how often the stage runs, which helps thermals and the other nodes, but only fewer pixels or the GPU shortens the stage itself.
• Skip passes you do not use. A colorized cloud needs depth-to-color alignment; an xyz-only cloud for occupancy, volume or picking needs neither alignment nor color lookup. Removing a pass removes a full-image resample and a per-vertex lookup — price each one with step 3's measurement.
• Check that a lowered rate actually took. If you set depth_fps:=15 and the stream still runs at 30, the driver fell back to the nearest profile both sensors support. Measure the delivered rate, set both streams to a combination the camera documents, and re-check the delivered resolution.

Build the SDK with GPU acceleration, then the wrapper

Binary wrapper packages from a distro's repository are built for the lowest common denominator and usually ship without the vendor's GPU acceleration enabled for your specific board. The per-pixel stages then stay on the CPU no matter what silicon the board carries. The fix is a build order:

1. Build the camera SDK from source with its GPU-acceleration option enabled for your platform.
2. Build the wrapper release whose notes document that exact SDK build. A wrapper pointed at a newer SDK than it supports stalls streams in exactly the way this article started with.
3. Re-run steps 2-4 of the test section. Expect the depth topic to return to its no-pointcloud rate at 640×480 and 30 fps. Whatever gap remains is copies and serialization — GPU accelerated vision streaming: trace the video pipeline shows how to follow the frames through it.

Record the SDK and wrapper versions next to the deployment image. An OS update that quietly swaps one of them will reproduce this stall months later, on hardware and settings you had already signed off.

When the platform is the wrong tool

If the requirement is a continuous 30 fps color-mapped cloud and the board has no GPU acceleration path, no launch argument will save it: the worked example caps a CPU-only path at the measured 8 fps at 640×480. Decide from the number the process actually needs:

• Visualization, mapping, occupancy up to ~10 fps — stay on the module at 640×480, publish the cloud at its honest rate, keep depth at 30.
• A 15 fps xyz cloud — drop to a 320×240-class profile on the module, or move the pipeline to an x86 controller and keep the resolution.
• 30 fps color-mapped continuously — a GPU-accelerated build on hardware that has one, or an x86 vision controller sized for streams plus cloud.

When the answer is compute, the vision controllers category is the hardware that carries this pipeline. Two controllers that fit the job:

The MV-VC5525-256G70-NN carries 32 GB of memory, 5+ LAN ports with PoE camera ports and 8 built-in light channels — one box for a multi-camera RGB-D cell with the cloud pipeline inside it.

The MV-VC4712-256G10-2TH pairs PCIe expansion with 6 built-in light channels, PoE camera ports and 16 GB of memory, leaving room for accelerator or frame-grabber cards on top.

For profile, dent and volume tasks, dedicated 3D sensors shift the work further upstream: a laser profiler returns a measured profile per scan, and logistics-class models report box dimensions and volume, so the host never deprojects a raw stream at all. The 3D camera finder filters them by type and resolution, and more pipeline fixes live on the 3D topic page.

Keep the budget visible after every change

• Re-measure both rates after any change to resolution, fps, filters or subscribers, and recompute the per-stage milliseconds. The two-rate method costs two minutes and catches regressions before they reach the cell.
• Keep the SDK and wrapper pair pinned, with both versions noted beside the deployment image.
• Bake the working resolution and fps into the launch file so a re-run cannot fall back silently.
• Load-test on the target board with the full node set running: a pipeline that holds 30 fps alone can lose half of it once detectors and extra subscribers start.
• Keep the µs-per-pixel constant and the stage table in the project folder; when someone adds a feature, one test tells you which stage paid for it.

If the table says the CPU is the bottleneck and the process cannot live with 8 fps, send us the stage table through the quote request and we will size a controller around the measured numbers, not around a guess.