Depth image flickering in rviz: fix the driver first

A depth image or point cloud that flickers in RViz — there one frame, gone the next — almost never means a broken sensor. Frames are failing somewhere between the sensor and the screen, and the failure is usually on the host: an out-of-tree kernel the camera driver was never built for, or a USB3 link that cannot carry the stream you configured. This guide gives you the three flicker signatures to tell apart, the causes ranked by how often each turns out to be the problem, the bandwidth arithmetic that settles the USB question, and a fix order that gets a stable cloud on screen in under an hour.

Read the flicker before you fix it

Whole frames vanishing, surfaces that boil, an overlay that swims: each signature points at a different layer. Sort them out before you swap any hardware.

The cheapest test comes first: open the vendor's own viewer beside RViz. A cloud that flickers there too is failing upstream of RViz — kernel, driver or link — and RViz is only an honest witness. A cloud that flickers only in RViz points at the display layer, covered under Cause 4.

Cause 1: a kernel the driver was never built for

ROS depth-camera stacks ride on kernel drivers: the USB video class driver below, the wrapper's nodelets above. Those stacks are built and tested against the kernels your distribution ships, with the distribution's own configuration and patches. A mainline tarball kernel — the kind numbered like 4.4.0-040400-generic, downloaded as raw packages rather than installed by the distribution — is built without them. The stream negotiates, the first frames arrive, then transfers fail in batches, and that is your flicker.

The bring-up that set this rule: Ubuntu 14.04 on the out-of-tree 4.4.0-040400-generic kernel flickered exactly this way, and the distribution's own 4.4.0-78 kernel fixed it. The repair sequence:

1. List the out-of-tree kernel packages: dpkg -l | grep 4.4.0-040400
2. Remove exactly those packages: sudo apt-get purge <packages>
3. Install a distribution kernel: sudo apt-get install linux-image-generic-lts-xenial on a 14.04 system, or sudo apt-get dist-upgrade on any supported release
4. Reboot: sudo reboot

Generalized: run any kernel from your distribution's own repository, at or above the minimum the camera driver's release notes state, and never a mainline tarball. On a production cell, validate one kernel version, then hold it and move only when you choose to.

Cause 2: the USB3 link cannot carry the stream

A depth camera streams more than depth: a typical RGB-D nodelet publishes depth, color and the registration data over the same cable. Budget the link before you add a hub, a second camera or an external drive to it.

USB 3.0 advertises a 5 Gbit/s line rate. 8b/10b coding spends a fifth of that, leaving 4 Gbit/s, and link and protocol overhead bring usable bulk throughput to about 3.2 Gbit/s. We budget 400 MB/s per controller and plan camera payloads to stay under 320 MB/s for margin.

Worked example, the classic RGB-D nodelet load of depth plus 1080p color at 30 fps:

• Depth, 16-bit: 848 × 480 × 2 B = 814,080 B ≈ 0.81 MB per frame, so 24.4 MB/s at 30 fps
• Color, RGB8: 1920 × 1080 × 3 B = 6,220,800 B ≈ 6.2 MB per frame, so 186.6 MB/s at 30 fps
• Together: 211 MB/s ≈ 1.69 Gbit/s payload — inside the budget, alone

The same load fails on USB 2.0, whose practical bulk throughput is about 40 MB/s: 6.2 MB color frames deliver about 6 fps instead of 30. When flicker starts the moment you enable the registered color stream, this arithmetic is your answer.

The ceiling bites larger sensors faster. A 4000 × 3000 sensor at 8-bit mono and 30 fps moves 12 MB per frame, 360 MB/s of payload — brushing the budget before any overhead, so a hub or a second camera on the same controller pushes it over. The 12 MP USB3 camera LS-CMR-0166 streams exactly this class of data, and the same arithmetic governs it and every area scan camera on a USB3 link.

Physical layer before arithmetic, though: a real SuperSpeed port on the host, a short shielded USB3 cable, no unpowered hub in the path. Then measure. Log the depth topic's arrival rate for 60 s and compare it with the configured frame rate — a camera set to 30 Hz that delivers about 15 Hz is losing every other frame in transport, and gaps between consecutive frame stamps show where. The USB hub bandwidth budget guide walks the hub arithmetic, and camera FPS drops suddenly: exposure, USB link, bandwidth runs the same test for any USB3 camera.

Cause 3: exposure makes depth boil, not flash

Coded-light depth cameras compute distance from how a projected pattern deforms on the scene, so exposure faults show up as depth fluctuation rather than missing frames. An exposure too long for the scene integrates ambient light and part motion; auto-exposure hunting between bright and dark regions changes the illumination frame by frame; mains-dimmed lighting injects a ripple that each exposure lands on differently. The cloud boils: edges shimmer, flat faces ripple — the second signature in the table above.

Fix: turn auto-exposure off, set a fixed exposure that holds scene brightness under steady lighting, and keep lamps off dimmers. Confirm that depth values on a static flat target hold still across 100 consecutive frames before you blame anything else. Depth camera frame rate drops: exposure settings cap the FPS covers the speed side, and depth auto exposure not applied at startup: fix the setting order covers the startup trap where your fixed value never takes effect.

Cause 4: the RViz display itself

With Decay Time above zero, RViz keeps every cloud on screen for that many seconds. Sound data renders dense and calm; dropped frames render as a pulsing mix of stale and fresh clouds — which is why raising Decay Time looks like a fix. It is cosmetic: the drops continue, and any algorithm subscribing to the topic still misses them. Debug with Decay Time at 0 so the display tells the truth.

If the arrival rate measures clean but the screen still stutters, the render loop is behind, not the data: a loaded GPU or an oversized display queue. Remember too that what a viewer draws is not always byte-for-byte what the driver delivered; why exported point clouds differ from the viewer render untangles exactly that gap.

Fix in this order

1. Reproduce outside RViz. Vendor's viewer, same machine. Flickers there: work through steps 2 to 5. Clean there: go straight to step 5.
2. Check the kernel. uname -r must print a distribution package, never a mainline build like the 040400 series. Purge the out-of-tree packages, install the distribution kernel, reboot, confirm with uname -r.
3. Clean the physical link. Direct SuperSpeed port, short shielded cable, no hub. Swap the cable before suspecting the camera.
4. Measure and budget. Arrival rate must equal the configured frame rate. If it does not, run the frame arithmetic above and move the camera to its own controller, or lower the stream until the payload sits under 320 MB/s.
5. Pin exposure and display. Fixed exposure, steady lighting, Decay Time 0.

A flicker that survives all five, on a second machine as well, is hardware: swap the cable first, then the camera.

Keep it fixed

A stable bring-up stays stable only if the cell keeps these habits:

• Hold the kernel. Validate one distribution kernel per cell, then pin updates until you choose to move; every kernel change re-tests the whole camera stack.
• Give each depth camera its own controller. One root port or USB3 controller per camera keeps the 400 MB/s budget individual instead of shared.
• Re-budget before adding streams. Raising resolution, adding color or adding a camera all change the payload; run the two-line arithmetic again first.
• Monitor arrival rate and stamp gaps in production. A publish rate that quietly halves is a cable failing, visible in the log weeks before a stoppage.
• Log the decimation you set. A cloud coarser than you configured is decimation doing its job, not a new fault; depth image resolution smaller than expected: check decimation first explains the mechanism.

Running several cameras from one host raises two further limits, the host USB buffer and the kernel's USB memory allocation: see two USB3 cameras at full resolution stream: raise the host USB buffer and USB3 camera low frame rate and latency: fix usbfs memory and exposure.

When it really is the camera

After a distribution kernel, a clean direct link, a measured rate that matches configuration and a fixed exposure, a flicker that still persists on two machines is the hardware. Cables fail far more often than sensors; two machines and two cables reproducing the flicker inside the vendor's viewer is the sensor's signature.

For a production cell rather than a bench, shortlist from the structured-light and ToF families in the 3D sensors range and ask one question of every candidate: does the vendor maintain the Linux driver and SDK against current distribution kernels? That answer predicts away most of this guide's failure modes. The two families below are where most production depth cells start.

To match a camera to your kernel line, stream and cell, start at the camera configurator or contact the applications team. The 3D camera topic page collects the rest of the depth-pipeline fixes.