ROS 2 point cloud black in Docker: fix GPU runtime passthrough

The topic publishes, the counter shows ten frames per second, and RViz2 still draws a black window instead of your point cloud. That combination — data flowing, display empty — sends most people chasing QoS settings, image transport builds and lighting. The usual culprit is none of those: the Docker container holding the 3D camera node has no working GPU access, so the depth stage computes nothing and every point goes out as NaN.

This guide shows how to confirm that in two commands, how to decode the bytes on the wire to prove the data itself is invalid, how to fix the container with the NVIDIA runtime and the right driver capabilities, and how to pin the setup so it stays fixed. The worked case is ROS 2 Foxy on an Ubuntu 20.04 host with a CUDA-capable NVIDIA GPU, but the fix is the same wherever a camera node runs in Docker.

Symptom: the rate is steady and the display is black

The signature is specific. ros2 topic hz on the cloud topic prints a solid rate, and the RGB topics from the same camera render normally:

average rate: 10.002   min: 0.092s   max: 0.109s   std dev: 0.00344s   window: 42

Ten messages per second, arriving within a 17 ms spread. Re-subscribing with a reliable, keep-last, depth-1 QoS profile changes nothing, and neither does swapping RViz2 for another viewer. That clears the publisher and the middleware: messages are delivered on schedule, so the question moves from transport to content. The pipe checks out too. A 1280 × 720 cloud with a 16-byte point step moves this much at 10 Hz:

147 MB/s needs a real gigabit link; a 100 Mbit path would drop frames and show a low or zero rate here instead. Scale the arithmetic by your own width, height and point step. A healthy rate with a black display means the bytes are the next thing to read.

Decode the bytes: every point is NaN

Run ros2 topic echo on the cloud topic and read the small fields before the data array. In this failure they print height: 720, width: 1280, four fields named x, y, z and rgb with datatype: 7 — float32 in the sensor_msgs PointField definition — plus point_step: 16, row_step: 20480 and is_dense: false. Two checks fall out. The row arithmetic must hold: 1280 pixels × 16 bytes = 20,480, which matches the printed row_step, so the publisher is internally consistent. Then look at the data itself:

data: 255 255 255 127  255 255 255 127  255 255 255 127  255 255 255 127 ...

That quartet repeats for every point. FF FF FF 7F read little-endian is 0x7FFFFFFF, and a float32 with an all-ones exponent and a non-zero mantissa is NaN. So x, y, z and rgb are NaN for every one of them: the node publishes a correctly framed cloud of nothing. That is the fingerprint of a failed compute stage, not a network fault.

The discriminating test: a handful of NaN points at object edges and dark surfaces is the normal depth-hole case, and a display that hides or dims them is a viewer-side problem — see missing points in a 3D camera point cloud and fixing muted point cloud colors. Every point NaN means the depth stage never ran. If your bytes decode to finite coordinates instead, stop here: your data is fine, and the black display is the QoS mismatch case.

Likely causes, in order of probability

Four causes produce this symptom, and they arrive in a stable order.

1. No container runtime attached. The service starts with privileged: true and the device nodes /dev/nvidia0, /dev/nvidiactl and /dev/nvidia-uvm mapped in, but without the runtime that injects the userspace driver libraries. Device access alone does not initialize CUDA: the SDK probes the GPU, finds nothing usable, and logs a line like 'No NVIDIA graphics card detected'. This is the most common cause by a wide margin.
2. Driver capabilities too narrow. The runtime is attached but NVIDIA_DRIVER_CAPABILITIES is unset or lists only utility. nvidia-smi then works — it needs the utility set — while CUDA (compute) and the OpenGL/EGL libraries (graphics) are missing, so the node starts and fills the cloud with NaNs.
3. Manually mounted driver libraries. A volume such as /usr/lib/nvidia-470 copied from the host shadows the runtime's injection with libraries that do not match the host's kernel module. The SDK fails with 'Err [804]: forward compatibility was attempted on non supported HW'.
4. A host-side problem. The driver is missing or older than the image's CUDA requires — nvidia-smi fails on the host itself. As a pairing rule, a CUDA 11.4 image needs a 470-series driver or newer, and CUDA 12.x images need a 525-series driver or newer.

Test each cause in order

Work down the list; each step either clears a cause or finds it. If the node does not even start, that is a different symptom — work through the GPU runtime libraries check and the Docker startup-failure guide first.

1. On the host: nvidia-smi. It should print the driver version and the CUDA version the driver supports. Failure here is cause 4; fix the host driver before touching Docker.
2. In the container: docker exec <container> nvidia-smi. An error means the runtime is not attached — cause 1. A printed GPU table means the utility libraries made it in; keep going.
3. In the container: docker exec <container> bash -c 'echo $NVIDIA_DRIVER_CAPABILITIES'. An empty answer or a short list is cause 2.
4. In the container: docker exec <container> ldconfig -p | grep libcuda. Nothing printed means the compute libraries are absent.
5. Launch the node and read its first log lines: the SDK's own verdict — 'No NVIDIA graphics card detected', or the forward-compatibility error — confirms which of causes 1 to 3 you have.

Then the decision, before any fix: nvidia-smi failing in the container points to the runtime; nvidia-smi working while compute fails points to the capabilities; the forward-compatibility line points to the manual mounts.

Fix: attach the runtime and open the capabilities

The host needs the NVIDIA Container Toolkit installed and Docker registered to use its nvidia runtime — the component the older guides call nvidia-docker or nvidia-docker2. No container-side change substitutes for it. Then apply one of these to the service that launches the camera node; the first is the classic form, the second is the form Compose documents today, and both need the toolkit on the host:

services:
  camera:
    runtime: nvidia
    environment:
      - NVIDIA_DRIVER_CAPABILITIES=all

services:
  camera:
    environment:
      - NVIDIA_DRIVER_CAPABILITIES=all
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

The reservation form is the one Compose documents for GPU access under deploy.resources.reservations.devices: you declare the nvidia driver, a device count and the gpu capability, and Compose hands the service a GPU at up time. Keep NVIDIA_DRIVER_CAPABILITIES=all with either form: all includes compute for CUDA, graphics for the OpenGL/EGL libraries that GL-linked SDK builds and renderers load, and utility for nvidia-smi.

Then remove the workarounds. Delete the devices entries for /dev/nvidia0, /dev/nvidiactl and /dev/nvidia-uvm, delete the /usr/lib/nvidia-470 volume, and drop privileged: true — the runtime creates the device nodes and mounts the matched libraries itself. Keep what serves the camera: /dev/bus/usb for a USB 3D camera (device passthrough there is its own subject, covered in the USB camera Docker guide), a sized /dev/shm, and the display setup if you run a window in the container.

Recreate rather than restart: docker compose up -d --force-recreate. An environment or runtime change takes effect only in a fresh container.

Verify the GPU before the camera node starts

Prove compute access before the camera enters the story. In the recreated container, nvidia-smi should print the GPU, the driver and a process list. Then run anything that makes a real CUDA call — the image's bundled sample, your SDK's GPU info tool, or a one-line interpreter check that enumerates CUDA devices. A device name plus total memory is the proof; nvidia-smi alone is not, because the utility set works even when compute is broken.

Relaunch the node and re-echo the cloud: the quartet pattern is gone, x, y and z read as plausible floats, and RViz2 renders the colored cloud at the same steady rate as before. Points missing at edges and dark surfaces from here on are the normal depth-hole behavior, not a relapse.

Prevention: pin driver and image, gate the launch

Three habits keep it fixed. First, pin the driver branch and the image's CUDA tag as a pair and move them in the same maintenance window: a CUDA 11.4 image with a 470-series driver, a CUDA 12.x image with a 525-series or newer one. If other software pins the host to an older branch, choose the image tag to match instead of upgrading the driver on hope.

Second, gate the launch on GPU visibility, so the container fails loudly instead of publishing NaNs for hours:

nvidia-smi > /dev/null 2>&1 || { echo 'no GPU visible; refusing to start the camera node'; exit 1; }

Third, keep the byte-level check in your acceptance test: echo a few points and assert finite coordinates, because a green publish rate says nothing about content.

If you are specifying the hardware

The same failure starts at purchase time. The Docker host needs a CUDA-capable NVIDIA GPU, the toolkit, and VRAM at least at the level the SDK's requirements list states — treat that number as a floor, because a desktop session on the same card eats into it. On a desktop or industrial mini PC the GPU question is the same: confirm the chassis takes the card and the power budget covers it. For dedicated shop-floor hardware, the vision controllers category is where this lives. The MV-VC5525-256G70-NN is a VC5000-series controller with a 12th-generation Intel Core i5, 32 GB of memory, 5+ LAN ports, PoE camera ports and PCIe expansion supported — the expansion path a discrete GPU needs. Tell us the card you plan to fit and we will match the controller, power and cooling to it.

The cloud you are fixing comes from a 3D sensor, and the NaN test travels with it: however the sensor splits its work, the depth stage runs somewhere, and this article is the host side of that split. The 3D sensors category and the camera finder filter the range by type and series; the 3D topic page collects the site's guides on the ROS 2 side of the same stack.

Checklist and the decision for each case

Run down this list after any change to the host driver or the image:

• Host: nvidia-smi prints driver and CUDA versions; the driver branch matches the image's CUDA tag.
• Host: the NVIDIA Container Toolkit is installed and the nvidia runtime registered with Docker.
• Compose: runtime: nvidia or a driver: nvidia device reservation is present.
• Compose: NVIDIA_DRIVER_CAPABILITIES=all set; no /usr/lib/nvidia-* mounts; no manual /dev/nvidia* device entries.
• Container: nvidia-smi prints the GPU; a CUDA call enumerates a device; echo $NVIDIA_DRIVER_CAPABILITIES answers all.
• Application: cloud bytes decode to finite x, y, z; the rate is unchanged; RViz2 renders color.
• Process: driver and image updated together; the entrypoint refuses to launch without a GPU.

The decision, one line per case: every point NaN at a steady rate — attach the runtime and set the capabilities to all, then recreate. nvidia-smi works but compute fails — capabilities only. The forward-compatibility line in the log — delete the manual driver mounts. No messages at all — it was never the GPU; follow the QoS fix instead. If the build still fights you after that, send the compose file and the node's first log lines through the contact page, and we will read them the way this article does.