Which camera topics to record for offline point cloud generation

Projecting a registered XYZRGB point cloud while you record triples the data rate, ties the cloud to one machine, and buys you nothing you cannot compute later. The efficient split is two passes: record only the raw streams on the robot, then regenerate the clouds on whatever hardware you like, with the camera unplugged. The whole scheme stands or falls on recording the right topics, and the one engineers miss is the smallest of the three. Here is the exact set for an openni_camera depth camera, how to discover what any nodelet subscribes to, both passes with the arithmetic, and the checks that catch an unusable bag before you strike the set-up.

The three topics to record

Record these three topics, and nothing else from the depth pipeline:

/camera/depth_registered/image_rect_raw
/camera/rgb/image_rect_color
/camera/rgb/camera_info

What each one carries:

• /camera/depth_registered/image_rect_raw is the depth image, already rectified and resampled onto the color camera's frame. Every pixel is one distance value in millimetres, so the two cameras' views never have to be merged later.
• /camera/rgb/image_rect_color is the color image, rectified into that same geometry. It is an ordinary image stream, the kind any area scan camera delivers; depth is what makes the pair three-dimensional.
• /camera/rgb/camera_info is the color camera's calibration: the focal lengths, the principal point and the image size, in a message a few hundred bytes long.

One distance image in the color camera's geometry, one color image in the same geometry, one calibration describing that geometry. Anything else the depth pipeline publishes is either redundant or replaceable. Do not record the live point cloud: an XYZRGB cloud carries at least 16 bytes per point (three 4-byte coordinates plus color), so one 640 × 480 cloud is 307,200 × 16 B ≈ 4.9 MB — eight times the 0.61 MB depth frame it was built from. And do not reach for a video container instead of a bag; the reasoning is in record depth and color to a bag file, not an AVI.

Why camera_info is the topic you cannot skip

A depth image stores one number per pixel: how far. It carries no direction. The directions come from the calibration — fx and fy, the focal lengths in pixels, and cx and cy, the principal point — and without them a depth recording is a distance picture, not a cloud. You cannot recover a camera's factory calibration from its pixels after the fact, which is why a bag missing camera_info is a bag of gray values.

There is a second, subtler point: the calibration you need is the color camera's, not the depth camera's. Because /camera/depth_registered/image_rect_raw has already been resampled onto the color frame, the color intrinsics are the ones that deproject it. Which camera's intrinsics govern a registered stream is worth understanding once; aligned depth point cloud intrinsics: which camera to deproject with works through it.

camera_info is tiny and arrives rarely compared with the image streams, so a recorder that starts late can miss it entirely. Start the recorder after the driver is up, and confirm at least one message before you trust the session.

How to find what a nodelet subscribes to

The general question has a general answer. Your cloud generator is a nodelet — a module inside a manager process, not a process of its own. rosnode list shows it, but rosnode info on the nodelet lists only bond topics, the internal keep-alive plumbing, because a nodelet owns no transport of its own.

Run rosnode info on the nodelet manager instead. The manager hosts every nodelet and owns every socket, so its subscription and publication lists cover all of them at once. Connections wired directly between two nodelets inside the same manager will not appear anywhere — they are passed in process, not as topics — which is why no graph tool shows that internal wiring.

If you already have a bag, the fastest check is empirical: replay it and run the generator standalone, as in pass two below.

Pass one: record the three topics

1. Bring the depth camera driver up and let it publish.
2. Confirm the three topics exist, using rosnode info on the nodelet manager.
3. Start the recorder on exactly those topics, and watch the data rate for ten seconds against the budget below.

rosbag record /camera/depth_registered/image_rect_raw /camera/rgb/image_rect_color /camera/rgb/camera_info

If the clouds must land in a robot or workcell frame rather than the camera's optical frame, add the transform stream that carries the camera's pose to the same bag. It is small next to the images and impossible to reconstruct later.

Budget the disk before the shift, not during it. Example numbers for a 640 × 480 depth camera at 30 frames per second, depth at 2 bytes per pixel (a 32-bit float depth stream doubles the depth line):

• Depth: 640 × 480 = 307,200 pixels × 2 B = 614,400 B ≈ 0.61 MB per frame; × 30 = 18.4 MB/s = 66 GB per hour.
• Color: 640 × 480 × 3 B = 921,600 B ≈ 0.92 MB per frame; × 30 = 27.6 MB/s = 100 GB per hour.
• Total ≈ 46 MB/s ≈ 166 GB per hour, plus kilobytes of calibration.

Pass two: rebuild the cloud offline

Two routes, both with the camera unplugged.

The first is the driver's own generator, run standalone against the replayed bag:

1. Play the bag.
2. Run the cloud generator as a standalone nodelet:

ROS_NAMESPACE=camera rosrun nodelet nodelet standalone openni_camera/point_cloud_xyzrgb

The namespace prefix restores the /camera part of the recorded names, and the nodelet resolves the same three topics it consumed live. It publishes the identical XYZRGB cloud, on any machine that can read the bag.

The second route is to deproject the frames yourself, which is four arithmetic operations per pixel:

u and v are the pixel's column and row, z_mm is its depth value in millimetres, and fx, fy, cx, cy come from the recorded camera_info. Worked example with round numbers for a VGA-class depth camera — fx = fy = 525.0 pixels, cx = 319.5, cy = 239.5 (yours are in your bag; read them, do not reuse these) — for pixel u = 400, v = 280, z_mm = 1500:

• X = (400 − 319.5) × 1.500 m / 525.0 = 120.75 / 525.0 = 0.230 m
• Y = (280 − 239.5) × 1.500 m / 525.0 = 60.75 / 525.0 = 0.116 m
• Z = 1500 / 1000 = 1.500 m

That pixel is the point (0.230, 0.116, 1.500) m. The same constants set the cloud's sampling: at Z = 1.5 m, neighbouring pixels sit 1.500 / 525 = 2.9 mm apart, so 2.9 mm is the lateral resolution of the cloud at that distance. And fx fixes coverage: 2 × atan(320 / 525) ≈ 62.7° of horizontal field of view across a 640-pixel row.

In code, one pixel looks like this:

# Example values; read yours from the recorded camera_info
fx = 525.0   # float, focal length in pixels
fy = 525.0   # float, focal length in pixels
cx = 319.5   # float, principal point x, in pixels
cy = 239.5   # float, principal point y, in pixels

u = 400      # int, pixel column in the depth frame
v = 280      # int, pixel row in the depth frame
z_mm = 1500  # int, depth value at row v, column u, in millimetres

x_m = (u - cx) * (z_mm / 1000.0) / fx
y_m = (v - cy) * (z_mm / 1000.0) / fy
z_m = z_mm / 1000.0

assert abs(x_m - 0.230) < 0.001
assert abs(y_m - 0.116) < 0.001
assert z_m == 1.5

Loop it over every pixel and you hold a structured cloud: one point per row and column, neighbours still neighbours. 2.5D height map to 3D point cloud: pixels to XYZ uses the same structure for height maps, and export a structured point cloud as xyz and PNG takes it to files.

Verify the recording before you rely on it

• One camera_info message for the color camera exists in the bag, and the first and last message timestamps span the whole session. A recorder that started late is the usual cause of a missing calibration.
• The depth stream is the registered one — depth_registered in the name. An unregistered depth recording cannot be color-merged later, because the depth camera's calibration was never recorded.
• Depth bytes survive replay unchanged. Any lossy step in the chain shows up as terraced surfaces on flat walls.
• A ten-second sanity capture of a flat wall you measured at 1.5 m deprojects to Z ≈ 1.5 m across the wall. One wall, one tape measure, and the whole chain is proven.

When this topic set is the wrong set

• The cloud must be expressed in a robot frame. Add the transform stream in pass one; deprojection gives you points in the camera's optical frame, and everything else is a chain of transforms you either recorded or do not have.
• You want raw depth and registration under your own control. Then record both cameras' raw streams and both cameras' camera_info, and own the registration math; depth to color registration: raw versus rectified frames shows what changes. This is more data and more work, justified only when the driver's registration is not good enough.
• Your depth camera records through a vendor SDK into its own container. The three-stream rule still describes what must survive into any container: depth, color, and per-camera calibration.

Choosing the depth hardware for two-pass capture

The two-pass workflow asks three things of the camera: a depth stream, a color stream, and published per-camera calibration. Any depth camera that provides all three records and rebuilds exactly as described above. Two examples of the depth-hardware classes behind these streams:

A time-of-flight camera measures each pixel's distance directly; a projected structured-light camera triangulates a projected pattern. Either way, the recording rules do not change: three streams, lossless depth, calibration in the bag. The full ranges are in 3D sensors and the camera finder.

Next steps

• Size the session with the disk arithmetic above, then push a ten-second test bag through both passes before a full recording.
• Building a new rig? Start with the camera in the configurator and work outward from there.
• More capture and calibration walkthroughs are on the 3D vision topic page.
• For help with the data rate on a specific rig, use the contact page.