image_raw vs image_rect_color: why both exist when distortion is zero

Two color topics arrive when you start a depth camera's RGB-D launch file: /camera/color/image_raw and /camera/color/image_rect_color. The camera_info that accompanies the raw stream reports every distortion coefficient as 0.0, which suggests the raw image is already rectified — so what is the second topic for? The confusion costs real debugging time: teams subscribe to the duplicate, double their bandwidth, or run a second rectification over pixels that were never distorted. This guide explains where each topic comes from, how to read the camera_info contract, how to deproject a pixel with the published k values, and which topic your node should subscribe to in each case. The logic applies to any depth or stereo camera whose driver publishes ROS image topics, from a ToF module to a structured-light head.

Why do image_raw and image_rect_color both exist?

The two topics exist because your driver publishes the device's own stream on image_raw and the RGB-D launch file runs a rectify nodelet that re-publishes a corrected copy on image_rect_color; when the device already rectified the stream, that second topic is a duplicate of the first.

The split is a ROS convention, not an accident of one driver. A driver publishes its unprocessed stream under a raw topic. A processing stack subscribes to that stream, applies the calibration, and re-publishes the corrected result under a rect topic. For an RGB-D camera, launch files such as rgbd_launch wire the whole chain: depth_image_proc nodelets handle the depth side, and an image_proc rectify nodelet handles the color side — that nodelet is what publishes image_rect_color.

The convention earns its keep on a camera whose raw pixels are distorted: image_rect_color is then the corrected image that matches the pinhole model in camera_info. On a camera that rectifies in hardware, the chain still runs, but there is nothing left to correct. Both topics exist because the launch file is written to serve both kinds of camera, and it cannot know which one you connected until it reads the calibration.

Why are the distortion parameters already zero on image_raw?

Because this class of depth camera applies the lens correction inside its own ISP, the driver receives already-rectified frames and publishes camera_info with every distortion coefficient at 0.0 — the topic name says raw, the calibration message says rectified.

sensor_msgs/CameraInfo defines d as the distortion parameter vector, and distortion_model names the model: plumb_bob for the five standard coefficients (k1, k2, t1, t2, k3). All-zero d is not a placeholder. It is the calibration message stating that the pixels on the wire already follow the pinhole model given by k.

The correction happens inside the camera — in the ISP, the image-processing pipeline that conditions frames before they are streamed to the host — not in the driver. The driver receives rectified frames, publishes them under image_raw, and writes honest calibration into camera_info: zeros in d, focal length and principal point in k. The working rule follows: trust camera_info over topic names. On ROS 1, rostopic echo /camera/color/camera_info/d prints the vector; the ROS 2 topic tool echoes the same message.

Are the two topics pixel-for-pixel the same?

Yes — with d all zeros the rectify nodelet's remap collapses to the identity map, so image_rect_color carries the same pixels as image_raw, possibly re-encoded on the way through, and neither image is warped or shifted relative to the other.

The mechanics: the rectify nodelet builds its remap tables from camera_info, and with d all zeros every output pixel samples its own input coordinate. cv_bridge can still re-encode the frame (rgb8 to bgr8 swaps channel order), and a timestamp can differ by the copy delay, but no pixel is moved. You can prove it on the bench in a minute: subscribe to both topics, compute the mean absolute difference per frame, and expect zero apart from channel-swapped values. Structured differences mean your driver does not rectify on-device, and image_rect_color is doing genuine work.

Stereo IR is the exception to keep in mind. There, raw and rectified left and right frames genuinely differ, and the real question is which intrinsics describe the message you hold — Rectified vs raw stereo IR images: which intrinsics apply works through that case. A stereo head such as the MV-DLS300P-04, a laser galvanometer stereo camera, raises exactly that paired-stream version of the question.

How do you deproject a pixel from image_raw?

Use the pinhole model with the k values from the same camera_info: multiply the pixel's offset from the principal point by the depth and divide by the focal length in pixels — a formula that is exact here because d is zero.

Worked example, with illustrative inputs declared here so the arithmetic is checkable — they are not a printed camera spec: W = 640 px, fx = fy = 570.3 px, cx = 319.8 px, cy = 239.6 px. You pick the pixel (u, v) = (450, 300) px on a part edge, and the depth stream aligned to the color frame reads Z = 1.200 m at that pixel.

x = (450 - 319.8) * 1.200 / 570.3 = 130.2 * 1.200 / 570.3 = 156.24 / 570.3 = 0.274 m

y = (300 - 239.6) * 1.200 / 570.3 = 60.4 * 1.200 / 570.3 = 72.48 / 570.3 = 0.127 m

The point sits at (0.274, 0.127, 1.200) m in the color camera's optical frame. Project it back to close the loop: u = 570.3 * 0.274 / 1.200 + 319.8 = 130.2 + 319.8 = 450.0 px, the same pixel you started from. The same k values also fix the field of view: HFOV = 2 * atan(640 / (2 * 570.3)) = 2 * atan(0.5611) = 58.6°.

Two conditions keep those numbers honest. Z must come from a depth stream aligned to the color frame — otherwise deproject with the depth camera's own intrinsics, as Aligned depth streams: data layout and which intrinsics apply explains — and the two frames must be time-matched, which the device-timestamp method in Sync RGB and depth frames by device timestamp, not queue order handles.

Which topic should your node subscribe to?

Take image_raw when camera_info reports zero distortion: it is the same rectified pixels one copy earlier in the graph, and everything image_rect_color could add is already on the raw topic.

Decide from camera_info, not from the topic names:

• d all zeros: the device rectifies; take image_raw and drop the rectify stage entirely.
• Any non-zero coefficient: the raw stream is distorted; subscribe to image_rect_color, or run your own rectify nodelet fed by image_raw and camera_info.
• Nodes already written against the rectified names: keep the graph as it is and let the duplicate exist — correctness first, bandwidth second.

If you are specifying the camera as well as the topic, the camera finder narrows the 3D sensors range by depth type, and these are representative modules:

Does subscribing to both topics cost anything?

Each extra subscription is one more serialized copy per frame: on a 640 × 480 color stream at 30 fps you pay about 0.92 MB per frame, roughly 28 MB/s, for pixels you already have.

The cost scales with resolution. One bgr8 frame at 640 × 480 is 640 × 480 × 3 = 921,600 bytes, which is where the 0.92 MB figure comes from; at 1920 × 1080 the same frame is 6.2 MB and the duplicate rises to 186 MB/s at 30 fps. Transport decides who pays: inside one nodelet manager, messages pass as shared pointers and the second topic costs almost nothing, while between processes every subscriber serializes its own copy. On an embedded or ARM controller, memory bandwidth is the first budget to run out, and a topic nobody consumes is still advertised — one launch-file edit away from being subscribed twice.

How do you check whether a stream still needs rectifying?

Echo the camera_info topic and read the d field: five zeros under the plumb_bob model mean the stream is already rectified, while any non-zero coefficient means the raw pixels are distorted and must be remapped before k applies to them.

The check takes a minute:

1. Echo d on /camera/color/camera_info. Five zeros: the stream is rectified. Anything else: plan for remapping.
2. Read distortion_model. plumb_bob means the five polynomial coefficients (k1, k2, t1, t2, k3); equidistant means a fisheye model, which arrives with non-zero coefficients almost every time.
3. Compare k and p on a rectified monocular stream: p normally repeats the fx, 0, cx opening of k.
4. Confirm on the bench: image a door frame or a checkerboard. Straight edges stay straight on a rectified stream; bowed edges together with non-zero d mean rectify before use.

If d is non-zero, rectify once and keep nothing raw downstream — correcting twice is its own failure mode, covered in Inverse distortion coefficients: why rectified depth looks warped. When published values look implausible, Depth camera intrinsics and distortion parameters: read, don't guess shows how to read them from the device instead of assuming them.

When is image_rect_color the right choice?

Keep image_rect_color in two cases: a camera whose driver publishes genuinely distorted raw, so the nodelet's remap does real work, and a codebase whose nodes already expect the rectified contract, so you do not rewrite every remap in the launch file.

A conventional camera streamed into ROS behaves the first way: there image_raw is the warped sensor image and image_rect_color matches k. Calibration-driven code for plane fitting, metrology or pose estimation wants the second: it should be able to assume the pinhole model, and image_rect_color plus camera_info guarantees that assumption no matter which camera is attached.

That is why the usual advice — always take the rectified topic — is wrong exactly where this question started: with zero distortion already published, the rectified topic adds a copy and no correction. The opposite advice, raw always means distorted, is equally wrong, because the ISP rectified the stream before the driver ever saw it.

The one-line decision: echo camera_info, read d, and let the calibration choose the topic. For the conventional cameras where the rectify step does real work, the area scan cameras range is the place to compare, and the camera step of the configurator pairs the camera with the rest of the station. The 3D vision topic page collects the guides that pick up after this one — registration, intrinsics, frame sync — and if a launch file still looks ambiguous, contact us with your camera_info attached; the fix is usually one remap.