Depth camera point cloud flat in XY: fix the optical frame

A simulated depth camera should be painting your workspace floor, but rviz shows a vertical sheet of points hanging in the air, and no amount of mount tweaking moves it. The cloud is not broken and the sensor model is not wrong: the data is labelled with the wrong frame convention. The fix is to label the cloud with the frame it actually lives in — the optical frame — and hand tf the one quarter-turn between that and your body frame. This page works through the symptom, the joint that fixes it, and the checks that prove it in minutes.

Why is my depth camera point cloud rotated 90 degrees in rviz?

The cloud is not bent and the sensor model is not wrong. The data carries optical-frame axes while tf plots them as body-frame axes, so depth along the viewing direction becomes height and the whole scene tips over by exactly 90 degrees.

The tell is the shape of the error. A floor that should spread across the XY plane renders as a curtain standing in the XZ plane, and the further a point is from the lens, the higher it floats. Because the error is a full quarter turn, nudging the sensor pose by 5 or 10 degrees never helps. If small pose corrections do not shrink the error, suspect the frame label before the geometry.

Take a level camera mounted 1.0 m above the floor. A floor point 2.0 m ahead of the lens and centred in the image sits 1.0 m below the optical axis. Its optical coordinates are x = 0 m, y = +1.0 m (down) and z = +2.0 m (depth). Published under a body-style frame with no rotation to the world, rviz reads those numbers as body axes: 0 m forward, 1.0 m left, 2.0 m up. Adding the camera's 1.0 m mounting height puts the point at world (0, 1.0, 3.0) m — 3.0 m in the air — instead of the true (2.0, 0, 0.0) m on the ground. Every point rises with its own depth, which is the curtain on your screen.

Which frame convention does a depth camera's point cloud actually use?

A depth cloud is built pixel by pixel from the depth image. The pinhole model that deprojects it works in the optical convention fixed by REP 103, the ROS coordinate-frame standard: z points forward along the optical axis, x points right in the image, and y points down.

The two conventions coexist on purpose. Body-style frames follow the platform rule — x forward, y left, z up — and describe where the camera is mounted on the robot. Optical frames describe what the sensor measures. They differ by one quarter turn, which is precisely the rotation tf has to be given. How the axes map onto stored point data is covered in Camera coordinate frame vs point cloud x y z: why the axes differ.

Why did renaming the frame to camera_frame_optical change nothing?

Because the suffix is a promise, not a switch. The label only helps if a frame with that name exists in tf and a fixed quarter-turn connects it to the body camera frame. That means publishing the frame in your URDF or xacro rather than renaming a string.

The sensor plugin does none of that for you. Its frameName parameter only stamps the label onto the message header. In a Gazebo-classic depth camera plugin, whatever string frameName carries goes onto the cloud unchanged. Rename the label without adding the frame and one of two things happens: rviz reports a frame lookup failure, or an identity edge somewhere in the tree keeps drawing the cloud exactly where it was. Every consumer resolves the label through tf — the viewer, your ground-removal node, the planner — so the frame and its rotation must exist in the tree, not just in the config file.

How do you publish the optical frame with the correct rotation?

Add one fixed joint to the URDF. It parents the body camera frame and children a camera_frame_optical link at the same origin, with rpy of -1.5708, 0, -1.5708 rad (-90 degrees roll, -90 degrees yaw). Then point frameName at camera_frame_optical so the labelled data finally meets its frame.

The joint is short enough to read at a glance (link names illustrative — match your own tree):

<joint name="camera_optical_joint" type="fixed">
  <parent link="camera_frame"/>
  <child  link="camera_frame_optical"/>
  <origin xyz="0 0 0" rpy="-1.5708 0 -1.5708"/>
</joint>

With URDF's fixed-axis convention — rotation equal to Rz(yaw) * Ry(pitch) * Rx(roll), pitch zero here — that origin works out to Rz(-90°) * Rx(-90°). It maps optical axes onto body axes as:

How do you verify the fix with view_frames and one measured point?

Run rosrun tf view_frames and confirm camera_frame_optical now exists in the tree as a child of the body camera frame. Then replay the worked point: optical (0, 1.0, 2.0) m must come out as body (2.0, 0, -1.0) m, which is the true ground position 2.0 m ahead of the camera.

1. Dump the tree with rosrun tf view_frames and open the saved drawing. Both links must be present, and the edge between them must come from your URDF's static publisher, not from a leftover node.
2. Drop a reference grid into rviz at 0.1 m spacing: the floor belongs on it, not hanging above it.
3. Check the arithmetic by hand: x_body = z_optical = 2.0 m, y_body = -x_optical = 0.0 m, z_body = -y_optical = -1.0 m. With the camera at (0, 0, 1.0) m the point lands at world (2.0, 0, 0.0) m, on the grid.
4. Aim at a wall you know is 2.0 m from the lens: it must now stretch 2.0 m ahead along +x, not 2.0 m upward.

The dump-and-measure routine takes minutes. It catches the failure mode where the frame exists but an old node still publishes a competing transform.

When is it better to rotate the cloud in code instead?

Only when you cannot touch the transform tree — a locked vendor driver or a quick bench test. Apply the fixed mapping x' = z, y' = -x, z' = -y in one place, your subscribing node, and leave a comment saying the input was optical-frame data.

A code rotation patches one pipe instead of the supply. rviz, the planner and every later node each need the same treatment, and the next engineer inherits a magic sign flip with no frame to inspect. Publish the optical frame once and let tf carry it. If you must manipulate arrays directly, Transform depth point clouds between optical frame and camera frame shows the operation step by step.

All of this — the frame dump, the viewer, the perception nodes — runs on the host that joins the robot network. A fanless vision controller with several GigE ports holds cameras, lighting and processing in one enclosure, keeping the transform tree and the data path on the same machine:

Does a real 3D sensor use the same optical convention?

Yes. The body-versus-optical split is a documented convention that real depth, stereo and profiler drivers also follow. The frame your nodes request, the sensor's own calibrated frames and your mechanical mount all have to agree before simulation results transfer to hardware.

When you swap the simulator for hardware, read the sensor's frame documentation before writing a line of code. The cloud message carries its own frame label, the deprojection maths is unchanged, and the mount orientation remains yours to define and measure. Mounting height and angle set your usable volume before any frame question, which the 3D camera working distance and positioning guide works through. If you stream colour alongside depth and deproject onto it, the intrinsics choice matters too; Aligned depth point cloud intrinsics: which camera to deproject with settles which camera's set applies.

Laser profile sensors suit inline checks where every part crosses a scan line, with each profile arriving in the sensor's calibrated frame. Browse the wider range of 3D sensors or filter by profiler type in the camera finder:

What else goes wrong if you leave the cloud rotated?

Downstream steps inherit the quarter turn and fail in ways that look like their own bugs. Ground removal fits a vertical plane instead of the floor, obstacle heights report range instead of clearance, and pick targets float metres above the table. Fix the frame before tuning any of them.

The wrong frame costs weeks of tuning downstream nodes that can never be right; the joint costs an afternoon. Make three habits part of the stack:

• Assert at startup: transform one synthetic point 1.0 m along the cloud's optical axis into the world frame and require it to land 1.0 m ahead, not 1.0 m up. The check runs in milliseconds and catches a deleted joint in seconds.
• Name frames by the convention: plain names for body frames, the _optical suffix for optical ones, so a label always tells the truth about its axes.
• Never patch twice: a URDF joint plus a leftover code flip compose into a 180 degree error. Remove one of the two.

An inverted or flipped cloud that survives these checks has a different cause — 3D ROI coordinate frame: diagnose an inverted point cloud walks through that case. Once the floor lies flat, removing ground points from a depth point cloud for obstacle detection becomes the straightforward job it should be.

If the cloud sits on the grid and the startup assertion passes, the simulated stack is ready to grow. Plan camera, lighting and host as one system in the configurator, find more frame and point-cloud fixes on the 3D vision topic page, or reach the applications team through the contact page.