A visual inertial SLAM paper hands you a residual like r = u − π(T_CB T_i⁻¹ ⊕ x), and every symbol in it silently carries a coordinate frame. Read one of them backwards — in practice that usually means the body-to-camera extrinsic — and the rig fails in a way that looks like a solver bug: residuals sit at several pixels and never settle. The sections below walk the chain term by term, push one landmark from the world frame all the way to a pixel, and give you the magnitudes that separate a healthy residual from a geometry fault.
The residual in one sentence
Every reprojection residual compares two pixels of the same landmark. u_ij is where the feature tracker measured landmark j in frame i. The product π(T_CB T_i⁻¹ ⊕ x_j) is where the current state estimate says that landmark should land. Subtract one from the other and r_ij is the disagreement, in pixels. Because the projection is the step that converts metres into px, the residual is always in px, no matter which frame the landmark lives in. The optimizer then minimizes the covariance-weighted sum of squared residuals over every frame-landmark pair in its window, usually through a robust kernel so one bad landmark cannot drag the state.
The four frames and who owns which
- World: fixed when the system initializes; the landmark map lives here, in metres. On a ROS-style stack this is the map frame, and a missing or mismatched one is its own failure mode — resolving the missing map frame in RViz covers that side.
- Body (IMU): rigid with the inertial unit. This is the frame the state vector estimates, because IMU samples arrive between camera frames and are integrated here.
- Camera: the optical frame — z along the optical axis, x right, y down. Depth cameras and stereo rigs output points here, and where z = 0 sits matters; choosing where stereo measures z from is its own decision.
- Image: pixels. u runs right, v runs down, origin at the top-left corner.
The equation term by term
| Symbol | What it is | Frames | Units |
|---|---|---|---|
| u_ij | measured pixel of landmark j in frame i | image | px |
| x_j | 3D landmark | world | m |
| T_i | body pose at time i; maps body → world | body ↔ world | m, rad |
| T_CB | calibrated extrinsic; maps body → camera | body ↔ camera | m, rad |
| ⊕ | action of a transform on a point | — | — |
| π | camera projection model | camera → image | px |
| r_ij | residual | image | px |
Read the product T_CB T_i⁻¹ right to left as it acts on a point: first T_i⁻¹ undoes the body pose (world → body), then T_CB carries the result into the camera. π does the projection; for a pinhole model that is the f·X/Z + c form below, plus lens distortion if your model keeps it. The projection step is walked through in projecting a 3D point to pixel coordinates with a calibrated camera.
What the ⊕ operator actually does
⊕ is the action of an SE(3) transform on a 3D point, not a way to merge two poses: (R, t) ⊕ x = R·x + t. Rotate the point, then add the translation — the same rotate-then-translate order you need when a point cloud goes through an extrinsic. Composing two transforms is matrix multiplication; ⊕ just says the point rides through the product.
The definition hides a one-second sanity check: T ⊕ 0 = t. The translation of any transform is always the source origin expressed in target coordinates. Push a zero vector through each of your extrinsics and compare the result with a tape measure; an inverted transform shows up immediately.
Which pose the state vector holds
T_i is the body pose in the world frame; writing it as T_W,B makes the direction impossible to misread. Visual inertial systems estimate the IMU body frame rather than the camera because the inertial measurements that bridge the gaps between images are physically expressed in the body frame. The camera pose follows from the body pose and the extrinsic whenever you need it — the FAQ below gives the composition. An initial pose guess belongs in that same body frame; which reference frame a pose initial guess means is a common stumbling block.
Worked example: one landmark through a yawed body, projected to a pixel
All inputs below are illustrative, chosen to keep the arithmetic visible; the frames and formulas are the ones above.
Inputs. Landmark x_W = [0, 10, 0] m. Body pose at frame i: t_i = [0, 2, 1.5] m with a +90° yaw, so the body's forward axis points along world +y and world +z stays up. Mount: camera 0.05 m forward and 0.10 m above the IMU, level with it; body axes x forward, y left, z up; camera optical z forward. Intrinsics: f = 700 px, principal point (640, 360) px. Tracker measured u_ij = (640.0, 503.9) px.
Step 1 — world to body. x_W − t_i = [0, 8, −1.5] m. With R_i = [[0,−1,0],[1,0,0],[0,0,1]], p_B = R_i^T (x_W − t_i) = [8, 0, −1.5] m: 8 m ahead of the body, 1.5 m below it.
Step 2 — body to camera. With the standard body-to-camera rotation R_CB = [[0,−1,0],[0,0,−1],[1,0,0]], R_CB·p_B = [0, 1.5, 8] m. The mount puts the camera origin 5 cm forward and 10 cm above the IMU, which makes t_CB = [0, 0.10, −0.05] m — the body origin in camera coordinates, per the zero-vector check. Then p_C = [0, 1.5, 8] + [0, 0.10, −0.05] = [0, 1.60, 7.95] m: 7.95 m along the optical axis, 1.60 m below it, centred sideways.
Step 3 — project. u = 640 + 700 × 0 / 7.95 = 640.0 px. v = 360 + 700 × 1.60 / 7.95 = 360 + 140.9 = 500.9 px.
Step 4 — residual. r = (640.0 − 640.0, 503.9 − 500.9) = (0.0, +3.0) px.
The decision. Divide by the focal length: 3.0 / 700 = 4.29 mrad ≈ 0.25°, worth 7.95 × 0.00429 ≈ 34 mm of landmark misplacement at that depth. A converged rig leaves a few tenths of a pixel — the detector's repeatability — so a persistent 3 px is geometry. Time sync first: f × speed × Δt / Z = 700 × 2 m/s × 0.001 s / 7.95 m ≈ 0.18 px, far too small to explain it at walking speed; at 10 m/s the same 1 ms offset is worth 0.88 px and sync becomes plausible. Below vehicle speeds, suspect the extrinsic — its direction first, then its lever arm.
Judging the residual you get
Judge residuals in two regimes. Before convergence, tens of pixels are normal. After convergence, expect a few tenths of a pixel per feature — that is tracker repeatability, not solver noise — and treat anything sustained above about 1 px as a geometry or sync fault rather than a tuning problem.
Three conversions make the number actionable. Angular: r_px / f_px is radians; 1 px at f = 700 px is 1.43 mrad ≈ 0.082°. Metric, at the landmark: Z × r_px / f_px; 1 px at 8 m depth is about 11 mm. Extrinsic error scales the same way: a hand-eye calibration off by 1–2 cm — the translation error typical of hand-eye calibration — is worth roughly 0.9–1.8 px at 8 m with f = 700 px, which is why extrinsic quality lands directly in this number. And a low reprojection error alone does not certify an extrinsic; what reprojection error misses in extrinsic calibration accuracy covers that gap.
From the equation to a mounted rig
The chain is one rigid-body bookkeeping exercise, and it is the same one you maintain on a physical rig. A depth camera bolted to a gripper or a mobile base outputs points in its own camera frame; the extrinsic to the mounting flange and then the robot or vehicle pose play the roles of T_CB and T_i. The pitfalls transfer too: chaining pixel-to-world transforms on an eye-in-hand rig is the projection run in reverse, and tool-frame offsets for a camera on the gripper is T_CB in mechanical clothing. The cameras below are that kind of rig: they deliver camera-frame points, and the world and body transforms around them are yours to calibrate and keep consistent.
- Type
- Stereo / structured light
- Type
- Stereo / structured light
Sanity checks before you blame the solver
- Push a zero vector through every transform; the result must equal the source origin measured in the target frame.
- Rename T_i to T_W,B in your notes and code comments; the direction becomes self-evident.
- Keep units straight: translations in m, focal length and pixels in px. One mixed unit poisons the chain silently.
- Convert residuals to millimetres at your working depth (Z × r / f) before deciding a number is large.
- Log pre-convergence and converged residuals separately; tuning weights to hide a geometry fault only hides it.
More on frames, projections and calibration is on the 3D topic page. If you are specifying the camera side of such a chain, start in the 3D sensors category, filter by the specs that matter in the camera finder, or contact the applications team with your mounting geometry.
Questions
Is T_CB the camera pose in the body frame, or the body pose in the camera frame?
In the paper's convention T_CB maps a point from body coordinates into camera coordinates — body to camera. Its translation t_CB is therefore the body origin expressed in camera coordinates. The camera's position in body coordinates runs the other way: minus R_CB^T t_CB. For the worked example's mount, 0.05 m forward and 0.10 m above the IMU, that gives [0.05, 0, 0.10] m — a value you can confirm with a ruler.
Why does visual inertial odometry estimate the IMU pose rather than the camera pose?
The IMU delivers acceleration and gyroscope samples at a few hundred hertz, including between camera frames, and those measurements are expressed in the body frame. Making the body the estimated frame lets inertial propagation fill every gap between images, while vision corrects it at frame times. Estimating the camera instead would leave nothing to integrate between frames.
The landmark is in metres and the pixel is in px — what units does the residual carry?
Pixels. The projection π is the step that converts metres to px: it divides camera-frame coordinates by depth, a dimensionless ratio, then multiplies by f, which is specified in px. That is also why the observation covariance Σ_ij carries px², and why the information-weighted residual is dimensionless. A residual reported in metres means a projection step was skipped or inverted.
How do I get the camera pose in the world frame from the estimated state?
Compose the body pose with the inverse extrinsic: T_W,C = T_i · T_CB⁻¹. T_CB⁻¹ maps camera-frame points into body coordinates and T_i maps body coordinates into the world, so the product maps camera coordinates straight to world — what robot guidance usually wants. If a reported camera pose disagrees with this product by roughly the lever-arm length, it was defined in the wrong frame.
Why is the residual large for the first seconds of a session?
Landmarks, scale and velocity are still unconstrained in the first frames, so predicted pixels can sit tens of pixels from measured ones; the optimizer pulls them in within a few iterations. Convergence that never settles below about one pixel is different: it usually means the scale is unobservable — insufficient acceleration excitation — or the extrinsic is wrong. Log residuals against time and separate the two regimes before tuning weights.