Camera IMU rototranslation calibration: build the transform

A depth head, a stereo rig or a custom sensor board usually arrives with an IMU bolted next to the lens and no word about where it actually sits. Everything that fuses the two streams — visual-inertial odometry, robot guidance, a stabilization filter — still needs that pose, and it needs it right, because every fused pose inherits its error. This guide walks through producing the camera-IMU rototranslation yourself: the rig, the sweep, the frame conversion that lets a non-frame sensor join in, and the checks that separate a usable transform from a plausible-looking one.

The transform you are solving for

The rototranslation is one 6-DoF rigid pose between the two devices: a 3 × 3 rotation matrix R and a 3 × 1 translation t, stacked into a 4 × 4 homogeneous matrix with [0 0 0 1] as the bottom row. A camera-IMU solver fits it together with a time offset between the two clocks — seven numbers in total: three rotation angles, three translation components, one clock shift. The translation is the lever arm between the camera's optical center and the IMU's reference point, so it carries units, and this guide uses meters throughout. Most tools report the transform that carries a point from the camera frame into the IMU frame; call it T_ic here.

Direction is the first trap. Half the integration bugs blamed on a bad calibration turn out to be the inverse, T_ci sitting in a slot that expects T_ic; the magnitude of t survives an inversion, its signs do not. The direction and units of the camera-IMU transform get an article of their own, and the matrix arithmetic is the same homogeneous transformation matrix that carries camera-guided motion.

Whatever consumes the pair — for example a Kalman filter for IMU and camera fusion — propagates the IMU forward between camera frames and corrects with the image, joined by exactly this matrix. If your device family already ships factory extrinsics, compare factory defaults with your own calibration before redoing the work: a confirmed default beats a rushed sweep.

Build the rig: camera, IMU and one clock

Three things decide quality before any software runs: a rigid joint, an honest clock and geometry that cannot move mid-run. Bolt the camera and IMU onto one plate or bracket, as close together as the hardware allows — a short lever arm forgives translation error, a 100 mm boom magnifies it. Lock the lens focus and iris with tape; a lens that refocuses during the sweep changes the intrinsics underneath the solver.

Global shutter keeps every row exposed at the same instant and makes the sweep forgiving. A rolling-shutter camera still calibrates, but only if you supply its line delay to the solver. On timing, a hardware trigger or strobe is the clean answer; otherwise stamp both streams from one host clock. The solver fits a residual offset either way, but free-running devices that disagree by tens of milliseconds spend accuracy on the fit.

The solver also needs the IMU's own error model — gyro and accelerometer noise densities and bias terms, read from the IMU datasheet. Those are inputs, not something the recording estimates for you.

For the camera side, any compact global-shutter model with a standard mount works. The MV-CU020-90GC pairs a 1/1.7-inch global-shutter sensor with a C-mount and GigE output at up to 3 MP — more pixels than a calibration grid needs — and the camera finder filters on each of those attributes if you want alternatives.

If you would rather have the whole chain picked in order, the camera configurator walks sensor, mount and interface one step at a time.

Size and mount the target

The target is a checkerboard or a coded grid of the kind camera-IMU tools consume, with a known square size, glued dead flat. Flatness and scale are the two things the solver believes about the physical world; both have to be true.

Size the board for the sweep geometry. Field of view follows the usual lens math:

Example: a 1/1.7-inch sensor is about 7.5 mm wide on the standard 16 mm-diagonal convention. With a 6 mm lens and the board held at a closest distance of 600 mm, FOV = 7.5 × 600 / 6 = 750 mm. A 400 mm board then fills 400 / 750 = 53% of the frame width at the near end — inside the healthy zone. Size the board so it fills at least half the frame at the nearest pose and stays fully inside it at the farthest, around 1 m. Swap any two of lens, board and distance to fit your own rig.

Mount the sheet where it cannot flex or wander: glued to 2-3 mm aluminium against a wall, or clamped square to a stand. The LS-MNT-0040 has a 400 × 450 mm base and 700 mm of height, which holds a board steady through an afternoon of takes.

Distance matters as much as flatness. Past about a meter a given board shrinks below the half-frame rule and corner precision collapses; why checkerboard calibration fails at distance covers that failure in detail, and the 3D camera working distance guide covers placement in general.

The workflow, step by step

With rig, target and clock settled, a single recording produces intrinsics, extrinsics and the offset. The open-source Kalibr tool and its equivalents all follow this shape:

1. Assemble the rig. Camera and IMU on one plate; note which mechanical axis points where.
2. Solve intrinsics first. Capture 30-50 static views of the board and fit focal length, principal point and distortion. Fixing the pose before hand-eye work explains why this order pays. Lock focus and iris before you start.
3. Declare the grid. Measure one square's edge with calipers and type that measured number, in the units the tool asks for — usually meters — never the nominal print size.
4. Synchronize. Hardware trigger if you have it; otherwise stamp both streams from one host clock. The multi-camera synchronization workflow applies here unchanged.
5. Set exposure. Bright, steady light and 1-2 ms while waving; the arithmetic below shows why.
6. Sweep. 60-120 seconds, continuous and slow — under roughly 30°/s, the blur budget above — covering roll, pitch and yaw, with lateral translations of 10-20 cm mixed in. Include 3-4 board orientations and keep the board 0.5-1 m away, at least half the frame wide.
7. Convert non-frame streams. Event-type cameras and similar sensors pass through the frame conversion in the next section before this step.
8. Solve. Feed the frames, the IMU samples and the datasheet noise values; collect T_ic and the time offset.
9. Check it. Run the verification in the section after that before the transform touches production code.

Two numbers that decide capture quality

Motion blur is geometry:

Take a 6 mm lens on 3 µm pixels (illustrative values): f_px = 6 / 0.003 = 2000 px. Sweeping at ω = 0.52 rad/s (30°/s), a 1 ms exposure smears the image by 2000 × 0.52 × 0.001 = 1.0 px — acceptable. At 5 ms the smear is 5.2 px, grid corners streak, and corner detection fails without an error message. Add light, not exposure.

The same geometry sets the accumulation window for event streams. At 0.52 rad/s and 600 mm, a 2 ms window smears board edges by 0.52 × 0.002 × 600 = 0.62 mm — crisp. A 20 ms window gives 6.2 mm, enough for the grid corners to drift apart within one rendered frame.

When the camera does not output frames

Event cameras and a few other sensor types report per-pixel brightness changes, not images, and no standard camera-IMU solver consumes their native output. The path around it is old and reliable: turn the stream into ordinary frames, then calibrate exactly as a frame camera.

Dump the events from the head unit as one row per event — x, y, polarity (0 or 1), timestamp (µs; that column order is an example, match your own export) — then accumulate each fixed window of events into a grayscale image, polarity 1 brightening and polarity 0 darkening. Render at a steady rate with the window chosen by the smear arithmetic above (1-5 ms at sweep speeds), and store the frames with their timestamps in the layout your solver reads. The solver never learns the sensor was not a frame camera; intrinsics, extrinsics and the clock offset come out in the same units as always.

This record-convert-solve split is the same offline pattern as recording stereo cameras for offline SLAM work: capture in the field on modest hardware, do the fitting on a workstation.

Read the result, then distrust it once

The solver returns the rotation (quaternion or 3 × 3 matrix), the translation in meters, and the time offset. Convert, then run four checks.

• Matrix health. R must be orthonormal with det ≈ +1; a quaternion must have norm 1. A copy-paste slip shows here first.
• Magnitude. A camera and IMU bolted to one plate sit 30-100 mm apart. A result of 0.4 m is a unit slip — a millimeter value in a meters slot inflates the distance a thousandfold.
• Signs and convention. Project a known point through T_ic and through its inverse; only one lands where the image says it should. Depth-camera families that publish their own IMU frames get the same audit — verifying a depth camera's IMU optical frame walks one.
• Residuals. Per-view reprojection errors should sit at a fraction of a pixel and look uniform. One view towering over the rest marks a blurred or mis-detected span; crop it and re-solve.

Then record the sweep twice more. Agreement within a few millimeters of translation and a few tenths of a degree of rotation is a healthy calibration. For scale: hand-eye robot calibrations commonly land at 1-2 cm, and 1-2 cm there is considered normal; a rigid camera-IMU plate should beat that with room to spare. Downstream, a mis-fitted transform announces itself as jitter or slow drift in the fused pose — the signature you also get when troubleshooting stereo depth jitter and IMU drift. If the pose wanders while the rig stands still, suspect the extrinsics or the clock offset before the filter.

Mistakes that quietly ruin the estimate

Most failed calibrations fail mechanically, not mathematically.

• A board that is not flat. Foam board bows by millimeters across half a meter; glue the sheet to 2-3 mm aluminium or a composite panel.
• Nominal instead of measured square size. A 60 mm square printed at 59 mm carries a 1.7% scale error into every pose; measure it, and fix the grid pitch if the sheet is already printed.
• Jerky motion. Starts, stops and reversals ring the mount; the IMU records a vibration the camera never sees sharply. Slow and continuous wins.
• Rolling shutter without the line delay. The solver can model it only if you tell it.
• Unit slips. Millimeter translations pasted into a meters slot inflate the lever arm a thousandfold.
• Mechanical changes after calibration. New lens, dropped rig, re-seated IMU — the transform died with the old assembly. Recalibrate and store the result against the serial numbers it was measured on.

Every one of these produces a confidently wrong T_ic; none of them produces an error message.

Before you ship the transform: a short checklist

• Lens focus and iris locked against the mount
• Board glued flat, square size measured with calipers
• One clock: trigger, strobe or shared host time
• Exposure at 1-2 ms, light added rather than exposure
• Sweep of 60-120 s, under ~30°/s, all three axes plus translation
• Board at 0.5-1 m, at least half the frame wide
• IMU noise values from its datasheet, not defaults
• det(R) ≈ +1 and translation 30-100 mm for a bolted rig
• Two extra recordings agreeing within a few millimeters and a few tenths of a degree
• Convention written next to the file: T_ic, meters, date, serials

Everything mechanical here is standard machine-vision hardware. The area scan cameras category covers the camera side, camera stands the mounting side, and the 3D sensors page lists integrated depth heads if you would rather start from a device that already streams image data. For help matching a camera to your IMU's clocking, use the contact page, and the 3D topic page collects the rest of the calibration and integration guides.