Depth and color field of view mismatch: why your pixels shift

Your detector draws a clean box on the color image. You carry the same pixel coordinates into the depth frame or the point cloud, and the box lands on the wrong part, or on nothing. The cause is rarely the detector. On most depth cameras the depth and color sensors cover different fields of view, so every point in the scene sits on a different pixel column in each stream. This guide gives you the two intrinsics numbers that set the shift, a worked example with the arithmetic shown, and the rule for choosing between aligning depth to color and aligning color to depth. A five-minute bench check near the end measures the mismatch on your own camera.

Two sensors, one scene, two fields of view

A depth camera is at least two cameras in one housing. A stereo pair, a ToF array or a structured-light projector-and-sensor set produces depth; a separate RGB sensor with its own lens produces color. Each sensor has its own focal length, pixel grid and calibration, and nothing forces the two fields of view to match.

Short-range stereo modules commonly cover about 85° to 90° horizontally with the depth pair and about 65° to 70° with color. The wide depth field lets the camera work close; the tighter color lens buys resolution. The mismatch runs the other way when the stereo pair is narrower than the RGB sensor, which is the case in depth image field of view: why stereo coverage is smaller. Pixel counts tell you nothing here: two 1280-column streams can cover 69° and 87°, with the same scene point on different columns. Most 3D sensors pair depth and color this way, so the mapping below applies across the class.

The pinhole math that sets the shift

A lens maps an angle to a pixel column. For a point at angle θ from the optical axis, with fx the focal length in pixels and cx the principal point:

The last two lines are the mapping you need: turn the color pixel into a normalized coordinate x_n with the color intrinsics, then back into a depth pixel with the depth intrinsics. The scale factor between the streams is fx_depth / fx_color, never width_depth / width_color. The offset is proportional to tan(θ), so it is zero on the optical axis and grows toward the frame edges.

These formulas ignore the physical offset between the two sensors and the lens distortion, both of which the SDK's alignment routine includes. Use the formulas to size the error; use the SDK to do the mapping.

Worked example: one target, two pixel columns

Take illustrative streams: depth at 1280 × 720 covering 87° horizontally, color at 1920 × 1080 covering 69°. A target sits 20° right of the optical axis.

• fx_depth = 640 / tan(43.5°) = 640 / 0.949 = 674 px
• fx_color = 960 / tan(34.5°) = 960 / 0.687 = 1397 px

The same point lands at:

• u_depth = 640 + 674 × tan(20°) = 640 + 674 × 0.364 = 885
• u_color = 960 + 1397 × 0.364 = 1468

One target, 583 columns apart. Read depth at the color column and you sample the scene at atan((1468 − 640) / 674) = 50.8°, about 31° away from the target. At a 1.5 m stand-off that is 1.5 m × (tan 50.8° − tan 20°) = 1.5 × 0.864 = 1.3 m sideways of the part you wanted.

Box widths shift the same way. A 60 mm part at 1.0 m spans 1397 × 0.060 = 84 px in color and 674 × 0.060 = 40 px in depth. Scaling columns by the resolution ratio 1280 / 1920 puts the box centre at column 979, still 94 px right of the true 885, and stretches the box to 56 px instead of 40. The correct scale is the tangent ratio 674 / 1397 = 0.48.

Stream color at 1280 columns instead and fx_color becomes 1397 × (1280 / 1920) = 931. The fx ratio is a property of the lenses, not of the resolution you pick, which is why no resolution choice removes the mismatch.

What each alignment direction costs

Alignment reprojects one stream onto the other's pixel grid using the factory intrinsics. The direction decides what you keep.

In the worked example, aligning depth to color keeps columns where tan θ ≤ 0.687 in magnitude, which is 674 × 0.687 = 463 px either side of centre: 927 of 1280 columns. The outer 9° per side drops out, about a quarter of the depth frame, and the surviving depth values are resampled onto the color grid, so depth edges pick up slight interpolation.

Aligning color to depth keeps the full 87°. The outer ring has no color data and reads as holes, and a library that resizes the color frame to the depth canvas without intrinsics leaves the 94 px residual from the example.

The decision follows from where your boxes live. If a color-image detector produces them, align depth to color and they carry straight across; then deproject the box with the aligned frame's intrinsics, as bounding box depth: mapping image pixels to 3D points and aligned depth point cloud intrinsics: which camera to deproject with detail. If the cell needs the whole depth scene, keep depth native and lay color over the overlap; how to register RGB depth images for a colored point cloud walks the textured cloud. Alignment runs per frame, so budget the extra step in the cycle time.

How to measure the mismatch on your camera

Pull the intrinsics for each stream at the resolution you will run: fx, fy, cx, cy. Compute HFOV = 2 × atan(width / (2 × fx)) for each, as the FAQ above works through; the worked example's depth stream gives 87° from fx = 674 px over 1280 columns. If the two fields of view differ by more than a degree or two, treat the streams as different frames and align before any pixel mapping.

A bench check needs no SDK calls. Put a target with a sharp corner near the frame edge, capture both native streams, and compare the corner's column in each. The edge offset is the worst case; it shrinks toward the centre in proportion to tan(θ).

Two gotchas waste hours. A depth image smaller than expected is usually decimation in the stream pipeline, not a narrower lens; depth image resolution smaller than expected: check decimation first sorts that out. Alignment that breaks after a resolution change is a stream-size mismatch; depth to color alignment breaks up: match stream resolution first covers it.

When the usual answer is wrong

Aligning depth to color is the default because detectors run on color, but four cases break it.

• An obstacle or a part profile near the frame edge lives in the 9° ring that depth-to-color alignment discards. Keep depth native for safety zones and volume scans.
• Crop order matters: align the full frame first, crop the region of interest after. Cropping first invalidates every intrinsic.
• Some cameras fit matched lenses on a common rigid carrier, so both streams share one field of view; there plain pixel scaling is valid, and the intrinsics confirm it in one calculation.
• A residual offset of a few pixels after correct alignment is factory calibration error; tens of pixels point to the wrong stream pair or a calibration fault. Stereo camera RGB calibration: fix depth-to-color alignment errors covers the calibration path, and depth to RGB registration offset in pixels: fix the rectified stream the rectification side.

Checks before you commit to a camera

Settle these on the bench before you design around a depth-plus-color camera:

1. Pull both streams' intrinsics at the working resolution and compute the two fields of view.
2. Evaluate u = cx + fx × tan θ at your worst-case target angle in both streams; the difference is the shift you must design out.
3. Confirm your driver or middleware supports the alignment direction you need; some wrappers expose only depth to color.
4. Run the edge-target test and check the measured shift matches the prediction within a few pixels.
5. Budget the per-frame alignment step in the cycle time.

If you are specifying hardware rather than fixing software, compare 3D sensors on their per-stream fields of view and on whether color leaves the camera already registered:

The color channel can also come from a separate area scan camera mounted beside the 3D sensor; the two devices then need an extrinsic hand-eye calibration instead of on-camera alignment.

Carry the two fx values into the station specification: they decide the alignment design more than any other number. The surrounding pipeline, from triggering to deprojection, is covered in more 3D camera guides.