Rectified vs raw stereo IR images: which intrinsics apply

Ask a stereo depth camera for the intrinsics of its two IR imagers and the answer depends on which stream you query. The rectified stream comes back with a full intrinsics set whose distortion coefficients are all zero. The raw stream often comes back with nothing at all. That asymmetry is not a fault: each format carries a different geometry, and each needs a different set of numbers. This guide is for anyone recording IR frames or building a stereo pipeline. It separates the two: what rectification does to the image, which intrinsics belong to which frames, where the SDK stores each set, and what to save alongside a recording so you can still redo the correction offline.

Two IR streams, two geometries

A stereo depth module holds two infrared imagers separated by a fixed baseline. Runtimes for this class of camera usually deliver their output in two formats, commonly labelled Y8 and Y16.

The 8-bit stream is rectified. Before a frame leaves the camera, firmware has applied the factory calibration: lens distortion removed, both images resampled onto a common pinhole model, matching points placed on the same pixel row. That property is what keeps stereo matching light on the processor — the search for a corresponding pixel collapses into a one-dimensional scan along a row — and it is why the rectified pair is the default output.

The 16-bit stream is the raw sensor read-out. No correction has touched those pixels: the lens distortion is still in them, and corresponding points lie on curved epipolar lines instead of shared rows.

The same raw-versus-rectified split runs through a depth camera's other outputs; depth-to-colour registration: raw versus rectified frames covers what changes when two frame types must be mapped onto each other.

Which intrinsics describe which image

An intrinsics set carries four numbers that place pixels in space: the focal lengths in pixels, fx and fy, and the principal point, cx and cy. With a distortion model and its coefficients added, you can undo the lens. The pinhole model is two lines:

A point at X, Y, Z metres in front of the camera lands at pixel (u, v); fx, fy, cx and cy are in pixels. Whether those four numbers are valid for a given frame is decided by which geometry the frame is in.

For a rectified frame, use the set the streaming query returns for that exact profile. Those fx, fy, cx, cy describe the synthetic camera the firmware built by rotating and resampling the physical imager's view onto a common plane — not the lens in front of the imager. The distortion coefficients come back all zero, and that is correct: the image has already been corrected, so any further correction warps it. When rectified data looks bent after you apply a non-zero model to it, that is the double correction at work; the inverse-distortion guide shows the failure mode.

For a raw frame, use the factory calibration: the imager's real focal lengths, principal point and distortion coefficients. The streaming query is defined for the stream models the runtime delivers, so on many cameras it returns nothing for the raw format. The values still exist — they live in the device's calibration table, and the next section shows how to read them. The wider background is in depth camera intrinsics and distortion parameters.

The decision in one table:

Mixing the columns is the classic error. Apply the rectified set to raw frames and the barrel distortion stays in; apply the factory values to rectified frames and every straight edge bows outward.

Where each calibration set lives

Three places hold calibration data, and each answers a different question.

The streaming query. Once frames flow, the SDK call that returns a profile's intrinsics — get_intrinsics in the Python API — gives the set that belongs to that profile. Query the rectified IR profile at its recorded resolution, and use exactly what it returns with those frames.

The enumeration tool in calibration mode. Launch the SDK's device enumeration command, rs-enumerate-devices, with the -c flag: it lists the intrinsics and extrinsics of every sensor at every supported resolution, no streaming needed. This is where you read values offline and where you copy the factory table from.

The maker's calibration read-out utility. Camera vendors ship a small tool that dumps the factory calibration itself — the pre-rectification numbers. Use it when you specifically need the raw values, and store what it prints with the project.

Two habits prevent most calibration mistakes. Read per resolution: every supported resolution carries its own intrinsics set, so values read at one resolution are invalid at another. Read per imager: the two IR imagers sit at different points on the baseline, so their sets differ; label them left and right and never copy one onto the other. If a recorded stream should already carry its calibration as metadata, raw stereo image calibration: verify streams and metadata shows how to check, and the distortion-model explainer decodes the model field when it reads back empty or strange.

Worked example: recording both IR imagers at 1280 × 800

The usual reason to keep raw frames is to stay able to redo the correction later. Here is what that choice costs, in the working case of a stereo module whose IR imagers run 1280 × 800 px, the common resolution for the class.

Inputs, and where each comes from: width 1280 px and height 800 px (the IR imager resolution of the working case); 1 byte per pixel for Y8 and 2 bytes per pixel for Y16 (the format definitions); two imagers recorded together (a stereo pair); 30 frames/s, assumed as a typical streaming rate.

The rule in words: bytes per frame = width × height × bytes per pixel; bytes per second = bytes per frame × frame rate; Mbit/s = bytes per second × 8 ÷ 1,000,000.

The arithmetic:

• Pixels per frame: 1280 × 800 = 1,024,000 px.
• One Y8 frame: 1,024,000 B ≈ 1.02 MB. One Y16 frame: 2 × 1,024,000 = 2,048,000 B ≈ 2.05 MB.
• One imager, Y8: 1,024,000 B × 30 frames/s = 30,720,000 B/s = 30.72 MB/s, and 30.72 × 8 = 245.76 ≈ 246 Mbit/s.
• Both imagers, Y8: 2 × 30.72 MB/s = 61.44 MB/s ≈ 492 Mbit/s.
• Both imagers, Y16: 2 × 61.44 MB/s = 122.88 MB/s ≈ 983 Mbit/s.
• One hour of the Y16 pair: 122.88 MB/s × 3600 s = 442,368 MB ≈ 442 GB (decimal MB).

The result: raw doubles everything, and recording the pair doubles it again.

The decision, with its thresholds. Gigabit Ethernet carries about 950 Mbit/s of payload, so a Y16 pair at 983 Mbit/s does not fit a single GigE link, while a Y8 pair at 492 Mbit/s fits with room to spare for the depth output. A USB 3 link moves about 4 Gbit/s in practice, so it carries the Y16 pair, but the disk must sustain about 123 MB/s for the IR streams alone.

So: if you only need corrected geometry later, record Y8 from both imagers and store the rectified intrinsics set once beside the data — kilobytes of text next to the roughly 221 GB per hour that a Y8 pair fills. Record Y16 only when you intend to redo the lens correction yourself, and pull the factory table with the enumeration tool in calibration mode once, before recording starts, not after. Before you treat any recording as rectified, run the checks in verify recorded stereo frames are truly rectified; if the frames feed a SLAM or mapping pipeline, the stereo recording workflow applies the same discipline offline.

When the usual answer is wrong

Three situations break the stream-to-intrinsics pairing.

The device has been dropped, or its lens has changed. Factory values describe the camera as built. A knock that shifts an element invalidates them, and the fix is a target-based stereo calibration: the target capture and stream setup guide walks through it.

You are solving pose on a rectified image. PnP-style solvers need the intrinsics of the exact image you feed them, rectified frames included — zeros and all. The pose estimation guide compares the sets.

You are deprojecting to a point cloud. Use the set that matches the image being deprojected; when depth and colour both feed one cloud, the aligned point cloud guide settles which camera's values to use.

With the factory table stored, raw frames stay correctable anywhere: the offline rectification guide turns saved calibration data into corrected images with no camera attached.

Checks before you record

A short list; each item exists because skipping it has cost someone a dataset.

• Query the intrinsics of the exact profile you will record — stream, format, resolution — not one that looks similar.
• Confirm the rectified set reports zeroed distortion; that read-back is your proof you hold rectified geometry.
• For raw, run the enumeration tool in calibration mode once per device and save its output beside the project.
• Label every set with imager, stream and resolution; a left/right swap biases every projection.
• Record one short clip in each format, redo the correction offline, and compare a straight edge before committing to a long capture.
• Store the device serial with the calibration text so a swapped unit is detectable.

If you are specifying the camera

Everything above runs on the stereo module itself, so the purchase that matters is the camera: two synchronised imagers, a fixed baseline and a factory calibration you can read out. The shop's 3D sensors range groups the classes that use this workflow — binocular structured-light stereo cameras, laser profilers and logistics volume cameras — and the camera finder filters them by type and series. More capture-side guides sit on the 3D topic page.

Two examples of the stereo class:

The MV-DPS200P-02 is a projected structured-light 3D camera: a projected pattern read as a stereo pair, the same two-imager geometry this article's numbers describe.

The MV-DLS300P-04 is a laser galvanometer stereo camera; the galvanometer steers the projected line across the scene while the stereo pair reads it. Plan the rest of the build at the camera step:

Whichever module you pick, the discipline is the same: read the calibration for the exact stream and resolution you record, store it with the frames, and never mix the two geometries. If you want a set-up checked before you order, contact the applications team — expert support comes with every order.