Stream left and right stereo image pairs from a depth camera

You configure a stereo depth camera, start streaming, and get a depth image plus one grey frame. The two raw camera feeds exist inside the device; the SDK hands out its computed depth image and the left frame unless you ask for the pair explicitly. This walkthrough shows how to enable both same-type streams, pull the left and right frames out of each frameset by stream index, and prove which frame came from which lens. It is for anyone integrating a binocular stereo or structured-light depth camera who needs the raw pair for their own matching, their own calibration or their own dataset capture.

What stream index 0, 1 and 2 mean

Depth SDKs that expose more than one camera of the same type share one indexing convention. A stream type with a single active stream runs at index 0: the depth stream alone, or the infrared stream when only the left camera is enabled. Enable two cameras of the same type and they take indices 1 and 2, left at 1 and right at 2. The number belongs to the stream type, not to the device, and it says nothing about whether a stream is measured directly or synthesized in software.

That convention is why the convenient helper fails. A call that returns the first frame of a stream type stops at the first match, the first infrared match is the left frame, and most SDKs ship no helper for the right camera at all. You select the right frame yourself, by index.

Steps: enable the pair and grab both frames

What you need: a depth camera with two same-type sensors, its SDK, and its stream profile list. The names below — stream type, stream index, stream name — are the generic forms every depth SDK exposes; read your SDK's exact spelling from the profile list before you write the configuration.

1. List the stream profiles and find the same type twice: two infrared profiles, 8-bit greyscale, at the resolution and frame rate you want. Enable only profiles the list holds; requesting an unlisted mode lands you in no matched video stream profile debugging instead.
2. Enable the infrared stream type twice, once at index 1 and once at index 2, and keep depth at index 0 if you still want the computed image. Do not pass a wildcard and let the SDK choose: it can settle for one infrared stream, and you will not notice until the selection code runs.
3. Start the pipeline with that configuration. Each grab returns one frameset holding the synchronized pair plus depth, so left and right stay matched frame for frame.
4. Iterate the frameset instead of calling a first-of-type helper. Match on stream type and take index 1 as left, index 2 as right, index 0 as depth.
5. Log each frame's stream name once at start-up. The name is the SDK's human-readable label for the camera and your cross-check that index 2 is really the right-hand sensor on your unit.
6. Run the cover test below before you build any processing on top.

Select the right frame in the grab loop

The block below is generic pseudocode: the variable names are illustrative and declared with their initial values, and the attributes you match on — stream type, stream index, stream name — exist in every depth SDK under some spelling.

frameset = pipeline.wait_for_frames()   # one synchronized grab
left_ir  = None    # filled when a stream of type INFRARED, index 1, arrives
right_ir = None    # filled when a stream of type INFRARED, index 2, arrives
depth    = None
for frame in frameset:                  # a frameset iterates one frame at a time
    if frame.profile.stream_type == "INFRARED" and frame.profile.stream_index == 1:
        left_ir = frame
    if frame.profile.stream_type == "INFRARED" and frame.profile.stream_index == 2:
        right_ir = frame
    if frame.profile.stream_type == "DEPTH":
        depth = frame

print(frame.profile.stream_name)        # start-up check: read the labels once
assert right_ir is not None             # fails loudly if index 2 was never enabled

If the assert fires, the right stream was never enabled: fix the configuration step, not the loop. If the loop stays empty, check formats before you suspect the code — the infrared pair is commonly 8-bit greyscale against 16-bit depth, and a read-only pixel format usually means the scan type must be set first. Some devices pack both cameras into one frame; then you split the packed stereo frame into two 8-bit images instead of selecting by index.

Prove which frame is which

Two checks settle it. The stream name in the start-up log gives the SDK's own label, left or right, for each index. Then cover one lens with a finger or tape while streaming: the stream that goes dark is the camera you covered, so note which index went dark and repeat on the other lens. Both infrared frames look almost identical on a plain scene because they see the same projected pattern, slightly shifted; that shift is the disparity your matcher will use.

If one stream stays dark with the lens uncovered, the fault is upstream of the indexing: work through the checklist for a stereo pair where one camera shows no live image, which starts at the sensor slot and cabling. And when you record captures for calibration, verify streams and metadata at capture time, as the guide on raw stereo image calibration lays out.

Worked example: what the raw pair buys you

With both raw frames you can run your own matcher instead of the on-device algorithm and tune it for your part. Three numbers decide whether that is worth the effort:

f is focal length, p is pixel pitch, B is the stereo baseline, Z is the working distance, d is the disparity in pixels and dZ is the depth error caused by one pixel of disparity error.

Illustrative module, chosen to make the arithmetic easy to follow: B = 55 mm, f = 1.9 mm, p = 3 µm, so f_px = 1.9 mm / 0.003 mm = 633 px. At Z = 600 mm the expected disparity is d = 633 × 55 / 600 = 58 px. One pixel of disparity error at that distance moves depth by dZ = 600² / (633 × 55) = 360,000 / 34,815 ≈ 10 mm. Double the distance and the same pixel error costs four times as much: 1,200² / 34,815 ≈ 41 mm. That quadratic growth, not sensor resolution, is what usually caps a stereo working distance.

Decision: if roughly 10 mm per disparity pixel meets your tolerance at 600 mm, building your own matcher on the raw pair is worth it. If it does not, widen the baseline or move the camera closer; if neither is possible, stay on the device's tuned depth output.

To compute depth yourself, rectify both frames and feed them with the calibration — stereo depth from host images lines up the inputs in order — and be careful which intrinsics apply, because raw and rectified IR frames take different ones. For the matching step itself, the SSD block matching walkthrough shows how a disparity map is actually computed.

Common mistakes

• Calling the first-of-type helper. It returns the first infrared frame, the left one, and your right image stays empty with no error raised.
• Enabling the infrared type once. One enabled profile means index 0 and a single stream; the pair only appears when the type is enabled twice, at indices 1 and 2.
• Trusting a right-only configuration. If it is software-emulated from the left image, your stereo output is fiction; compare the two frames bit for bit before relying on either.
• Mixing the pipeline API with the sensor-level API in one program. Use the pipeline for framed grabs; drop to sensor level only for options the pipeline cannot reach, and stay there.
• Assuming every device numbers its streams the same way. The profile list and the stream name are the authority; the selection structure stays identical either way.

When no pair exists

Some depth hardware has no pair to fetch, and no stream setting creates one. A ToF depth camera measures light travel time on its own sensor and outputs a single depth image. A laser profiler triangulates one sensor against a swept laser line. On the question that often starts this hunt: the stereo pair is not colour. The matcher needs the projected pattern, which is projected in the infrared, so the pair is two mono images and the colour camera is a single texture stream added on top. Read the profile list once: if the same stream type does not appear twice, the path to a stereo pair is a hardware choice, not a configuration.

Cameras this applies to

Binocular structured-light stereo cameras are the hardware class here: two imaging sensors and a projected pattern, built so the raw pair is the native output and depth is the computed one. Browse the 3D sensors range or the camera finder, filtering for stereo and structured-light types:

Both compute depth on board; taking the raw pair as well is for your own matcher, your own calibration and dataset capture for training.

Where to go next

Confirm on your unit first: one call to the profile list tells you whether both raw streams exist. If you are specifying hardware for a new station, the camera configurator steps through sensor, mount and interface, and the 3D camera topic page collects the calibration and streaming guides that build on this one. Every order includes free expert support, so you can ask us to review the stream configuration for your camera before you write the processing code.