Depth camera SDK layers: driver, middleware and your code

Unzip a depth camera SDK and nothing asks to install itself: no wizard, no entry in the program list, just a folder of libraries, samples and a readme. The usual first reaction is that something went wrong. Nothing did. A depth camera stack has three layers, and only one of them — the USB or Ethernet driver — is something the operating system registers. This guide names each layer, shows which settings and failure modes live where, and follows the bytes from sensor to point cloud with the arithmetic shown. When you finish, you can place any symptom — missing device, missing frames, missing data — in the right layer and fix it there.

Three layers, one pipeline

Every depth camera — structured light, time of flight or stereo — feeds your code through the same three layers.

1. The driver. The component the operating system loads for the camera's USB or Ethernet connection. It owns enumeration — whether the camera appears at all — and moves packets across the wire. Your code never calls it directly.
2. The middleware, which is the SDK itself. A vendor library that opens the device, negotiates one of the modes the camera lists, decodes the packed depth data, optionally registers depth onto the colour image, and hands finished frames to any client that asks. This is the folder you unzipped: header files, runtime libraries, plugins loaded on demand, and sample executables such as a simple depth viewer.
3. Your application code. Everything downstream of a finished frame: convert depth to XYZ, segment, track, measure, decide.

A fourth component rides along without being a layer: the firmware inside the camera. SDK and firmware releases come as matched pairs, and mixing versions is a classic way to break a stream that used to work. If you build the scanner yourself from a camera and a projector, an open-source framework such as SLStudio takes the middleware role, bundling calibration, pattern projection and reconstruction in one package.

Structured-light cameras show the layering at its most literal: a projected pattern, that pattern deformed by the scene, and middleware that turns the deformation into depth. If you are still choosing between the 3D technologies themselves, the industrial 3D camera selection guide compares structured light, stereo and time of flight at system level. The shop's 3D sensors range spans laser profilers, structured-light and time-of-flight cameras, and all of them drive through the same three layers.

Time-of-flight cameras swap the physics underneath — the sensor times its own emitted light instead of decoding a projected pattern — but the boundaries between driver, middleware and your code stay exactly where they are.

Why nothing appears in the program list

The program list only shows software that ran an installer and registered itself with the operating system. A depth camera SDK does neither, because it is not a program. It is a set of libraries your project links against, plus runtime plugins that load when a device is opened, plus sample executables you run straight out of the folder.

That is why unzipping the SDK and setting the environment variables named in its readme is a complete installation. The variables tell the operating system's loader — and the SDK's own device-discovery code — where the runtime libraries live. Miss them and the camera still appears in the device list while every SDK call fails; the classic pattern is a context that refuses to open or a crash on first start, which the 3D camera context creation crash guide works through.

The first preview costs nothing more. Find the viewer sample executable inside the SDK folder and run it: live depth in that window proves the driver, the firmware and the middleware in one test.

What each layer owns

Two boundaries in this table cause nearly all the confusion. The driver-to-middleware boundary is invisible from your desk: when samples cannot find libraries, enumeration was never the broken part. The middleware-to-code boundary is where units and coordinate frames get settled — depth arrives as integer counts, 16 bits per pixel on most cameras, and the SDK documents the scale that turns counts into millimetres.

From first preview to point cloud

1. Run the viewer sample before writing code. It exercises driver, firmware and middleware without a line of your own, so anything it shows is not yours to debug.
2. Pick the depth mode your application needs and request it by its exact entry in the mode list. Modes bundle resolution and frame rate, and a name that is almost right negotiates nothing — the no matched video stream profile guide is the reference case.
3. Enable depth-to-colour registration only if you need colour aligned pixel-for-pixel with depth. Registration costs host processing and can soften depth edges; skip it while you are only measuring geometry.
4. Read depth frames as integers and scale them with the unit the SDK reports. Save one raw frame to disk on day one; it becomes the reference when a measurement looks wrong months later.
5. De-project each pixel to XYZ with the factory intrinsics the SDK delivers:

Here u and v are the pixel's column and row, cx and cy the principal point, and fx and fy the focal lengths expressed in pixels. All four come from the factory calibration, never from your measurements. A part that reads consistently short is a scale question, and the depth map to real distance calibration guide shows how to pin the scale against a known part.

6. Hand the cloud to your processing stack. The open-source Point Cloud Library (PCL) is the common destination for segmentation and tracking: the measure distance and bend angle from depth point clouds guide starts from exactly this kind of cloud, and the point cloud optical frame transform guide covers the frame change a robot expects. Building that first project against an SDK on an ARM board has its own traps — CMake Boost targets undefined is the pattern there.

What the pipeline moves: a worked example

Take a representative mid-range mode and follow the bytes. Assumptions: 640 × 480 depth at 30 fps, 16 bits per depth value, plus registered colour at 8 bits per channel. Your camera's mode list sets the real numbers; the arithmetic is identical for any mode.

• Pixels per frame: 640 × 480 = 307,200.
• Depth on the wire: 307,200 × 2 B = 614,400 B per frame; at 30 fps that is 18,432,000 B/s ≈ 18.4 MB/s, or 147 Mbit/s.
• Registered colour: 307,200 × 3 B = 921,600 B per frame; at 30 fps that is 27.6 MB/s, or 221 Mbit/s.
• Combined camera traffic: 18.4 + 27.6 = 46 MB/s. USB 3's 5 Gbit/s line rate leaves roughly 500 MB/s of payload after encoding, so one camera is easy and two or three fit with care. USB 2's 480 Mbit/s line leaves well under 40 MB/s of payload, so it cannot carry the 46 MB/s this mode needs — which is why a shared hub shows up as streams that refuse to start. The depth camera on a USB hub guide is that exact failure.
• Point cloud in your code: 307,200 × 3 × 4 B = 3,686,400 B ≈ 3.7 MB per cloud; at 30 Hz your application touches 110.6 MB/s of memory, or 4.6 MB and about 138 MB/s with colour attached.

The decision these numbers give: the wire is not the bottleneck, the point-cloud stage is. Converting 110 MB/s of clouds at full rate is processor work, and that is the number that sizes the machine below.

Sizing the machine under the stack

Decode and registration run on the host CPU, so the machine under a depth camera matters more than under a comparable 2D camera. A vision controller gathers what the stack needs in one fanless box: a USB 3 or LAN port per camera, separate ports for the plant network, and built-in light channels so illumination fires in step with capture.

A single-camera bench at the worked example's rate sits fine in the entry class. The MV-VC3301H-128G66 pairs an Intel Core i3 with 8 GB of memory, 5+ LAN ports and 4 built-in light channels. Two cameras at full rate plus lighting want the faster class. The MV-VC5525-256G70-NN runs an Intel Core i5 with 32 GB of memory, 3-4 USB 3 ports, PoE camera ports and 8 built-in light channels. Memory is rarely the binding constraint — a ten-frame backlog of 4.6 MB clouds is 46 MB — processor and port bandwidth bind first, so buy the faster CPU class before more RAM.

Five-minute triage by layer

Work down the list and stop at the first test that fails.

1. Camera absent from the device list (Device Manager on Windows, lsusb on Linux). Driver or physical layer. Move the camera to a rear-panel USB 3 port directly on the machine, never through a hub. For a GigE model, ping it and fix the subnet first: the 3D camera not pingable on a Linux controller guide walks that check, and the depth stream missing on Linux guide traces where frames stop on the OS side.
2. Device present, viewer shows no frames. Middleware mode negotiation. Request a listed mode exactly and check the pixel format your build decodes: the unknown pixel format guide and the empty depth image buffer guide cover the two common versions. A flickering preview in a visualization tool points back down at the driver — fix the driver first before touching configuration.
3. Viewer works, your program finds nothing. Environment and paths, not the driver. Re-check the readme's variables, match the runtime libraries to your build's 32- or 64-bit architecture, and reopen the IDE after changing variables so the new environment is inherited.
4. Everything opens, capture slower in your code than in the viewer. Your stream handling. Copy the viewer's buffer and stream settings before optimizing your own loop: the frame rate low on software trigger guide and the capture time slower in the API guide both live at this boundary.
5. Streams clean, numbers wrong. Calibration, units or coordinate frames — installation was never the problem. Compare an exported cloud against the viewer's render first; differences there are systematic, as why exported point clouds differ explains, and the scale fix lives in the depth map calibration guide linked above.

Checks before you write code

• Run the viewer sample first; it clears driver, firmware and middleware in one test.
• Copy mode names from the SDK's mode list character for character; near-misses negotiate nothing.
• Set the readme's environment variables, then restart the IDE or shell so running processes inherit them.
• Record the SDK release and camera firmware as a pair in your project notes; updates ship matched, and the firmware and SDK alignment guide is the mismatch pattern.
• Give every camera its own host-controller port; hubs pool bandwidth and reset devices mid-run.
• Keep one raw depth frame from day one as the reference for scale and units.

The decision, case by case

• Nothing in the program list after unzipping the SDK: normal, not a fault. The install is unzip plus the readme's environment variables; verify with the viewer sample.
• Camera missing from the device list: driver or physical layer. Direct USB 3 port, or the right subnet for GigE.
• Camera present, no frames: middleware mode negotiation. Request a listed mode by its exact name.
• Viewer fine, your code blind: environment and paths, then your stream settings.
• Frames fine, numbers wrong: scale, units and frames. Pin depth scale against a part of known size.
• More than one camera, or cameras plus lighting on one cycle: move the stack onto a vision controller with a port per camera.

Every symptom ends one layer above where it starts. If you would rather have the whole stack sized as a system — cameras, controller, light channels, trigger wiring — the 3D camera finder filters the current range by type and interface, and the team answers layout questions through the contact page. For the layer above this one, turning clouds into accept-and-reject decisions, the 3D topic page collects the guides on this site.