Person detection with a depth camera: choosing a 3D sensor that works

Person tracking is the application that strands teams: a sample package that shipped against one depth camera's unreleased middleware gets pulled, and the clone you found no longer builds against anything. The camera on your bench still measures distance — what died was the software layer wrapped around it. This page is for integrators building people presence, counting and tracking cells with 3D sensors. You will leave with the rule that makes the next camera swap painless — the sensor publishes standard depth frames over USB3 or GigE, and the detector is a separate layer you own — plus the numbers that choose between time-of-flight and stereo, the pixel budget that sets detection distance, a lens calculation for the RGB camera you may add, and the checks to run before you order.

Why did my person-tracking package stop building?

Because it was never a standalone tracker: the code called one camera family's middleware, that middleware was withdrawn before its public release, and no compiler fix restores a dependency the supplier no longer distributes. The pages of errors in your build log are real; their cause is not in your tree.

Middleware that never reached release cannot be fetched from a mirror, and camera software belongs with its manufacturer or distributors in any case. There is nothing to recover and nothing to patch. What broke was the coupling, not the camera — the sensor on the bench still measures distance today.

Design against that ending, because every camera generation reaches it. The rule for the rest of this page: the camera's whole job is to output depth frames or an organized point cloud through a documented interface, and person detection is a layer you own, written against the data rather than against one vendor's demo.

Time-of-flight or stereo: which depth camera detects people best?

Indoors at 0.3–8 m, choose a time-of-flight camera; outdoors, in sunlight, or past 8 m, choose a stereo pair — and rule out multi-frame structured light, because its projected pattern sequence needs a still target and people never hold still.

A time-of-flight camera floods the scene with modulated infrared and reads range per pixel from the returning phase — indirect time of flight: how phase becomes depth covers the mechanism. Indoors, depth error runs near 1% of distance, about 40 mm at 4.0 m, so a 1.8 m person standing on a fitted floor plane separates from furniture with a 45:1 height-to-noise ratio.

A stereo pair solves depth from disparity between two synchronized images, and its error grows with the square of distance while shrinking with baseline:

With a 900-pixel focal length, a 120 mm baseline and 0.5-pixel disparity noise, the error at 4.0 m is 4000² × 0.5 / (900 × 120) = 74 mm, and at 8.0 m it is 296 mm. Setting the error budget at 150 mm gives that pair a confident counting distance of 5.7 m; widening the baseline to 200 mm drops the 8 m error to 178 mm. Baseline is your range dial.

Sunlight loads the same infrared band a time-of-flight camera modulates, so outdoor cells belong to passive stereo with a narrow infrared pass filter and short exposure. Indoors the reverse holds: plain clothing gives a passive pair almost nothing to match, and projected texture is the fix. Stable tracks also need low depth jitter — stereo depth jitter and IMU drift can cost you more accuracy than the error budget above.

The MV-DT01WMU is a time-of-flight sensing camera for the indoor rows above.

The MV-DLS300P-04 is a laser galvanometer stereo camera — projected texture of exactly the kind that carries matching across plain fabric.

For the counting-specific trade-offs, read active infrared depth versus stereo for people counting.

How many depth pixels does a person need?

Budget 30 pixels on a person's height for presence detection, 80 for pose or occupancy grids; a 320 × 240 depth image with a 70° × 55° field of view holds past 8 m, so rated range gives out before resolution does. Those are the figures of a common compact time-of-flight grade, and they generalize to any wide-field QVGA unit.

Worked example, using a 1.8 m reference person:

• Vertical pixel angle: 55° / 240 px = 0.229° per pixel (horizontal: 70° / 320 px = 0.219°).
• At 4.0 m the person subtends atan(1.8 / 4.0) = 24.2°, which is 24.2 / 0.229 = 106 pixels of height.
• At 8.0 m: atan(1.8 / 8.0) = 12.7°, which is 55 pixels — still nearly double the floor.
• The 30-pixel floor is not reached until 1.8 / tan(30 × 0.229°) = 14.9 m, past any compact sensor's rated indoor range.

Decision: on wide-field QVGA depth, rated range and mounting set your detection distance — spend your budget there, not on depth resolution.

In Z the case is even easier: the person stands 1.8 m above a fitted floor while depth noise is about 40 mm at 4 m, and no lighting change moves that ratio. Mount on a wall at 2.2 m and the 70° horizontal field covers 2 × 4.0 × tan 35° = 5.6 m of floor width at 4 m — one camera watches a full corridor lane. 3D camera working distance and positioning works through that geometry.

When the usual answer is wrong

• Narrow-field units flip the failure: at 25° horizontal, 320 px means 0.078° per pixel and a person at 8 m spans 163 px — detection is easy, but the frame is only 2 × 8 × tan 12.5° = 3.6 m wide, so coverage fails before detection does.
• Past its unambiguous range a time-of-flight camera wraps phase, and a person at 9 m can plot at 1 m — read choosing a time-of-flight unambiguous range before trusting far readings.
• Matte black PPE absorbs the infrared return at the far end of range; rate your cell at two-thirds of the sensor's rated distance and the holes stay outside your alarm zone.
• Compact time-of-flight stops near a 0.3 m minimum range, so an overhead camera at 2.5 m is blind directly below itself — lay out that dead circle before you drill.

What frame rate and interface does person tracking need?

Take 30 fps and any gigabit-class link — USB3 or GigE — for walking-speed people; a 320 × 240 depth stream at 30 fps is about 28 MB/s as XYZ points, under a third of GigE's usable bandwidth. That is the whole interface decision for a room or cell.

The bandwidth arithmetic: 320 × 240 = 76,800 points, × 30 fps = 2.30 million points per second, × 12 bytes for three 4-byte floats = 27.6 MB/s. GigE carries roughly 100 MB/s effective and 5GBASE-T several times that on the same cabling — 5GBASE-T camera interface: bandwidth, cables and distance covers the long runs.

At a 1.4 m/s walking pace, 30 fps means 1400 / 30 = 47 mm between frames, and a nearest-neighbour tracker holds identity with no motion model. At 15 fps the same walker covers 93 mm and a 2.4 m/s runner covers 160 mm, which pushes prediction into your tracker. Specify 30 fps sustained at full frame; accept 15 fps only in slow zones.

The plug follows the cable run: passive USB3 holds about 3–5 m, while GigE reaches 100 m on copper — which is why ceiling- and gate-mounted people sensors are usually GigE.

Do you still need an RGB camera next to the depth sensor?

No for presence, counting and zone alarms — depth alone answers them without ever storing a recognizable face; yes the moment the task needs appearance, and then a global-shutter color camera beside the depth sensor is the correct pick. Depth-only is also the privacy-friendly default, because a range image identifies nobody; if you add RGB, that stream carries personal data, so apply your local privacy rules to it, retention limits included.

When you do add RGB, size it like any area scan cameras purchase. Example numbers: a 1/1.7-inch sensor (about 7.4 × 5.5 mm) with 4000 × 3000 pixels gives 5.5 mm / 3000 px = 1.83 µm pixels. Mount it 4.0 m from the lane and cover a 2.4 m tall window — a 1.8 m person with headroom:

• Magnification m = 5.5 mm / 2400 mm = 0.00229.
• Focal length f = WD × m / (1 + m) = 4000 × 0.00229 / 1.00229 = 9.1 mm — fit a standard 8 mm C-mount lens.
• At 8 mm the vertical field is 5.5 mm × 4000 / 8 = 2.75 m, so the window fits with 0.35 m of margin.
• The person's 1.8 m images to 1800 × (5.5 / 2750) = 3.6 mm, which is 3.6 mm / 1.83 µm = 1970 pixels tall.

Run your own numbers in the lens focal length calculator.

Motion sets the exposure: a 1.4 m/s walker at 4.0 m sweeps 0.35 rad/s, which through an 8 mm lens is 2.8 mm per second on the sensor. A 1 ms exposure therefore smears 2.8 µm — about 1.5 pixels at 1.83 µm pitch. Hold exposure at or below 1 ms, start near f/2.0 to f/2.8 under indoor ambient light, and add scene lighting before you stop the lens down.

The MV-CU020-90GC is a 2.5 MP color GigE camera with a global shutter and a 1/1.7" sensor that fills this role.

How do you keep the person detector replaceable?

Make the camera's only contract a depth frame — any format your code can read from the manufacturer's current SDK — and put detection in its own process, so a camera change touches one adapter instead of the whole pipeline. Three layers, each replaceable on its own:

1. Acquisition. The manufacturer's current SDK, with camera firmware and SDK versions pinned together — version skew is the top cause of silent capture failures, as 3D camera header version mismatch: fix firmware and SDK alignment and pinning one library version everywhere both show.
2. Neutral data. Convert immediately to a depth image plus intrinsics, or an organized point cloud. No vendor pixel format crosses this line.
3. Detection, yours. Fit the floor plane, delete it, cluster what stands 0.8–2.2 m above it — from seated users to tall people — and keep clusters of 30 pixels and up, tracked by nearest neighbour at 47 mm per frame.

A ROS node boundary is one clean way to enforce the split, and when a driver behaves in the vendor viewer but not in your node, the break is on the integration side — when a tracking camera works in the viewer but not in ROS walks through that class of failure.

The payoff: when this camera's SDK is eventually retired, you re-point layer one at the new sensor and the detector never notices. That is the piece the stranded package never gave you.

What should you check before ordering a depth camera for people?

Check five lines on the data sheet in this order — rated range, depth output format, frame rate at that range, host interface, and SDK availability today — because a missing SDK page kills a project faster than any specification gap. Then run the layout numbers against this list:

• Rated indoor range at least 1.5× your farthest person. Multipath off racks and glass, plus dark clothing, consume the last third of range.
• Depth image or organized point cloud output. A viewer render alone is not an interface your detector can read.
• 30 fps sustained at full frame, in the mode you will actually run.
• Interface matched to the cable run. USB3 under 5 m; GigE to 100 m.
• SDK on the manufacturer's site today, versioned so you can pin it, with expert support behind it.
• Near limit below your closest approach. Compact time-of-flight units stop near 0.3 m.
• Illuminator safety as shipped. The infrared source in a current 3D sensor is engineered to IEC 60825 Class 1 eye safety — keep its window in place and never drive a light source past its rated duty cycle.
• In production status. Filter the 3D sensors list by Availability: In production; an end-of-life sensor is the stranded-SDK story retold.

Start in the camera finder, or configure a camera with your cell dimensions — range, pixels and lens size fall out of the arithmetic above.