Widen the field of view of a body tracking depth camera setup

Full-body skeleton tracking with a fixed depth camera hits two walls at once. The field of view is narrow enough that a standing adult only fits in frame from about two metres away, and the tracked joints jitter enough to make the output feel shaky. This guide is for anyone building a body tracking station — interactive installations, gesture control, rehab and sport metrics — who has considered fixing it in hardware.

You will get the geometry that converts field of view into standoff, the reason the lens is not a replaceable part, and the two software changes that remove most of the jitter: a 640x480 depth stream and a small deadband on joint positions. The triangle behind how camera field of view changes with working distance governs your station too; the difference is that a depth camera does not let you re-cut the triangle with a new lens.

The decision up front

Three answers cover the whole problem.

1. The lens is not a spare part. A depth camera triangulates between a projector and an imager through factory-calibrated optics, so no 3D sensor in the body tracking class takes a swapped lens. There is no wider optic to buy for the module you own.
2. Standoff is the lever, and it scales linearly with the height you need in frame. For a standing adult on the 58° to 70° angles this class specifies, budget 1.8 to 2 m of clear space. The common field report — a full person only fits from two metres back — is the geometry, not a defect.
3. The shakiness is a sampling problem, not an optic problem. Run the depth stream at 640x480 instead of the 320x240 the sample code starts with, and hold each joint still until it moves more than about 12 mm. Most of the visible jitter goes away.

Why the lens is not replaceable

A structured-light module projects an infrared pattern and reads how that pattern deforms over the body. Depth comes from triangulation across the projector-to-imager baseline, matched against reference images captured through those exact optics at the factory. Change either optic and the reference no longer matches the projection: the depth values come out wrong, not merely soft. Stereo modules fail the same way through their rectification tables, and time-of-flight modules through per-lens amplitude and offset calibration.

A 2D camera survives a lens swap because the image is still an image. That is why machine vision lenses are a whole catalog of focal lengths for area scan cameras, while depth modules ship with integrated optics, focus-set and sealed at the factory. Nothing on the module is meant to come off.

The geometry: standoff scales with body height

Call the frame height H — the body plus margin — and the vertical field of view theta_v. The standoff that fits H in frame follows directly:

The relation is linear in H: track twice the height, stand twice as far back. Nothing else in a fixed setup buys coverage.

For the standard case, H = 2.0 m — a 1.8 m person with 0.1 m above the head and 0.1 m below the feet, because trackers drop head and foot joints that touch the frame edge:

Two details the spec sheet hides. The usable field is smaller than the optical one: edges fall off, and stereo rectification crops outright, which is why stereo depth coverage is smaller than the stated field of view. And the horizontal axis matters when arms go up — fingertip-to-fingertip span runs about the same as height, so at this frame height the horizontal standoff lands within a few centimetres of the vertical one whenever the two angles are close.

Mount the camera at half the frame height, 1.0 m here, so the body sits centred top to bottom. The sizing walk-through for a general camera is in camera field of view at a given mounting height.

Worked example: a 1.8 m person at a 60 degree vertical FOV

Inputs, each with its origin:

• Person height 1.8 m — the usual adult design value; use your tallest user.
• Margin 0.1 m above the head and 0.1 m below the feet, so frame height H = 1.8 + 0.1 + 0.1 = 2 m, the standard case above.
• Vertical FOV 60°, the middle of the 58-70° angles this class specifies.
• Mounting height 1.0 m, half the frame.

The rule in words: standoff equals frame height divided by twice the tangent of half the vertical angle.

Arithmetic, written out:

• Half angle: 60 / 2 = 30°, and tan(30°) = 0.577.
• Standoff: 2 m / (2 x 0.577) = 2 / 1.155 = 1.73 m.
• Install at 1.8 m, keeping the margins real against floor slope and tripod placement.
• Cross-check against the field: a full adult needing the whole frame implies theta_effective = 2 x atan(2 / (2 x 2)) = 2 x 26.6° = 53°. Redo the standoff at 53°: 2 / (2 x tan(26.5°)) = 2 / 0.997 = 2.01 m.

Decision: mount at 1.8 to 2 m and design with the effective 53°, not the 60° on the data sheet. If the room gives less than 1.4 m of clear standoff, no single module in this class holds a standing adult — even 70°, the class ceiling, needs 1.43 m. Switch to upper-body tracking or to two cameras.

Remove the jitter: resolution first, deadband second

Body tracking sample code commonly starts the depth stream at 320x240, and that is the direct cause of most shaking. At a 2 m frame height, 240 depth rows put 2000 / 240 = 8.3 mm of body in each row, and a joint estimated from that grid snaps between cells from frame to frame. Consumer structured-light and ToF modules output depth between roughly 240x180 and 640x480, and body tracking belongs at the top of that band: 480 rows give 2000 / 480 = 4.2 mm, half the step. Depth camera pixel size from field of view and resolution is the same division with your numbers.

The cost is host load: 640x480 is 307,200 pixels against 76,800 at 320x240, four times the work per frame. If the frame rate drops with your pipeline running, capture at the higher resolution and process a decimated copy, or stay at 320x240 and lean harder on the deadband. On cameras that expose it, the depth mode is chosen at stream start — changing depth resolution and field of view at capture time is a startup setting, not a live one, and it moves point spacing and capture time with it, as 3D camera subsampling shows.

The tracking framework applies no dampening of its own, so add a deadband: hold each joint at its last position until the new one has moved past a threshold, then accept the new one whole. Set the threshold to three depth pixels: 3 x 4.2 mm = 12.6 mm, so 12 mm at this standoff.

# Example joint dampening written for this guide; not tied to any vendor SDK.
DEADBAND_MM = 12.0   # float, millimetres: 3 depth pixels at 640x480, 2 m frame
last_pos = {}        # dict of joint name -> (x, y, z) in millimetres, starts empty

def damped_joint(name, pos):
    prev = last_pos.get(name)
    if prev is None:
        last_pos[name] = pos
        return pos
    moved = ((pos[0]-prev[0])**2 + (pos[1]-prev[1])**2 + (pos[2]-prev[2])**2) ** 0.5
    if moved > DEADBAND_MM:
        last_pos[name] = pos
    return last_pos[name]

# Test: a right hand at (300, 900, 1800) jumping +/- 2 mm per frame returns
# (300, 900, 1800) every frame; the same hand stepping 20 mm per frame returns
# a new position every frame.

A hand moving 0.5 m/s covers 500 / 30 = 17 mm per frame at 30 fps, above the threshold, so real motion passes at once; quantization noise of about half a pixel, 2 mm, stays held. Two cautions from practice. Jitter concentrates at floor contact points and under self-occlusion, where depth evidence is thin — raise the camera or open the sightline rather than widening the deadband. And test the filter on a moving person: a temporal filter shows no difference on static frames. Depth-side problems such as drift and sync loss mimic jitter too; the stereo camera depth jitter and IMU drift troubleshooting page separates them from sampling noise.

Choosing a replacement sensor: structured light or ToF

Measure the standoff before anything else. Under about 2.5 m, structured light wins: stripe and speckle projection give the cleaner depth map at near range, which is where full body tracking sits. Time-of-flight keeps working at ranges where structured light thins out — published comparisons put the crossover near 10 m — and it suits volume measurement, so a room that forces a long standoff points that way.

Buy against three numbers: the standoff you measured, the vertical angle the data sheet prints, and the highest depth mode the sensor lists. A module that tops out at QVGA-class depth will need a larger deadband to look steady. Compare the fixed geometry side by side before you buy, because on these units the field of view is a selection input, never a later adjustment.

The time-of-flight option in card form:

When one camera is not enough

Below 1.4 m of clear standoff the single-module answer is gone. Three ways out, in order of effort:

1. Track from the waist up. The frame drops to about 1.2 m of body and the standoff to 1.2 / (2 x 0.577) = 1.04 m. Most gesture and interaction work only needs the upper skeleton.
2. Two cameras on one axis with overlapping fields. Each covers half the body and the overlap carries the handover; planning multi-sensor 3D capture for full body coverage sizes the overlap. The mounting discipline is the same as fixing overlapping camera fields of view in robot cells. If both units are structured light, keep the projected patterns from crossing, or capture them alternately.
3. Respect the near limit. Depth modules stop returning depth below a minimum distance, so pushing the camera closer eventually returns nothing at all — depth camera minimum distance versus field of view covers where that wall sits.

Checks before you commit

• Measure standoff along the camera axis to the body plane; room corners do not count.
• Read the vertical angle off the data sheet and design with 5 to 7° less, the margin between the 60° spec and the 53° effective value in the worked example.
• Confirm the host holds full frame rate at 640x480 depth with your whole pipeline running, not with the viewer alone.
• Set the deadband to three depth pixels and convert it with your own frame height; 12 mm assumes the 2 m frame.
• Test the dampening on a person moving at working speed, never on a static scene.
• Watch joints near the floor and at self-occlusion; fix the sightline before widening the deadband.

If the numbers leave you between two sensors, start in the configurator at the camera step and work through it with your measured standoff:

For stations with hard room constraints, bring the standoff and the body height to the contact page and we will size the set-up with you. More depth-camera geometry lives on the 3D vision topic page.