Temporal depth filter shows no difference: feed it moving frames

You wire the temporal filter exactly as the manual shows, set alpha and delta, and render the filtered and unfiltered depth maps side by side — and the two windows are pixel-for-pixel identical. Nothing is broken. A temporal depth filter is a history filter: its first output frame equals its input by construction, and its job on quiet pixels is to stay invisible. This guide is for engineers adding depth post-processing in Python. It explains where the identical-output behavior comes from, lists the five pipeline faults that genuinely disable the filter, gives you a changed-pixel count that proves the filter runs, and works out the settling time that tells you when a difference should appear.

Why the first output frame equals the input

A temporal depth filter is a first-order recursive average, run once per pixel. Each frame, every output pixel mixes the new measurement with that pixel's own accumulated history:

The weight is the option most depth SDKs name filter_smooth_alpha. At 0.4 the current frame contributes 40% and the history 60%. The frame that opens a session has no history, so the accumulator starts from that frame's own values — and the output equals the input, pixel for pixel. Replay the same static scene and the arithmetic never moves: identical input, identical output. Two identical windows are the design working, not a fault.

The second knob, filter_smooth_delta, is a change threshold in depth units. When a pixel's new value departs from its history by more than delta, that is a real scene change. The filter resets that pixel's history and takes the new value straight away, which is what keeps a moving edge from smearing into a ghost. Changes smaller than delta count as noise and get averaged down. So the filter stays silent exactly while nothing moves, and it speaks within a few frames once something does. The same arithmetic holds whether the depth comes from a dedicated 3D sensor or from a stereo pair built around two area scan cameras: history is history.

Rule out these five causes, in order

The identical-image symptom has five roots, and each test below takes seconds. Work down from the top: the first three cover almost every case.

Cause 1 is the most common by far: the filter is doing its job by refusing to invent differences. Causes 2 to 4 are wiring mistakes. Cause 5 is tuning — with delta under the noise band, nothing ever averages.

Run the filters before depth-to-color alignment

Keep the pipeline in the order acquire raw depth, temporal filter, spatial filter, then align to color. Post-processing belongs on the camera's native depth geometry. Alignment resamples depth onto the color camera's grid, and resampled pixels are already interpolations, so filtering after alignment works on softened data and its effect looks smaller. The raw-versus-rectified distinction is laid out in Depth to color registration: raw versus rectified frames, and the accuracy cost of resampling in Upsampled depth frames: alignment resolution vs accuracy.

Alignment first is not forbidden. Some projects measure better with it, and the price is cosmetic: jagged edges where interpolation guesses. The spatial filter, meanwhile, works inside a single frame — it smooths slopes and fills small holes, so its changes appear immediately, but only where holes or steep edges exist. If a reorder leaves color trailing depth by a frame, that is queueing rather than filtering; Color lags depth by one frame: sync RGBD frames by sequence number walks that fault down.

Prove the filter is in the path with a pixel count

Colormaps round; arrays do not. Take both depth maps of the same frame index as unsigned 16-bit arrays and count:

import numpy as np

# raw_u16, filt_u16: H x W numpy uint16 arrays of the same frame index,
# captured after at least 30 frames of live motion; 0 = invalid pixel
diff = np.abs(filt_u16.astype(np.int32) - raw_u16.astype(np.int32))
changed = int(np.count_nonzero(diff))     # pixels that moved at all
big = int(np.count_nonzero(diff > 20))    # moves beyond delta = 20 depth units
print(changed, 'of', filt_u16.size, 'pixels changed;', big, 'beyond delta')

Read the two counts like this:

• First frame of a session: changed = 0 is correct — no history exists yet.
• Thirty frames into live motion, 1.0 s at 30 fps: changed must be greater than 0. A 640 × 480 panel holds 307,200 pixels, so even 1% means 3,072 pixels moved.
• changed = 0 after thirty frames of live motion: the filter is out of the path. Either the display is fed the raw frame (cause 3) or the instance rebuilds every loop (cause 2).
• changed > 0 with big = 0: the filter is averaging sub-delta noise and nothing else, which is exactly its job on a still scene.

Mask invalid zero pixels before counting, the way you would when decoding confidence bitflags; 3D camera confidence image values: decode bitflags and mask bad pixels shows that masking.

Worked example: settling time at alpha 0.4 and delta 20 mm

Inputs, with their sources:

• alpha = 0.4, from the filter_smooth_alpha option.
• delta = 20 depth units, from filter_smooth_delta; at the common 1 mm per unit depth scale that is 20 mm.
• Step h = 10 mm, an illustrative sub-delta change: a hand easing 10 mm closer. It sits below delta, so the filter smooths it instead of accepting it.
• Stream: 640 × 480 at a typical 30 fps.

The rule: a first-order recursive average answers a step of size h with a residual of h × (1 − alpha)^k after k frames.

The arithmetic:

• k = 1: 10 × 0.6 = 6.0 mm outstanding; the first frame has already moved 0.4 × 10 = 4 mm toward the new depth.
• k = 2: 10 × 0.36 = 3.6 mm.
• k = 3: 10 × 0.216 = 2.2 mm.
• k = 6: 10 × 0.047 = 0.47 mm, inside half a millimetre.
• Time to settle: 6 frames ÷ 30 fps = 0.2 s.

Result: at these settings a sub-delta change settles to within 0.5 mm after 6 frames, or 0.2 s of stream.

Decision: give the filter at least 10 frames of live motion, 0.33 s at 30 fps, before you judge it. Identical maps on frame one, or on a frozen scene, are correct behavior. A change larger than delta = 20 mm bypasses the average entirely and lands in both maps at once, by design.

When a dozen lines of your own beat the black box

The stock filter hides its window length. If the application needs a known window and a controlled refresh — the usual case on a station where you also want to freeze depth during a measurement — an explicit running average is predictable to the frame:

import numpy as np
from collections import deque

history = deque(maxlen=8)   # window: the last 8 raw depth frames

def running_average(raw_depth):
    # raw_depth: H x W numpy uint16 array; returns the H x W uint16 average
    history.append(raw_depth.astype(np.float32))
    stack = np.stack(history)                  # shape (n, H, W)
    valid = stack > 0                          # dropouts (0) never count
    total = np.where(valid, stack, 0.0).sum(axis=0)
    count = np.maximum(valid.sum(axis=0), 1)   # never divide by zero
    return (total / count).astype(np.uint16)

# refresh control: keep the result only every 5th frame
# -> a 6 Hz output from a 30 fps stream

With N = 8 frames the box average is fully settled after 8 frames, 0.27 s at 30 fps, and it behaves like a delay of about N/2 = 4 frames, 0.13 s. Recomputing every 5th frame bounds the cost and yields a 6 Hz output. Choose this route when the window and the update rate need to be knobs, or when the average must hold still while you measure; freezing history during the read pairs naturally with the per-part averaging in How to average measurement values per part across repeated scene runs. If the raw noise itself is the complaint, Stereo depth map noise: smooth raw depth into usable data sorts the smoothing options by noise type.

When the fix is the sensor, not the filter

A history filter trades lag for stability. It cannot make a fast-moving part measurable, because during the 6-frame settle above, the part has already moved on. If you reached for temporal smoothing because per-frame depth is too noisy, fix the noise at the source first: exposure, mounting rigidity and working distance all move it. Ranges under the sensor's minimum produce noise no filter survives — Depth camera minimum range: why close objects look like noise explains that floor.

If per-frame stability is still short, pick a sensor that forms a complete depth image from every capture, so history is optional rather than structural. Structured light and time-of-flight designs work that way. A projected structured-light 3D camera and a time-of-flight sensing camera are two current examples:

Prevention checklist

• One filter instance, built before the loop. A rebuilt instance loses its history every frame.
• Pipeline order acquire, temporal, spatial, align. Post-processing on native geometry, resampling last.
• Judge after 10+ frames of live motion. Earlier, identical output is the design, not a fault.
• Compare raw uint16 arrays, never colormaps. Palette bands hide millimetre changes.
• Delta at least three times your noise and no more than the smallest movement that must register instantly. With ±3 mm noise and a 10 mm minimum move, delta = 10 fits; the 20 mm default suits a static fixture.
• Keep the stream's depth scale beside delta. The threshold is in depth units, so the scale sets what it means.

If the conclusion is a depth source with steadier per-frame output, shortlist with the camera finder or spec the whole station in the camera configurator. More depth-pipeline fixes sit on the 3D topic page. If a filter still refuses to show its work, contact us with your pipeline order and two raw frames — the changed-pixel count usually names the cause on sight.