Motion detection with a depth camera SDK trips over one habit carried over from cv2.VideoCapture: asking one frameset for two frames. A frameset is a single capture event, one depth exposure matched with one color exposure, and its color getter hands back the same frame object on every call. Your previous and current images are then the same image, and cv2.absdiff returns black on every pass. This guide is for anyone porting an OpenCV motion detector to a 3D camera pipeline so that detected motion also yields points. You will see why the duplicate happens, the buffer pattern that fixes it, how far apart the two frames really are in time, and how to crop the moving region out of the current frameset for PLY export.
Why does get_color_frame() return the same frame twice?
Because a frameset is one coherent capture, and every get_color_frame() call on that same frameset returns the same frame object: one capture holds one color image, and no second exposure is waiting inside it. The pair you diffed is one image, copied twice.
The pipeline pattern in these SDKs has three steps: enable the streams on a config object, start the pipeline with pipeline.start(config), then loop on pipeline.wait_for_frames(). The wait blocks until the SDK has matched every enabled stream, depth as z16 and color as bgr8 in the classic 640 x 480, 30 fps set-up, into one set. That set is a snapshot of one instant. get_color_frame() is a getter, not a capture, so call it twice and you receive two references to the same image:
frames = pipeline.wait_for_frames()
prev = frames.get_color_frame()
curr = frames.get_color_frame() # the same frame object as prev
cv2.absdiff on those two is zero everywhere, the 50-level threshold finds no contours, and the loop reports no motion forever. The frame differencing for motion detection: subtract two frames pattern is sound; only the source of the second frame is wrong.
How do I capture a real previous and current frame?
Capture one frameset per loop pass, convert its color frame to an array, and keep the previous pass's array in a buffer variable: pass N's buffer is your previous frame, and pass N+1's fresh capture is your current frame.
import cv2
# pipeline, config and rs are the objects the camera SDK provides
config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)
pipeline.start(config)
prev = None # numpy ndarray, uint8, BGR: the previous color image
moving = False # bool: True when this pass found contours
while True:
frames = pipeline.wait_for_frames() # one capture event
curr = frame_to_array(frames.get_color_frame()) # array conversion
if prev is not None:
diff = cv2.absdiff(prev, curr)
diff_gray = cv2.cvtColor(diff, cv2.COLOR_BGR2GRAY)
diff_blur = cv2.GaussianBlur(diff_gray, (5, 5), 0)
_, thresh_bin = cv2.threshold(diff_blur, 50, 255, cv2.THRESH_BINARY)
contours, hierarchy = cv2.findContours(thresh_bin,
cv2.RETR_TREE,
cv2.CHAIN_APPROX_SIMPLE)
moving = len(contours) > 0
prev = curr.copy()
if cv2.waitKey(20) == ord('q'):
break
Two details matter. frame_to_array stands for the call that turns a color frame into a numpy array; swap in the one named in your SDK API reference. And prev = curr.copy() matters because several of these SDKs reuse the frame's internal buffer for the next capture, so store a copy, as frame differencing with two image buffers for change detection explains. One wait per frame is also what keeps frames arriving on schedule, the failure covered in Frame didn't arrive within timeout: one wait call per frame.
Does adding time.sleep() between the two calls fix it?
No, sleeping between two get_color_frame() calls on the same frameset still returns the same frame, and the snippet that appeared to work was fixed by its second wait_for_frames() call, not by the sleep.
The sleep only pauses your Python thread; the frameset it sits between never changes, so with a single wait the second read is still the first image. The working variant added a second wait:
frames = pipeline.wait_for_frames()
prev = frame_to_array(frames.get_color_frame())
frames2 = pipeline.wait_for_frames() # blocks for the NEXT capture
curr = frame_to_array(frames2.get_color_frame())
Delete the sleep. At 30 fps the second wait already blocks for about 33 ms, and 0.1 s of extra sleeping drops the loop from 30 pairs per second to about 7.5, since 0.033 s + 0.100 s = 0.133 s per pair, while delaying every detection by 100 ms. Two waits in one pass work, but the buffer pattern above yields the same pair with one wait per pass and never blocks twice.
How far apart in time are the two frames?
One frame period, 33 ms at 30 fps, because 1 / 30 = 0.033 s, so the pair spans exactly the gap between adjacent framesets, and that gap, not your loop speed, sets how much motion one diff can see.
Worked example, a part on a conveyor:
- Stream: 30 fps color, so T = 1 / 30 = 0.033 s.
- Part speed v = 150 mm/s, so d = 150 x 0.033 = 5.0 mm between the two frames.
- Scale s = 0.75 mm per pixel; measure yours by imaging a ruler and dividing its span in millimetres by its span in pixels.
- Pixel motion p = 5.0 / 0.75 = 6.7 px.
A 6.7 px edge shift clears a 50-level threshold on the blurred gray diff with margin. Exposure sets the blur inside each frame: at 10 ms the same part smears 150 x 0.010 = 1.5 mm, about 2 px, so keep exposure short when parts move fast. If p falls under about 3 px, diff every second frame instead, so the pair spans 67 ms and d = 10 mm, about 13 px, or raise the frame rate.
Once contours are stable, the direction of motion comes from the same diff, as Detect object motion direction with OpenCV frame differencing shows.
How do I verify the two frames are really different?
Log the maximum of diff_gray while you wave a hand in front of the camera: the broken one-frameset version prints 0 forever, and the fixed buffer version jumps above the 50 threshold the moment anything moves.
Add one line inside the loop:
print(moving, int(diff_gray.max()))
Expected readings: 0 forever with the bug on a moving hand, then False with 0 on a still scene and True above 50 with the fix. The second check is the per-frame sequence number these SDKs stamp on every frame: the two frames in your pair must carry different numbers, ideally one frame apart, the same numbering that fixes Color lags depth by one frame: sync RGBD frames by sequence number. If wait_for_frames() raises instead of returning, you have a link or configuration problem, not a differencing problem.
How do I export the points where motion was detected?
Take the depth image from the same current frameset, the one whose color frame produced the contour, crop it to the contour's bounding box, and write it out with the SDK point-cloud export; PLY is the usual output.
The frameset is already the matched pair you need: the SDK paired that depth exposure with that color exposure, so pairing by frameset, or by sequence numbers and timestamps when you queue work, as Color image and point cloud synchronization: pair frames by timestamp describes, keeps the points on the object you detected. The crop is one bounding rectangle around the largest contour, applied to both arrays before export. Whether you keep one combined stream or split capture and export into two passes is the trade-off 2D and 3D capture strategy: combine or split RGB and point cloud works through, and turning the cropped depth into XYZ coordinates follows the conversion in 2.5D height map to 3D point cloud: pixels to XYZ.
Which camera should run this pipeline?
Any camera whose SDK returns coherent depth-plus-color framesets runs this loop: a ToF sensing camera for room-scale motion, a projected structured-light 3D camera for close-range parts, or a stereo camera where surface texture is rich.
For room-scale detection with points, start with a ToF sensing camera:
- Type
- Depth / ToF camera
For close-range parts, a projected structured-light 3D camera gives dense points:
- Type
- Stereo / structured light
Two more routes: a laser galvanometer stereo camera for textured scenes, and a plain USB3.0 area scan camera for the day you decide points are not needed:
- Type
- Stereo / structured light
- Sensor format
- 1"
- Lens mount
- C-mount
That last case deserves a thought: without points you do not need a 3D pipeline at all, and a standard area scan cameras loop with the MV-CS200-10UM, a 20 MP USB3.0 mono camera, is simpler to maintain. The 3D sensors category carries the depth side and board-level cameras the embedded builds; the camera finder and the configurator camera step narrow either list. One habit worth keeping with any projected-light or laser model: do not stare into the emitter at close range.
What else fakes motion between two frames?
Exposure and gain changes: with auto exposure active the camera re-brightens between framesets and whole regions clear the threshold with nothing moving, so lock exposure and gain before you trust a diff.
The starting values are listed in 3D camera 2D stream settings: exposure, iris and gain defaults. Then run this check before commissioning:
- Lock exposure, gain and white balance; any per-frame change in them is a false diff across the whole frame.
- Copy each frame into your own buffer, so the SDK reusing its internal image memory cannot rewrite history.
- Mount the camera rigidly; bracket vibration reads as motion, the failure 3D camera mounting: stillness during capture decides image quality exists to prevent.
- Drive scene lighting from a constant controller output, because intensity that breathes with mains frequency modulates every diff.
- Keep the 5 x 5 blur and the 50-level threshold from the original loop; together they already swallow one or two counts of sensor noise.
The decision, case by case: porting a VideoCapture loop, keep one wait per pass and a copied buffer. Needing points on detection, export depth and color from the same current frameset, cropped to the contour. Chasing speed, delete the sleep and let the frame period pace you.
Questions
How many frame pairs per second does the fixed loop deliver?
One pair per frameset, so 30 pairs per second at a 30 fps stream. Each wait_for_frames() blocks until the next coherent set, which paces the loop for you. If the absdiff and contour code takes longer than 33 ms, frames queue behind the pipeline and you fall behind real time; measure your processing time, and when it passes 33 ms, diff every second frame so the pair spans 67 ms.
Why is absdiff still all zero after the fix?
Three usual reasons: prev and curr were both taken from one frameset; prev points at a buffer the SDK overwrote before you used it, so store a copy; or the scene is genuinely still. Log diff_gray.max(): it must rise above your 50 threshold when something moves, and the two frames' sequence numbers must differ. If both checks pass, capture is correct and the scene simply has no motion.
Do I need the depth stream enabled to detect motion?
No. Color alone detects motion, and disabling depth cuts bandwidth and USB load. Keep depth enabled when you export points on detection, because the current frameset then already carries a depth image paired with the color frame that found the contour. Depth-only differencing also works: absdiff runs on the z16 arrays unchanged, and a threshold of 50 then means 50 depth units rather than 50 gray levels.
What timeout should wait_for_frames() get?
Give the wait a timeout comfortably above one frame period, 100 ms at a 30 fps stream being a safe value, through the timeout setting your SDK wait call exposes. A timeout at or below the frame period fails on the first slow frame. If the call still times out, the fault is in the link, power or stream configuration, not in the differencing code.
How do I keep the exported PLY aligned with the detected region?
Export from the same frameset you diffed as current, because the SDK matched that depth exposure to that color exposure. Crop the point cloud to the contour's bounding box before writing, so the PLY holds only the moving object. When detection and export run in separate threads or processes, pair the frames by the SDK per-frame sequence numbers or timestamps instead of by arrival order.



