Moving object detection with a moving camera using depth

A camera moving past a stationary wall produces image motion, just as a moving object does. Thresholding optical flow alone therefore cannot tell you which objects are actually moving. For engineers building depth-based perception, the useful distinction is motion that cannot be explained by the camera's measured change in pose. Use depth to predict the static scene's image displacement, subtract that prediction from measured correspondence, and validate the remaining motion in three dimensions before creating object tracks.

Choose motion compensation before segmentation

Start with calibrated depth, image correspondences and camera poses evaluated at the image exposure times. Predict where each visible static point should appear in the next image. A consistent difference between that prediction and the measured correspondence becomes evidence of independent motion.

This is more useful than subtracting successive images. Brightness subtraction responds to lighting changes and moving shadows, while uncompensated optical flow responds to camera translation and rotation. Neither directly measures object motion in the world.

Keep three outputs separate:

• Motion evidence: pixels whose measured displacement differs significantly from the static-scene prediction.
• Object candidates: spatially connected, depth-consistent groups of motion evidence.
• Tracks: candidates associated across time, with estimated position, velocity and uncertainty.

Motion evidence is not yet an object boundary. A uniformly colored object may provide flow only along its edges, while a textured object may produce several disconnected groups.

Establish calibration and timing first

The required inputs are an image pair, depth associated with each image, camera intrinsics, and relative camera motion between their exposure times. If color supplies the correspondences, depth must be registered into that color image's geometry.

Check whether the depth output represents axial distance along the camera's optical axis or radial range from its optical center. The projection formulas below assume axial depth. Radial measurements require conversion using the calibrated viewing ray.

Also establish the direction of every transform. Here, T_A_from_B maps coordinates expressed in frame B into frame A. A pose labeled only as a camera transform is insufficient unless its direction and reference frame are documented.

Use camera intrinsic parameter conversion when resized or cropped images require updated intrinsics. For acquisition alignment, follow the checks in depth and color synchronization.

A pose timestamp should correspond to exposure, not simply the time a frame reached the application. Interpolate poses using an appropriate rigid-body interpolation method and record the interpolation gap. If inertial measurements contribute to the pose estimate, align events with camera exposure before judging residual motion.

Predict static-scene image motion

For a pixel in the first image, recover its camera-frame position using its depth. Transform that position into the second camera frame, then project it into the second image.

Here, M is the map frame, and fx, fy, cx and cy describe the image geometry used for correspondence. Apply the appropriate intrinsics separately if the image geometries differ. These equations assume rectified or undistorted pinhole images; otherwise use the calibrated distortion model.

Exclude points projected behind the camera or outside the second image. Preserve a validity mask rather than replacing invalid projections with zero displacement.

For pure rotation, static image motion can be predicted without scene depth using the calibrated rotational image transform. Translation is different: nearby static points generally move farther in the image than distant points. A single planar warp cannot describe that parallax across a scene containing multiple depths.

The coordinate conventions in mapping image pixels to 3D points provide the foundation for this step. The additional requirement here is transforming each point between acquisition times before comparing its image position.

Work through a lateral-motion example

Consider a simplified, rectified scene with no camera rotation. The camera translates sideways by 0.05 m between exposures separated by 0.10 s. A feature lies at an axial depth of 2.0 m, and the horizontal focal length is 600 pixels.

For this geometry, static image displacement is:

Suppose the measured feature displacement is instead -9 pixels. Its residual is:

The positive residual indicates lateral object motion in the positive axis direction under these assumptions. Do not apply this shortcut to arbitrary camera rotation, changing depth or general motion. Use transformed endpoint positions for those cases.

Sampling also limits what you can segment. If a scene spans 4,000 mm across an image width of 1,200 pixels, a 10 mm feature spans only 1,200 × 10 / 4,000 = 3 pixels. That leaves little support for reliable correspondence near a depth boundary.

Use the resolution calculator to check feature sampling before choosing a camera. Sampling alone does not establish motion accuracy.

Decide which residual representation to use

Image-space residuals are a useful first gate, but their physical meaning depends on depth and viewing direction. A fixed pixel threshold does not represent the same displacement throughout the scene.

For full endpoint displacement, take the measured correspondence in the second image and sample valid depth there. Back-project both endpoints, then express them in the same map frame.

The endpoints must represent the same physical surface feature. A correspondence that jumps from a foreground object to the background creates a convincing but false displacement.

Avoid creating a 3D vector from image flow and one depth value alone without stating the missing assumptions. That construction cannot generally recover changing distance along the viewing direction.

Reject visibility and pose artifacts before clustering

Several common errors resemble independent motion. Test them before growing connected regions or fitting object tracks.

Occlusion and disocclusion

Forward-projecting the first depth image can place several points at the same destination. Resolve predicted visibility using the nearest valid projected surface, rather than treating all projected points as visible.

Compare the predicted depth with the observed depth at the predicted destination. A substantially closer observed surface can indicate occlusion. Newly exposed surfaces may have no valid first-frame correspondence.

Mark ambiguous points as visibility-uncertain. A visibility conflict may also involve a genuinely moving object, so it should not automatically classify the region as static. Let later observations and object-level evidence resolve the ambiguity.

Depth edges and correspondence errors

Depth interpolation across a foreground boundary can create positions between unrelated surfaces. Reject depth samples whose interpolation neighborhood crosses a large discontinuity, or use an edge-aware sampling method with documented behavior.

Use forward-backward correspondence consistency and image-quality checks. A useful diagnostic experiment is to exclude a 2-pixel band around depth discontinuities, then compare false detections against the unfiltered result. Adjust that band from measured registration error rather than treating it as universal.

If the depth comes from stereo, resolve the issues described in stereo disparity map troubleshooting before tuning motion thresholds.

Pose error and timestamp error

A small pose error often produces coherent residuals over much of the background. Inspect residual direction versus image position and depth. Broad patterns are more suggestive of registration or odometry problems than isolated compact clusters.

For a simplified lateral-motion case, a timing error produces approximately:

A threshold below that error would repeatedly flag static surfaces. Rotation introduces an additional image-dependent contribution that this estimate omits.

Rolling shutter can also invalidate a single-pose-per-frame model during camera motion. Either constrain the operating motion, use an appropriate shutter and exposure arrangement, or model acquisition timing across the image.

Cluster motion evidence into object candidates

Cluster points using spatial proximity, compatible displacement and acquisition confidence. Avoid grouping pixels solely because their flow arrows point in the same direction. Separate objects can move together, while different surfaces of a rotating object can have different velocities.

A practical processing sequence is:

1. Remove invalid depth, low-confidence correspondence and visibility-uncertain samples from the trusted motion set.
2. Apply residual thresholds based on depth, pose uncertainty and correspondence quality.
3. Form local groups using 3D proximity and compatible displacement.
4. Reject groups with insufficient spatial support or poor consistency.
5. Associate surviving groups across successive observations.
6. Refine object boundaries using depth continuity, appearance or instance segmentation where available.

As an initial evaluation setting, require a candidate in 3 consecutive observations and measure the resulting missed detections. At 10 observations per second, confirmation after the first observation takes approximately 2 / 10 = 0.2 s. Persistence reduces isolated noise but adds delay and can miss briefly visible objects.

Do not use those example values as acceptance criteria. Establish minimum detectable object size, speed, range and reporting delay from recorded operating scenes.

If the odometry estimator uses the same images, moving objects can corrupt the camera-motion estimate itself. Use a background-supported robust estimator and exclude established dynamic regions where the estimator supports that workflow. When moving surfaces dominate the view, report reduced pose confidence instead of manufacturing a confident segmentation.

Maintain tracks in a consistent map frame

Transform accepted candidate points into the map frame before computing a representative object position. A robust center of the valid cluster is usually more stable than the center pixel of an image bounding box, which may fall on background.

Store the observation timestamp, frame identifier, position estimate, uncertainty, extent and validity state. Estimate velocity from timestamped observations, not frame count, because dropped frames change the elapsed interval.

Mapping systems may revise poses after loop closure or relocalization. Such a correction can make every tracked object appear to jump. Keep short-term velocity estimates in a continuous local frame, or re-express the track history consistently after a map correction.

Before accepting the pipeline, record these cases:

• A fully static scene with camera translation and rotation.
• A moving object against both near and distant backgrounds.
• Motion mainly toward or away from the camera.
• Partial occlusion, reappearance and crossing objects.
• Missing depth, blurred images and degraded pose confidence.
• Pose resets, delayed frames and map corrections.

Measure false moving regions, position error, track continuity and reporting delay separately. A visually convincing overlay does not establish reliable 3D tracking performance.

Choose hardware and check integration before ordering

Select 3D sensors by the depth behavior and acquisition interfaces required by the pipeline, not simply by the availability of a point cloud. The catalog identifies different sensing approaches, but the listed product information does not establish suitability for continuous moving-camera tracking.

The MV-DPS200P-02 is a projected structured-light 3D camera. Confirm its acquisition behavior, timestamp support and usable depth performance for your scene before selecting it for this workflow.

The MV-DT01WMU is a ToF sensing camera. Apply the same integration checks rather than assuming the sensing method guarantees useful motion measurements.

If a separate image camera supplies correspondence, compare area scan cameras for the required image quality and acquisition timing. An additional camera also introduces extrinsic calibration, synchronization and overlapping-coverage requirements.

Before ordering, confirm:

• Access to calibrated depth, image data and validity information through supported software.
• Documented timestamp meaning and a usable alignment method for camera poses.
• Suitable exposure behavior, shutter characteristics and motion tolerance.
• Depth coverage on the actual surfaces, including dark, reflective and weakly textured regions.
• Sustained acquisition and processing throughput with the required streams enabled.
• Recorded-scene acceptance tests, including the limits where motion classification becomes unreliable.

Use the camera finder to narrow acquisition choices, then start the camera configuration with your field of view and timing requirements. For integration questions, send the required range, object size, camera motion and representative scene conditions through applications support.