Keep a mixed reality camera feed static while the headset moves

You built a green screen rig: a real camera watches the play space, a headset tracks your head, and a game-engine scene composites the camera's image behind the virtual world. In the demo you followed, the background holds perfectly still while you turn your head; in yours, the whole scene slides with every movement. The fault is almost never the camera, the headset or the tracking. It sits in the scene graph and the render settings — which object the feed camera is parented to, and what each pass does with the previous frame's color buffer. Work through the one-minute test below and you will name your case, apply the matching fix, and verify it with a measured head turn before the next take.

Why does a mixed reality camera feed move with the headset instead of staying static?

A real image cannot move on its own: every frame is a fixed bitmap, so the motion you see comes from the virtual camera that samples it — either that camera is inheriting the tracked rig's pose, or the framebuffer it draws into is still holding the previous pass.

The two mechanisms have different on-screen signatures.

• Pose inheritance. The image rotates one-to-one with your head. Turn 90° and the background turns 90° with you, as if the real room were glued to the headset.
• A pass that never clears. Remnants of earlier frames smear behind everything that moves. The feed itself stays put; the leftovers travel.

Both read as "the scene is moving", and both are fixed in the scene, not in hardware. The sample video you compared against looks still because its defaults are correct: the feed camera sits at the world root, and every camera clears its own render target.

Which fault is yours: a parented camera or a render pass that never clears?

Two faults cover almost every report, and a one-minute test separates them: if the feed camera's gizmo moves in the Scene view while only the headset moves, the pose is inherited; if the Scene view stays still while the Game window smears, the clear setting is at fault.

Run the test before touching any setting. Open the Scene view so the feed camera's gizmo is visible, put the headset on, and rotate your head only.

1. The gizmo moves. Something drives the tracker pose into the feed camera — usually a parent link under the tracked rig, sometimes a script or plugin that copies the headset pose onto every camera.
2. The gizmo stays still and the Game window smears. The object is static and the framebuffer is not being emptied. Go straight to the clear setting.

The reported case followed branch two exactly: the demo ran unmodified except for one change — the camera's Clear Flags set to "Depth only" to hide a blue sky color — and the recorded background moved with the head. Switching that clear back restored the still image. Keep both branches in mind; the test costs a minute and prevents a wrong fix.

How do you re-parent the feed camera to the world in one minute?

Select the feed camera, move only the headset, and watch its transform: a parented camera tracks the rig one-to-one, so the fix is to cut that parent link and place the object once, at the calibrated pose of the physical camera in the play space.

1. Select the feed camera in the Hierarchy and read its parent. It must not sit under the tracked head, the rig, or any object those drive.
2. Unparent it to a world-anchored root: the scene root, or an empty object at the origin that nothing moves.
3. Enter its transform once, from calibration: the measured position and rotation of the physical camera relative to the play space. Note the axis convention you used, and keep it consistent with the frame rules described in transforming depth point clouds between the optical frame and the camera frame if your pipeline also carries depth.
4. Enter play mode and turn your head 90°. The feed's edges must hold against the room while the virtual world rotates normally.
5. If a script sets camera poses each frame, point it at the rig object only. The feed camera gets no pose, ever.

Step 3 is a one-shot calibration. Write the values down, because you will re-enter them after any change to the mount.

Which clear setting makes a static feed look like it is moving?

A clear mode that keeps the color buffer — the "Depth only" setting is the usual one — leaves the previous camera's image in place, so every frame the feed draws over a stale view and the leftovers read as motion.

"Depth only" empties the depth buffer and keeps the color buffer. Render order then decides what you see: the feed camera draws the background, the virtual scene draws over it, and the headset's eye cameras draw last. When a pass keeps its color target, the image of the last camera to render is not cleared before it draws again, so the next frame starts with the previous frame behind it and every moving element drags a ghost of where it was.

Restore a full clear on each camera. "Skybox", the demo default, empties color and depth every pass. If you chose "Depth only" because a sky color showed through uncovered areas, fix the coverage: aim or scale the feed until the real image fills the frame, or clear to a solid color you can key out in the edit.

The same leftover-layer mechanism appears outside headsets: a preview that keeps showing an old frame until you clear every layer is covered in why a camera preview overlay will not turn off and how to clear every layer.

How much do capture rate and latency add to the apparent drift?

On a 90 Hz headset the render loop has 1000 / 90 = 11.1 ms per frame, while a 30 fps plate camera delivers a new image every 1000 / 30 = 33.3 ms, so each real frame is held for 90 / 30 = 3 rendered frames.

Those figures assume the common pairing: a 90 Hz headset and a 30 fps capture. The 2.0° is the plate's irreducible lag during a slow 60°/s scan, and a correct compositor hides it — virtual objects take the newest head pose on every 11.1 ms frame, while the real plate stays pinned to the world and is simply replaced every 33.3 ms. The plate needs no pose at all, which is exactly why inheriting one breaks it.

Faster head motion shrinks the margin, not the rule: at 180°/s the same 33.3 ms spans 6.0° (180 × 0.0333), and 60 fps capture halves the interval to 16.7 ms and the lag to 3.0°. Keep the plate's timestamps aligned with the pose stream; when the two clocks drift apart, the seam shifts even with both fixes in place — the mechanism in depth and color timestamps that drift apart while running.

What should you check before you record again?

Rotate your head a measured 90° while watching a fixed mark at the frame edge, then record ten seconds and scrub them: a correct rig holds that mark within about a degree, and anything that slides with your head points back at the hierarchy or the clear.

• Two speeds. Repeat the turn at a slow scan (about 30°/s) and a fast snap (180°/s). Slow exposes leftover pose scaling; fast exposes any pose you still inherit.
• Scene view watch. With tracking live, confirm the feed camera's gizmo holds position on every frame.
• Ten-second take. Record, scrub, and check the mark at the frame edge frame by frame.
• Known state between takes. Restart the scene from its defaults instead of layering changes, the same discipline as starting a flaky camera test from a known state.
• Headset and recording together. The live view and the file must agree, because both are built from the same composited frames.

When is the physical camera mount the fault, not the render settings?

If the feed holds a constant offset instead of sliding — the green screen edge sits a few centimetres off from the first frame, identically on every take — the world pose of the feed camera no longer matches the physical camera, which almost always means the mount moved after calibration.

Turning your head does not change the offset; only recalibration does. Put a fixed reference in the room, align the feed's projection to it, and save the transform from the re-parenting steps. A bumped tripod, a slipped clamp or a re-mounted camera all produce this signature, and it is the same failure that moves a calibrated inspection camera off target when a bracket shifts — static camera calibration accuracy and why moved parts drift off target covers that drift. Two minutes of recalibration fixes what no render setting can.

Which host runs the camera capture, the tracking and the render loop together?

One box has to ingest the camera frames over USB 3 or PoE, run the tracking input, and render two eye views at 90 Hz, so port count, memory and expansion decide whether the rig runs on one machine or needs two.

On the content-creation side any GPU-equipped workstation clears that bar. The engineering version of the same rig — a live machine-vision sensor as the plate camera, overlaying real views onto a simulated cell — runs its capture through a vision controller, and the constraints carry over unchanged.

The MV-VC4712-256G10-2TH provides PoE camera ports, USB 3 ports and PCIe expansion for a render card. The MV-VC5525-256G70-NN adds 32 GB of memory and 8 built-in light channels if the cell also drives inspection lighting. Pick the sensor side from the 3D sensors range, the host from vision controllers, or browse both in the camera finder.

So which fix applies to your case?

Hierarchy moves with the headset: re-parent to the world root; Scene view still but the Game window smears: restore a full clear; constant offset from frame one: recalibrate the mount — then verify each fix with the measured 90° turn before you record again.

Both symptoms can coexist, and an inherited pose partly masks a dirty buffer, so fix the hierarchy first and re-test before touching clear settings.

If the plate camera is a machine-vision sensor rather than a webcam-class device, send its model, the headset refresh rate and the host constraints to expert support, or start from the system configurator. The 3D topic page collects the rest of the depth-camera integration fixes.