3D camera test flaky on leftover settings: start from a known state

A capture test that passes on your desk and times out on the bench machine is not a flaky camera. The camera kept state your test never sees: exposure, trigger mode, ROI, packet size and averaging all survive in the device's own memory, so a manual tuning session, an earlier test run or an aborted teardown changes what the next run meets. When the wait call, a WaitForFrame with a 1000 ms budget in the shape most C++ capture SDKs use, starts failing, raising the number treats the symptom. In ten minutes you can confirm leftover settings as the cause, put a reset into a global test fixture so every run starts from a known good state, and size the timeout from the camera's worst-case exposure plus transfer time.

What the flake looks like

The failure surfaces inside make check, not in your code. The test opens the device, starts acquisition and asks for a frame with a timeout; 1000 ms is the common default. Nothing arrives, the wait returns empty, the assertion trips, the suite goes red. Three details mark it as leftover state rather than a speed problem: the same commit passes on another machine; a manual factory reset fixes it for a while; and it returns after someone tunes the camera by hand, runs a different suite, or a previous run crashes before its teardown. A speed problem fails everywhere and every time. A state problem fails somewhere, sometimes, which is exactly what flakiness means here.

Likely causes, in order of probability

Rank them before you change anything.

1. Settings left behind by a previous session. A manual session raised exposure for a dark target, armed a software trigger with no trigger source, widened the ROI or left frame averaging high. User sets live in the camera and survive power cycles, as Save camera settings across power cycles with a user set explains, so yesterday's experiment becomes the next morning's red build.
2. A timeout that was never sized. A 1000 ms value copied from an SDK example is a hope, not a budget. Diagnose a camera frame acquisition timeout walks the whole timeout path in production.
3. Order and worker coupling. An earlier test mutates a setting and never restores it, or two parallel workers drive one physical camera at once. Flaky-test guidance across test frameworks keeps reaching the same conclusion: more parallel workers, more timeouts.
4. A capture that is genuinely slow. Long exposure or heavy averaging is legitimate for the part you image, and exposure then caps the frame rate. Depth camera frame rate drops: exposure settings cap the FPS quantifies that mechanism.

Tests that separate the causes

Four checks, in this order, take one bench session and no code changes.

• Log state at connect. Read exposure, trigger mode, ROI and packet size immediately after opening the device and print them on one line. Diff a green run against a red one; any value that differs is your leftover.
• Compare clean against dirty. Factory-reset the camera by hand, run the test ten times, then run it again after a manual session that changes exposure. The production twin of this disease, settings that differ from what the code assumes, is the one behind Camera works in setup but grey image at runtime.
• Bisect the suite. Run the failing test alone. Green in isolation points at an earlier test leaking state or at worker collisions; drop to one worker and re-run. If the suite changes exposure deliberately between shots, Multi exposure settings consistency for HDR capture shows how to keep those changes explicit rather than ambient.
• Time every frame. Wrap the wait call in a monotonic clock and log the elapsed time on pass and on failure. A capture that takes 25 ms green and 1100 ms red has told you nearly everything.

The fix: a known state in a global test fixture

Put the reset where it runs before every test automatically, not where a developer remembers it.

1. Provision the bench camera once. Set exposure, trigger, ROI and network the way the test needs them and save them into a named user set in the camera.
2. Reset in global set-up. Register a fixture that runs once before the whole suite. In a C++ harness built on ::testing::Environment, that is a derived class whose set-up method the runner calls first. Let it load the factory default or issue the SDK's reset call, then load your saved user set. Reset camera settings programmatically before capture shows the reset pattern; the fixture simply reuses it.
3. Assert the state. Read exposure, trigger mode and ROI back and compare them with the values you saved, failing with the numbers in the message. Read-back turns a should-be-fine assumption into a check that cannot rot.
4. Start from a stopped interface. Begin each capture test with acquisition stopped; a start without a matching stop leaves the device busy. Single frame acquisition still busy: why repeated starts need a stop covers that failure in detail.
5. Tear down symmetrically. Stop acquisition and close the device in a fixture that runs even when a test fails, so every run starts from the same place.
6. Keep one worker per camera. Two processes driving one device is a state race no reset can fix; give each camera its own lane in CI.

Worked example: budgeting a 1000 ms timeout for a 10 ms exposure

Inputs first, with where each comes from:

• Timeout: 1000 ms, the value the failing test passed to its wait call.
• Exposure 10 ms, transfer plus driver overhead 15 ms, illustrative bench values for a laser profiler on a small ROI at defaults. Read your own pair from the camera's exposure register and your elapsed-time log.
• Jitter factor 3, a rule of thumb rather than a constant. A CI host runs builds and other suites alongside your test, run-to-run spread on a busy machine reaches tens of percent, and 3× absorbs that with room to spare.

The rule in words: the timeout must cover the worst-case sum of exposure, transfer and overhead, multiplied by the jitter factor.

The arithmetic:

• Clean camera: 10 ms + 15 ms = 25 ms of capture, so the budget is 3 × 25 ms = 75 ms. The 1000 ms call holds 1000 ÷ 75 ≈ 13.3 times what the rule asks for.
• Leftover 100 ms exposure from a manual session: 3 × (100 ms + 15 ms) = 345 ms, still inside 1000 ms. The timeout was never the cause; the state was.
• Leftover 500 ms exposure, the kind auto-exposure leaves after aiming at a dark wall: 3 × (500 ms + 15 ms) = 1545 ms. Every frame now misses the budget and the wait fails.
• Break-even: 1000 ms ÷ 3 ≈ 333 ms. Exposure plus transfer above 333 ms breaks a 1000 ms budget even on a perfectly clean camera.

Decision: when 3 × (exposure + transfer) sits well inside your timeout, 75 ms against 1000 ms here, a timeout failure under unchanged code points at state, and the fixture reset is the fix. Raise the timeout only when the product exceeds it, and then to the computed value rounded up, not to a bigger round number.

When raising the timeout is the right call

Sometimes the number truly is too small. Measure ten clean captures after a reset; if the worst takes 400 ms, the budget is 3 × 400 ms = 1200 ms. Set that, or cut exposure instead, because at 400 ms the exposure itself is close to the frame-rate ceiling. Two knobs pull the number down: shorter exposure under more light, for which the machine-vision lights finder is the place to look, and less averaging. Worker count is the other silent multiplier: the same capture takes longer on a loaded host, so drop to one worker per camera before you conclude the camera is slow.

Bench hardware that keeps runs comparable

A reproducible test needs a bench that does not change underneath it: one camera per lane, each on its own network port, with the office network on a separate interface. The camera under test is normally a 3D profiler or a stereo unit, both in the 3D sensors category. The two profilers below are typical of what a bench like this tests.

The harness itself runs on a fanless vision controller. The MV-VC3301H-128G66 carries 8 GB of memory, 4 built-in light channels and 5+ LAN ports and includes a VM6200Pro license, which is the shape of a bench box for several cameras and the runtime that drives them.

Prevention checklist

Tick these off before the next red build surprises you.

Before you merge the test:

• Reset plus user-set load lives in the global fixture, not in individual tests.
• Read-back assertions on exposure, trigger mode and ROI fire at connect.
• The timeout is computed as 3 × (exposure + transfer) from measured numbers, with those numbers in a comment beside the call.
• One worker per physical camera, enforced in the CI configuration.

Before every CI run:

• A provisioning script reapplies the IP address and the saved user set after any reset.
• The per-frame elapsed-time log is kept, so a slow-down shows up as data rather than as an unexplained red build.

If you are fitting out a bench, the camera finder filters the 3D range by type and series, and the build a system configurator pairs a camera with a controller and lighting in one pass. The 3D vision guides collect the rest of the series, and for a stubborn fixture you can contact the team with the log line from the first check.