Test 3D camera code without hardware: record and replay fixtures

You are mid-way through writing the inspection logic for a 3D camera, and the camera sits on the production line, in another plant, or still in transit. The viewer that comes with the SDK shows a simulated sensor, but that demo device almost never accepts connections from your own code, so it cannot stand in for the camera in your tests. You can still develop and regression-test the whole software stack without hardware: record a capture set from a real camera once, replay it through a small fake device that speaks the same call sequence, and reserve the live camera for the short list of things replay cannot reach. This guide shows what to record, how much disk the set needs, how to write the replay fixture, and where simulated testing ends.

The viewer's simulated sensor is a demo, not a test bench

Most 3D camera tool suites include a viewer with a built-in simulated device so you can explore the interface before hardware arrives. Two things stop it from being a test bench. First, API access is rarely exposed: device discovery enumerates real cameras on the network, and the simulated device lives inside the viewer process, so your connect call sees nothing. Second, where a documented simulation mode does exist, it serves a generic scene — not your parts, your fixtures, your rejects or your lighting — so thresholds tuned against it prove nothing about the cell.

The workable split: use the viewer's demo to learn the call order (discover, connect, set parameters, capture, read results), then build your own replay layer from real captures. Controls engineers run the same play when they validate logic in a PLC test environment built on simulation instead of wired hardware: the interface gets exercised, and the content comes from your own plant.

For the wider menu of emulation options across 2D and 3D SDKs, see developing machine vision without a camera. The rest of this guide builds the 3D record-and-replay pattern end to end.

Record a capture set on a real camera once

An hour of access to the cell camera at shift change is enough to record the whole set. Record on the same sensor family the station runs: if it is a 3D laser profiler such as the MV-DP4180-03P, save the profiler's own output bundle, not a converted image, so replay feeds your pipeline exactly what the SDK delivers in production.

Save each capture losslessly, in the units the SDK delivers:

• the depth or range map, raw float32 or uint16 exactly as delivered;
• the intensity or reflectance image, as 16-bit PNG;
• the confidence or validity map, as 16-bit PNG;
• the calibration the SDK attaches: intrinsics, extrinsics and any mount transform;
• a parameter snapshot: exposure, filter and ROI settings as set on the day;
• metadata in a JSON manifest: timestamp, trigger source, measured capture duration, part revision, pose label and a checksum for every file.

The pattern for saving camera frames to disk for offline testing carries over directly; the 3D additions are the confidence map and the calibration block.

Plan the matrix, not a pile: part revisions × poses the station actually sees × surface states, plus one capture per known defect class. A single station lands around 60 to 200 captures. Choose the recording resolution the inspection needs: resolving a 0.5 mm feature across a 300 mm field takes 300 / 0.5 = 600 pixels across, and a 2 to 3× margin puts you at 1,200 to 1,800 pixels, so a 2 MP class camera covers the set.

Lateral pixels are half the story; z has its own resolution behaviour, covered in the 3D point cloud x, y and z resolution settings guide. Area-style sensors record the same way: the MV-DPS200P-02 returns depth, intensity and confidence from each projected-pattern capture, so a single file bundle per scene is enough.

Size the capture set: the disk arithmetic

Storage never decides the matrix, but check it before you record. The arithmetic is short:

Worked example — substitute the frame size your camera delivers; the method is identical. Take a 2,048 × 2,048 capture with float32 depth, 8-bit intensity and 16-bit confidence:

• pixels per frame: 2,048 × 2,048 = 4,194,304;
• depth: 4,194,304 px × 4 B = 16,777,216 B = 16 MiB;
• intensity: 4,194,304 px × 1 B = 4 MiB;
• confidence: 4,194,304 px × 2 B = 8 MiB;
• total per capture: 16 + 4 + 8 = 28 MiB;
• a 60-capture matrix: 60 × 28 MiB = 1,680 MiB ≈ 1.6 GiB.

Round up: a 200-capture set at the same frame size is 200 × 28 MiB = 5,600 MiB ≈ 5.5 GiB, and even 500 captures stay under 14 GiB. Every laptop and CI runner handles that, which is the point: scene variety decides the set's value, and every pose, finish and defect you record today is a regression test you never re-stage.

Two habits pay off later. Note depth's native units in the manifest, because millimetres versus metres is the classic silent bug in replayed pipelines. And regenerate point clouds from stored depth plus calibration instead of storing clouds separately; the depth map to real distance calibration guide shows the transform you re-apply at replay.

Write the fake device that replays it

List what your application actually asks of the camera — discover, connect, read and write parameters, trigger or free-run, receive a frame bundle, disconnect, handle errors — and implement exactly that surface in a small class backed by the capture folder. Keep the names your SDK already uses; the names below are illustrative.

class FrameBundle:
    def __init__(self, depth, intensity, confidence,
                 timestamp_s: float, capture_ms: int) -> None:
        self.depth = depth              # 2D float32 array, SDK's native units
        self.intensity = intensity      # 2D uint8 array
        self.confidence = confidence    # 2D uint16 validity map
        self.timestamp_s = timestamp_s  # recorded capture time
        self.capture_ms = capture_ms    # recorded capture duration

class FakeCamera3D:
    def __init__(self, capture_dir: str) -> None:
        self.capture_dir = capture_dir
        self.manifest = read_manifest(capture_dir)  # records from the JSON manifest
        self.index = 0                              # next capture to serve
        self.connected = False

    def connect(self) -> None:
        self.connected = True

    def capture(self) -> FrameBundle:
        record = self.manifest[self.index]
        self.index = (self.index + 1) % len(self.manifest)
        time.sleep(record.capture_ms / 1000.0)      # honest pacing
        return load_bundle(self.capture_dir, record)

Put this behind a thin adapter so the application only ever sees FrameBundle: production wires the adapter to the real SDK calls, tests wire it to FakeCamera3D, and no inspection logic changes. That swap is the whole point, and it is the camera-side twin of testing PLC code in a simulation mode with an automated test manager instead of a powered panel.

Three behaviours separate a useful fake from a stub. Serve bytes that match the manifest checksums, so tests fail loudly if a transcode sneaks in. Accept parameter writes that match the recorded snapshot and raise the same error type your SDK uses for out-of-range values, so parameter handling gets exercised too. Serve captures in a fixed, labelled order, so a failing test names the exact scene that failed.

Prove the fixture with a test rather than by eyeballing output:

def test_replay_serves_recorded_bytes() -> None:
    cam = FakeCamera3D("fixtures/bracket_rev_b")
    cam.connect()
    frame = cam.capture()
    assert frame.depth.shape == (2048, 2048)
    assert sha256_of(frame.depth) == cam.manifest[0].depth_sha256

If your pipeline consumes point clouds, regenerate them from replayed depth and calibration inside the fixture, exactly as production does.

Make replay timing honest

The sleep line is not cosmetic. A structured-light capture commonly runs a few hundred milliseconds; 320 ms is a typical measured value, so store the real one in the manifest. Without the delay, replay serves frames far faster than hardware ever will, and timing bugs stay hidden: code that starts conveyor motion before frame-ready, or samples a part-present signal at the wrong moment, passes fast replay and fails on the line.

The arithmetic for the 60-capture suite: reading 28 MiB from an SSD at about 500 MB/s takes 29.4 MB / 500 MB/s ≈ 59 ms per capture, so raw replay finishes 60 captures in roughly 3.5 s. With the recorded 320 ms restored, a full pass takes 60 × (320 + 59) ms ≈ 23 s — fast enough for every commit, slow enough to be true. Expose the delay as a fixture option: honest pacing for timing tests, fast pacing for logic tests.

Capture timing has its own rules on the real device; the trigger call versus exposure time guide covers how captures are paced, and mapping camera timestamps to PC time keeps recorded timestamps comparable to host time.

What replay cannot test: keep this for the live camera

Replay exercises code; it cannot exercise physics. Keep a short live session — 30 to 60 minutes per release is usually enough — for this list:

• trigger wiring and I/O timing against the actual PLC or sensor;
• network behaviour under real load, when several cameras share one switch;
• calibration and temperature drift, which move the depth values your thresholds sit on;
• how new finishes, coatings or lighting changes actually read on the sensor;
• robot-cell alignment, re-checked with the 3D camera robot integration checks.

Run the replay suite first, fix everything it catches, then spend the live hour on this list. Walking into the cell with replay-clean code turns the session from debugging into verification, and one borrowed hour confirms a release instead of opening a week of trial and error. More capture-side guides are collected on the 3D vision topic page.

Run the suite on a bench controller

Replay runs on any laptop, but a dedicated bench machine changes the habit: tests run after every push and around the clock, on hardware that mirrors the cell's network layout. A compact VC2000-series controller such as the MV-VC2140-128G40-1T fills that role — keep the capture archive on its hard drive and the working set on its SSD, and wire it to the same switch topology the cell uses so network-related tests stay representative.

When the bench later hosts the live-hour checks, the same machine drives the camera: match the cell's sensor class in the camera finder, or let the configurator pair controller, cameras and lights in one pass.

Checks before you trust the fixture

• The checksum test passes on every capture in the set, proving replay serves recorded bytes untouched.
• One test asserts each seeded defect capture trips the inspection, so the set is a pass/fail oracle and not just data.
• A replay result has matched a live capture of the same scene at least once, before the borrowed camera went back.
• Every manifest names the SDK and firmware version that recorded it, so replay results stay attributable when either changes.
• Depth units are stated in the manifest and asserted in one test.
• The suite runs green in CI with no device attached, which is the acceptance test for the fixture itself.

That last item is the habit that pays: if a commit can only be validated at the cell, the fixture is not finished.

When the usual answer is wrong, and the decision

Three cases change the plan. Motion-coupled applications — capture triggered by conveyor position or robot pose — cannot validate the loop on replay; record captures, then test the loop live with a scripted, safe motion profile. A camera still in procurement means synthetic scenes (planes, CAD renders sampled into depth maps) can bootstrap parsing code, but no inspection threshold gets committed until real captures exist. A documented, API-facing simulation mode in your SDK is worth using for interface work — and still record real captures, because a generic scene cannot tune thresholds.

The decision per case: camera on the cell, code on your desk — record an hour, replay everywhere, verify live for 30 to 60 minutes per release. Camera shared or absent — synthetic scaffolding now, thresholds from real captures later. Motion-coupled — replay the logic, test the loop live. Every case ends the same way: nothing ships on viewer-demo data, and no commit needs the cell to prove the code still works.

To spec the bench, start from the 3D sensors category and the vision controllers category, or contact the applications team with the cell's sensor class and target cycle rate.