Multi-camera 3D processing performance: plan throughput

A multi-camera rig can deliver every image on time and still produce stale tracking or reconstruction results. Capture rate, processing throughput and result latency describe different limits. Treating them as one frame-rate figure hides the bottleneck.

This guide is for planning a system that combines several views into tracking results, depth maps or a changing scene model. You will build separate transport and compute budgets, then test them together. The worked figures are hypothetical planning examples, not benchmarks or performance claims for the catalog parts shown below.

Define the output before specifying the frame rate

Start with the result your application needs, not the maximum rate shown in a camera specification. A system might capture images at 30 frames/s, update tracking at 30 results/s and integrate a scene model at 10 updates/s. Those rates can coexist if the processing pipeline supports them.

Write an acceptance statement covering:

• The number of camera units and independently delivered sensor streams.
• Image dimensions, pixel format and capture rate for every stream.
• Required tracking, depth and reconstruction update rates.
• Maximum result age, measured from a defined exposure timestamp.
• Allowed missing views, dropped sets and recovery time after an interruption.

Also define what “reconstruction” means. Updating a persistent scene model is different from exporting a complete point cloud or surface mesh after every exposure. Exporting can add substantial memory traffic and processing work.

For example, require synchronized input sets at 30 sets/s, tracking results within 60 ms, and scene integration at 10 updates/s. Specify whether that latency limit applies to every accepted result or a stated percentile.

For the broader capture architecture, use the multi-camera capture design guide. The additional decision here is how much processing each captured set requires.

Count sensor streams and intermediate data

A camera unit is not always a single image stream. A stereo unit may supply two sensor images, plus optional color, depth or confidence outputs. Count the actual streams delivered in your selected operating mode.

For uncompressed image payload, use:

This is a payload estimate. It excludes transport overhead, padding, metadata, retransmissions and any additional copies inside the host.

Consider four hypothetical stereo units, each delivering two monochrome images. Each image contains 1920 × 1080 pixels, with one byte per pixel, at 30 frames/s:

These figures use decimal units: 1 MB equals 1,000,000 bytes. If the delivered format instead occupies two bytes per pixel, the input payload doubles to approximately 995 MB/s. A nominal sensor bit depth does not tell you whether the delivered data is packed or padded.

Intermediate outputs can exceed the image input. Suppose each stereo unit produces a full-resolution depth map using four bytes per pixel:

Materializing three floating-point coordinates per pixel uses 12 bytes per point before color, confidence or alignment padding. At the same dimensions and rate, those coordinate arrays alone represent approximately 2.99 GB/s.

Not every pipeline transfers or stores all these arrays. A device-resident pipeline may avoid some host transfers. Conversely, repeated copies can move the same payload several times. Use the four-camera host bandwidth sizing guide to examine transport sharing separately from this processing budget.

Calculate compute capacity independently

The capture interval sets the available average processing budget for a stable pipeline:

If a serialized processing resource needs more than 33.3 ms of work per incoming set, its queue grows unless you reduce work or discard inputs.

Consider this hypothetical workload. The depth and tracking figures are per stereo unit; fusion and mapping operate on the combined set.

Under this deliberately serialized assumption:

That pipeline cannot sustain 30 complete results/s, even if capture and transport work perfectly.

Concurrency can change the result, but an asynchronous function call does not prove that device work overlaps. Different stages may compete for the same execution units or memory bandwidth. Measure actual completion times and steady-state throughput rather than adding submission times.

Reducing mapping frequency alone may also be insufficient. Running the assumed mapping stage every third input set changes its average contribution to 8 / 3 ≈ 2.7 ms. Total average work becomes 24 + 16 + 5 + 2.7 = 47.7 ms per set, still above the budget.

These estimates identify questions for measurement; they do not establish a host requirement. For stage-level diagnosis, use the GPU processing bottleneck guide.

Separate synchronization from processing speed

Synchronization determines whether views describe sufficiently similar moments. It does not increase the host's ability to calculate depth, track objects or update a model.

A shared trigger can align exposure events where the cameras support the required mode. Exposure duration, shutter behavior and trigger latency still matter. Rolling-shutter images expose different rows at different times, so matching a frame timestamp does not make every pixel simultaneous.

Use the shared-trigger exposure timing guide when designing capture timing. Verify the documented trigger and timestamp capabilities of the actual cameras before planning the wiring.

Processing completion order introduces another problem. One view can finish later than another even when their exposures align. Assemble sets using frame identifiers and documented timestamps, not whichever image happens to arrive next. The timestamp-based frame matching guide addresses that boundary.

Define a timeout for incomplete sets. On expiry, either reject the set or publish a clearly marked partial result if the application permits it. Do not silently combine a fresh image with an old view and report the result as synchronized.

Benchmark one stage at a time

Use the same input mode and documented host configuration throughout the baseline tests. Record software versions, pixel formats, exposure settings and processing options so that a later change has a traceable cause.

1. Capture one camera unit without processing. Check delivered frame counts, incomplete images and timestamp intervals. Confirm that exposure duration permits the requested capture rate.
2. Add camera units one at a time. Keep processing disabled. Watch transport errors, receiver buffers and whether devices share an upstream connection.
3. Enable depth calculation only. Measure input sets accepted, outputs completed and queue age. Do not include display rendering unless display performance is part of the requirement.
4. Add tracking and cross-view fusion. Check identity continuity as well as throughput. A faster result is not useful if missing or mismatched views break tracking.
5. Enable scene integration. Track memory use as the represented scene grows. A short test of an empty scene can miss costs that emerge later.
6. Add required export, recording and visualization. Measure each addition independently, then run the full intended workload together.

A useful starting protocol is a 60-second warm-up followed by a 10-minute measurement window. Extend testing to cover representative scene growth and the intended duty cycle. These durations are planning choices, not proof of long-term stability.

Log at least input counts, complete-set counts, output counts, queue depth, result age and errors. Report median, 95th-percentile and maximum latency with the test duration and sample count.

Live acquisition and recorded-input tests answer different questions. Recorded inputs help compare processing settings consistently. They do not reproduce every live transport, exposure or synchronization fault.

Select parts against the measured bottleneck

Start with area scan cameras and the camera finder when selecting image sources. Filter for the acquisition requirements first, then verify the processing software's supported interfaces and formats.

The LS-CMR-0053 is a GigE area scan camera, while the LS-CMR-0166 uses USB3. Both are listed with rolling shutters.

All 2 share: Sensor size 1/1.7". Values as printed in each product's data sheet; open a part for its drawing and full specifications.

These are acquisition examples, not integrated stereo reconstruction systems. Their catalog entries do not establish matched-pair synchronization, depth accuracy or compatibility with a particular tracking package. Verify those requirements separately. Do not transfer the hypothetical workload figures above into a performance promise for either camera.

Consider vision controllers only after identifying the required processing runtime, acceleration support, camera interfaces and sustained workload. Port availability is necessary, but it does not establish reconstruction throughput.

The MV-VC5525-256G70-NN is listed with PoE camera ports and built-in light outputs.

Its catalog entry alone does not establish accelerated depth or tracking performance. Request confirmation of software compatibility and benchmark the intended workload before committing to a processing platform.

Select camera cables through the accessory finder after confirming interface, connector retention, routing and motion requirements. A suitable connector does not by itself establish that the complete connection supports your operating mode.

The LS-CBL-0199 is listed as a high-flex GigE camera cable with screw locks. The LS-CBL-0219 is listed as a high-flex USB3.0 camera cable.

Check camera-side mating details, host-side connectors and the documented cable requirements before ordering. Neither cable selection nor extra host ports compensates for an overloaded processing stage.

Tune the variable that limits useful output

Once the baseline identifies the bottleneck, change one setting and repeat the same test. Keep an accuracy metric beside the speed metric.

Source-side cropping may reduce delivered data, but it can also change stereo overlap and calibration requirements. Check the selected mode's behavior rather than assuming any crop preserves the geometry. The source-side ROI guide explains why cropping after transfer does not reduce camera transport load.

The usual advice to lower resolution is wrong when fine spatial detail is the limiting requirement. In that case, reducing an optional export rate or unnecessary memory copies may be a better first experiment.

Similarly, dropping queued sets can improve freshness without increasing processing capacity. It may be unsuitable when tracking requires a particular temporal cadence or every event must be inspected.

Check latency and ordering before deployment

Throughput can appear correct while the system works through old inputs. At 30 sets/s, three waiting sets represent roughly 3 × 33.3 = 100 ms of queue delay before processing begins.

Measure result age from a documented capture timestamp to a defined publication event. If camera and host clocks differ, establish their relationship before subtracting timestamps. Distinguish exposure start, exposure midpoint and frame-delivery time in the test report.

Before ordering the complete system, check:

• Stream accounting: every sensor image and enabled auxiliary output appears in the budget.
• Compatibility: acquisition and processing support the selected interface, format and operating system.
• Timing: exposure alignment and set matching meet the application's motion requirements.
• Queue policy: backlog is bounded, and missing inputs have a defined outcome.
• Memory behavior: usage remains acceptable as scene size and operating time increase.
• Accuracy: faster settings still meet tracking and reconstruction requirements.
• Full workload: recording, display and export are included when they are required in operation.

Use documented host power and performance settings appropriate to the hardware. Repeat the acceptance test in the intended operating configuration; a brief peak result is not a sustained capacity figure.

Turn the budget into an acceptance test

Write the final test as observable requirements: accepted input sets per second, completed output rates, latency limits, allowed losses and required accuracy. Run representative motion and scene complexity, not only a static target.

A successful result has bounded queues, explainable losses and repeatable output quality. If throughput meets the target but result age keeps increasing, the system has not passed. If a speed improvement damages the required geometry or tracking continuity, it has not solved the application either.

Start the camera configuration process with the acquisition requirements, then bring the transport calculation and staged test results to applications support. That evidence makes the next hardware or processing change a specific decision rather than a frame-rate guess.