Camera acquisition stalls under load: isolate recovery faults

When a busy inspection line stops receiving images, the first visible alarm may appear downstream, long after acquisition failed. Restarting a robot or consumer cannot repair a camera session that has stopped delivering frames. A blank preview, however, does not prove that acquisition stopped.

This troubleshooting procedure is for shared, multi-camera systems where restarting the entire project interrupts otherwise healthy stations. You will trace the earliest failing boundary, measure whether queues can recover, and identify the smallest vendor-supported recovery scope. For implementation-level callback and buffer issues, use the companion guide to multi-camera acquisition freezes.

1. Identify which boundary stopped advancing

Treat image acquisition and target delivery as separate stages. A downstream consumer can remain faulted because its upstream source is unavailable, even after that consumer restarts successfully.

Start with counters rather than the preview window. Record the last accepted trigger, completed frame, processing result and delivered target. A frozen display may indicate a display-thread problem, an empty queue, incomplete transport or stopped acquisition.

These are diagnostic boundaries, not universal camera features. Use only counters and events documented for the installed camera, transport and application.

The decision is straightforward: if complete-frame counters continue advancing, investigate processing or delivery before resetting acquisition. If exposure continues but complete frames stop, investigate transport and receive ownership before changing the inspection algorithm.

2. Build a timestamped timeline before restarting

Collect evidence from the acquisition owner, processing workers and downstream consumers. Keep the healthy cameras in the same recording so you can distinguish a local failure from a shared resource problem.

For each event, log:

• Camera identifier, session generation and event type.
• Trigger sequence and frame identifier, when available.
• Host receive, enqueue, worker-start and worker-finish times.
• Queue depth, oldest-item age and outstanding buffer count.
• Result publication and acknowledgment identifiers.
• The first exception and subsequent state transitions.

Use a host monotonic clock for durations within one process. Record clock offsets and synchronization uncertainty before comparing timestamps from different devices. A camera timestamp and host timestamp cannot be subtracted meaningfully unless their relationship is known.

Keep event recording bounded. A ring buffer can retain recent history without allowing diagnostic logging itself to consume increasing memory. Preserve that history when the first fault occurs, rather than replacing it with repeated downstream alarms.

A worked event sequence

Consider an illustrative timeline relative to the first recorded fault:

• At 0 ms, a worker fails to release an acquired buffer.
• At 180 ms, the receiver reports no available application buffers.
• At 240 ms, the preview stops updating.
• At 620 ms, the downstream consumer reports unavailable targets.

The consumer alarm is the last symptom, not the initiating fault. This sequence supports investigating buffer ownership, but only if the logs actually establish those events and their order.

Add a session generation whenever acquisition is recreated. Frame identifiers may reset or wrap, so a frame number alone cannot reliably identify an image across recovery.

3. Measure load where work is serialized

Average processor utilization does not establish whether acquisition has enough capacity. A single callback, shared lock or serialized inspection worker can be saturated while other execution resources remain available.

Measure processing duration, callback duration, queue depth and queue age under representative peak load. Include image copying, result construction and communication, not just the inspection algorithm.

For a simple queue, use these estimates:

These estimates assume consistent units and approximately steady rates. They are screening calculations, not guarantees for bursty workloads or workers competing for shared resources.

Suppose 10 cameras each deliver 20 frames per second. The aggregate arrival rate is:

That represents 80% utilization of one continuously available serialized worker, before additional overhead. If mean service time rises to 0.007 s per frame, demand becomes 1.4 seconds of work per second. That worker cannot keep pace indefinitely.

For a separate queue example, suppose arrival is 30 frames per second and completion is 25 frames per second. With 100 available slots:

Increasing buffer capacity delays this failure; it does not remove the sustained deficit. A deeper queue also permits older results to reach downstream equipment unless the application rejects stale work.

Compare quiet periods with dense-product bursts. If target production drives more processing or communication, a constant camera rate can still produce a variable application workload. The guide to vision picking request timing explains why processing and downstream motion need separate timing budgets.

4. Test likely causes with controlled isolation

Change one variable at a time. Use a vendor-supported test mode or an isolated test environment whenever a diagnostic change would alter production behavior.

First: processing backlog or callback blockage

Check whether frame receipt continues while worker completions stop. Inspect outstanding buffer ownership, callback duration and waits on shared resources.

Callbacks should return promptly according to the acquisition library's documented buffer-lifetime rules. If work is deferred, either retain the buffer through a supported ownership mechanism or copy the required image data before returning it. Never keep a pointer after its documented lifetime ends.

Where supported, compare acquisition-only operation with full processing. A stable acquisition-only test narrows the investigation, but does not prove the production application is fault-free: removing processing also changes scheduling and memory pressure.

Second: trigger demand exceeds current readiness

Compare source pulses with accepted triggers and exposure events. Missing accepted triggers can result from readiness limits even when the electrical signal reaches the camera.

Test at a lower controlled trigger rate without changing other settings. Follow camera hardware trigger timing when relating external pulses to exposure and acquisition limits.

If the input signal itself is absent or inconsistent, investigate the documented electrical interface. The trigger common-reference troubleshooting guide addresses that separate failure path.

Third: shared transport or host receive pressure

If exposure events continue but complete frames stop, inspect incomplete frames, transport errors and host receive resources. Correlate failures across cameras sharing a link, adapter or other transport resource.

For network-connected acquisition, use the network congestion diagnostic procedure. Do not assume a transport fault merely because several cameras fail together; a shared application lock can produce the same visible pattern.

Fourth: resources accumulate over time

A failure that appears only after extended operation deserves trend measurements. Record process memory, outstanding buffers, thread count and queue age at consistent intervals.

Repeat the same workload after a clean, documented initialization. A repeatable upward trend is more useful than one high reading. If feature-access activity correlates with growth, consult camera feature-access memory leak troubleshooting.

5. Determine the supported recovery scope

Recovery scope depends on ownership. Establish which component owns the camera handle, stream, receive buffers, processing queues and downstream target session before choosing a restart action.

The multi-camera acquisition session architecture guide explains why independent cameras can still share a failure domain.

Use the following hierarchy as questions for the vendor documentation, not as a universal restart procedure:

Stop product flow and place affected equipment in its approved safe state before recovery that could invalidate inspection or target data. Do not defeat interlocks or force downstream equipment to consume uncertain results.

A supported recovery sequence must account for pending callbacks, outstanding buffers, stale queued results and downstream acknowledgment state. Do not free memory while callbacks can still access it, or clear target records while equipment may still act on them.

After reconnection, some features may require a documented acquisition-state transition before they become writable. Use feature write-protect troubleshooting after reconnection rather than repeatedly retrying writes without understanding their access conditions.

Once acquisition is healthy, validate the separate project start and stop handshake. A correct handshake does not repair a failed acquisition owner, and a repaired owner does not automatically restore downstream coordination.

6. Verify recovery under the original workload

A moving preview is not sufficient acceptance evidence. Verify the entire chain from trigger acceptance through downstream acknowledgment.

1. Confirm the documented acquisition state and read back relevant settings.
2. Check that complete-frame counters advance without unexplained incomplete-frame growth.
3. Verify that processing completions keep pace and oldest-item age remains bounded.
4. Confirm that pre-recovery frames and targets cannot appear as new work.
5. Check that every delivered target belongs to the intended session and consumer.
6. Restore representative peak load and record the same measurements used during diagnosis.

Test longer than the previously observed failure interval, with enough repetition to expose the suspected accumulation or burst condition. Record the actual duration and workload; do not declare success from elapsed time alone.

If queues remain bounded but downstream work is still unavailable, investigate conveyor target availability rather than continuing to reset a healthy camera session.

7. Check hardware fit only after isolating the fault

Replacing a camera is not a remedy for an unbounded queue or incorrect buffer lifetime. Consider hardware changes when measurements identify a genuine interface, acquisition-capacity or compatibility constraint.

Use the area scan cameras category and camera finder to review interface, shutter and optical requirements. Confirm the required acquisition behavior in the exact product documentation; an interface label alone does not establish trigger support or recovery behavior.

For a GigE candidate, review the LS-CMR-0053.

For a USB3 candidate, review the LS-CMR-0166.

Neither candidate should be treated as a direct replacement without checking application compatibility and the inspection requirements. Retain a known-good configuration and an approved rollback plan before changing hardware.

When reviewing vision controllers, ask how camera connections, acquisition ownership and processing resources are allocated. Port count alone does not prove independent recovery or sufficient processing capacity.

Before ordering, confirm software compatibility, connector details, power requirements, lens compatibility and the vendor-supported session recovery procedure. Request clarification wherever the documentation leaves ownership or restart behavior uncertain.

8. When a smaller restart is the wrong answer

A local restart is attractive because it limits interruption, but it is inappropriate when shared state cannot be reconciled safely. Examples include a common acquisition service failure, unknown buffer ownership or downstream equipment holding targets from an invalid session.

Conversely, repeatedly restarting the whole project can conceal a growing queue or resource leak. A restart that clears symptoms without changing the initiating condition is temporary containment, not a verified correction.

Before returning to unattended operation, complete this checklist:

• Preserve the first-fault timeline and workload conditions.
• Identify the earliest counter that stopped advancing.
• Show that processing capacity meets sustained demand.
• Define bounded queues and approved overload handling.
• Document the smallest supported recovery scope.
• Verify that recovery cannot deliver stale images or targets.
• Repeat the peak-load test and retain the results.

If the fault boundary remains uncertain, prepare timestamped logs, configuration details and counter trends for application support. If measurements justify a hardware change, start with the camera configuration step and carry the measured workload into the selection.