Vision server connection drops intermittently: fix XML-RPC failures

A vision link that dies every 10-40 picks is more disruptive than one that never worked: the operator clears the fault dialog, re-initializes the vision server, and the cell runs another five minutes before it stops again. This guide is for anyone whose robot program talks to a camera's vision server over XML-RPC and hits intermittent connection failures mid-run. You will name the cause with one ping test, replace the per-cycle connect-and-close habit with one persistent session, size the call timeout from your cycle budget, and build a recovery path that needs no operator.

Why does the vision connection drop every 10 to 40 cycles?

Almost always because the robot program opens and tears down the XML-RPC connection on every pick, and after some tens of those cycles a socket or the vision server process wedges. The fault is in the connection pattern, not in the camera, the lens or the robot.

XML-RPC encodes each call in XML and carries it over HTTP. Unless your code keeps one session open, a typical RPC client opens a fresh TCP connection for every call and closes it afterwards. A pick program that does this creates and destroys hundreds of connections per hour on a small embedded server, and the server eventually objects.

It objects in one of two ways. A client that times out and walks away leaves its request half-open, and the server holds that socket until its own timeout reaps it; a small worker pool fills with these strays and starts refusing new connections. Or the server leaks a resource per connect cycle. Either way the signature matches: pings still answer, every call fails at connect, and the documented re-initialization step clears the fault until the wedge rebuilds.

The client side rarely runs out first. One fresh connection per 12-second cycle leaves about five sockets in TIME_WAIT on a typical Linux stack, which holds them 60 s, against roughly 28,000 ports in its default ephemeral range. Port exhaustion cannot explain a fault every 25 cycles; the wedge is on the server side, which is where the fix below aims.

You see the same law one level down in the camera API itself: repeated camera open and close cycles leak handles, and the cure there is the same — one session for the whole run.

What is one fault costing per shift?

At a 12-second cycle, a mean of 25 picks between faults puts one stop every five minutes on the cell, which adds up to 96 stops and about 192 minutes of lost production across an eight-hour shift.

The vision call is one station in the pick cycle; the 2D vision guided pick-and-place cycle walkthrough shows where trigger, exposure, grip and handoff sit around it.

Worked example, with the numbers from a typical wrist-camera cell: a mean of 25 picks between faults — the midpoint of the 10-40 spread — a 12-second cycle, and a 2-minute recovery per fault to clear the dialog, re-initialize the server and restart the program.

Interval: 25 × 12 s = 300 s = 5 min. Stops per shift: 480 min / 5 min = 96. Downtime: 96 × 2 min = 192 min, which is 192 / 480 = 40 percent of the shift. These are example inputs; substitute your own interval and cycle time and the formula does the rest. With the pattern fixed, the same arithmetic returns zero stops, and that difference is the business case for the next two sections.

Which cause do I test first?

Test the connection pattern first, then let one ping test split the rest: pings that keep flowing while the calls fail convict the server side, and pings that drop in time with arm motion convict the cable, the connector or the switch port.

Work down the list; each test costs minutes:

1. The per-cycle connection pattern. Read the pick block in the robot program. If every cycle opens a connection and closes it, you have found the mechanism; the next two sections are the fix.
2. A hung server. While the fault is live, ping the camera and fire one call. The ping answers and the call times out at connect: the server or its socket handling is stuck.
3. Cable, connector or switch port. Run a continuous ping through 20 full cycles. Timeout lines stamped where the arm moves implicate the physical path; clean pings send you back to item 2.
4. Port power saving. Energy-saving switch ports power down a quiet link mid-run. Lock the port speed, disable per-port energy saving, run 100 cycles and compare the port's error counters.
5. Address or access changes. Check what changed on the camera side last: a password or access control newly enabled on the service fails every connect outright, and an address that moved after a DHCP renewal does the same — the IP checks in the section below sort it out in minutes.

How do I keep one connection alive across every cycle?

Open the connection once at program start, keep the session for the whole run, and make one call per cycle: with 12 seconds of cycle and 10 seconds of motion plus grip, the vision window is 12 - 10 = 2 seconds, and that is the call timeout.

A persistent session takes the handshake off the per-cycle path. The win is not milliseconds; it is that the server never sees another connect storm, so nothing wedges. Close the session only at program end or inside the recovery routine below.

Size the timeout from the cycle budget, then check it against measured calls. Log the duration of every vision call: the worst case across 100 cycles plus margin should sit under your timeout. A call that once took 400 ms has no business eating a 2-second budget; if the worst case climbs, the server is telling you before it hangs.

Keep the discipline of one client and one wait. A second thread or a second client instance opening its own session while the first one waits is a classic way to wedge a small server, and every request should have exactly one bounded wait — the one-wait-call rule for frame timeouts applies to RPC replies identically.

What goes into the automatic recovery routine?

On a timed-out call: stop motion, close the session, wait 5 seconds, reconnect and re-handshake, run the documented server re-initialization once, retry the request once, and alarm the operator if the retry fails — the whole path runs with nobody at the cell.

1. Catch the timeout. Treat it as a fault, not as a prompt to re-send immediately.
2. Stop motion and hold the gripper where it is.
3. Close the session cleanly so the server can release the socket.
4. Wait 5 seconds, long enough for the server's own timeouts to reap half-open requests.
5. Reconnect and re-handshake. If the server still refuses, run the documented re-initialization once — the step the operator does by hand today.
6. Retry the vision request once.
7. Fail with an alarm and a timestamped log line if that retry fails. Never loop.

Prove the routine before trusting it. Start a run, pull the network plug at the camera for 10 seconds, and watch the cell reconnect, re-handshake and finish the pick with nobody at the pendant. Then read the shift log: zero manual re-initializations is the target.

If your cell also loses the vision job after a robot fault, add the re-handshake at cycle start routine. For the server-side twin of this fault, a smart camera that dies after days of running, follow the network failure recovery checklist.

Could the cable, connector or switch port be the cause?

On a moving arm, yes: flex fatigue and creeping RJ45 plugs cause real link drops, and one continuous ping through 20 full cycles convicts them — timeout lines that line up with arm motion implicate the cable, clean pings send you back to the server side.

A link drop shows as timed-out pings in time with motion. The usual culprits are flex fatigue in a fixed-install cable and an RJ45 latch that creeps out of the jack. Route with strain relief at the arm, respect the bend radius, and use cable rated for the motion; the routing and fatigue checks for robot camera cables cover the full sweep.

Lock the connector first: the LS-CBL-0117 is a 15 m RJ45 camera data cable with a locking stud, so the plug cannot work loose.

On the flexing run, use cable built for the motion: the LS-CBL-0199 is a 3 m high-flex GigE cable with screw locks, rated for drag-chain use.

The trigger path fails the same way: the LS-CBL-0300 carries camera power and I/O on an M8 8-pin connector over 7 m, drag-chain rated.

Both data cables sit in the camera data cables range, and the accessories step of the configurator pulls the cable and connector set for a moving arm together in one pass.

Could the IP setup be the cause?

Less often, but it mimics the wedge exactly: static addresses on different subnets, a camera address that changed after a DHCP renewal, a duplicate IP, or access control newly enabled on the camera service — each fails at connect, and each has a short test.

Set static addresses on both ends, in one subnet. A client and camera on different subnets fail every connect with a no-route error; a camera that drew a new lease fails the same way until you point the client at the new address. Power-cycle the camera and ping. Then unplug the camera and ping again: if something answers, a duplicate address shares the IP, and the switch's address table names the port.

None of this is unique to vision. Controls teams chase the same intermittent-drop signature on Profinet connections to a PLC, and the diagnosis reads the same in random MODBUS TCP connection drops on a PLC server.

For the link-layer walkthrough in vision terms, the vision camera network connection troubleshooting guide covers subnets, ARP and switch checks end to end.

Should the vision server live on the sensor or on a controller?

Neither choice stops the drops on its own — the pattern causes them — but the hardware decides how much the server can carry: a vision sensor at the arm suits one straight-down pick, while a controller with separate LAN ports isolates the robot link as cameras multiply.

At the arm, a vision sensor keeps cabling short and runs the inspection on the device itself. On the server side, a controller gives the vision service its own compute and its own ports, with room to add cameras and light channels later. Filter the smart cameras range by interface and shutter with the smart camera finder, and browse the vision controllers range when the server side needs its own box.

The two shapes of the decision:

From the gripper side, the MV-SC3050XC-16M-ATN-H is a 5 MP color vision sensor with a global shutter that serves results over Fast Ethernet.

From the server side, the MV-VC3301H-128G66 provides 5+ LAN ports, 4 built-in light channels and 8 GB of memory, with a vision software license included.

How do I keep the fault from coming back?

Run the cell on three counters: manual re-initializations per shift, calls that needed the re-initialization fallback, and reconnects with timestamps; targets are zero, zero, and a log you can read, and the first non-zero week gets investigated before it becomes downtime again.

Build the prevention list into commissioning:

• One persistent session per run, opened at program start, closed only at the end or inside recovery.
• Every vision call bounded by the cycle-budget timeout, with its duration logged.
• The recovery routine proven by a pull-the-plug test before the cell ships.
• Static addresses on one subnet, port speed locked, per-port energy saving off.
• Drag-chain-rated cable and locked connectors on every moving run.
• A weekly read of the fault log: re-initializations, retries, reconnect timestamps.

More field-tested fixes like this sit on the systems topic page. If the fault survives one persistent session, bounded timeouts and this list, send the timestamped log and the pick-program excerpt through the contact page and we will read it with you.