Real-time camera feed for robot control: daemon, not nodes

You want the robot to follow the camera, frame by frame, and the tempting first move is a program node on the pendant that reads the camera and moves the arm. That path dead-ends. A node is a saved recipe: its script exists when the program is saved, and its settings store holds the values you typed, not measurements arriving 30 times a second. The architecture that works puts the camera work in a daemon on the vision controller and keeps only a thin read loop on the robot. This how-to shows where each piece lives, how the pieces talk over XML-RPC or a TCP socket, how to size the loop against a 33 ms frame period, and how to verify the robot tracks every frame before you trust it in production.

Why a program node cannot carry the feed

Three properties of program nodes decide the whole design.

The script a node contributes is generated once, at save or parse time. Whatever target value existed then is baked into the motion that follows, and there is no second channel that injects fresh commands into a running program from the pendant side. Live variation has to come from values the script fetches while it runs.

The node's settings store — the data model saved alongside the program — holds configuration across saves, loads and copies. It is a file, not a pipe. Writing per-frame values into it does nothing to the running script, and per-frame work near the pendant's graphical thread stalls the interface, which is why the pendant-side plugin classes should stay away from live data entirely.

What a node is good for here is narrow and useful: launching the daemon process, holding its address and port as saved settings, and emitting the loop skeleton that runs on the controller. The loop never changes; the values it fetches do.

The architecture: daemon, latest-value slot, read loop

Keep the three parts separate and the system stays predictable.

The feed. A depth camera streams into the vision controller; pick it from the 3D sensors category or with the camera finder. Enable only the stream the loop consumes, because extra streams cut the delivered frame rate before your code runs — a low delivered rate on software trigger is the usual first symptom, as in 3D camera frame rate low on software trigger.

The daemon. One process subscribes to the frames, computes the target pose in the robot's base frame and publishes it. Publish means overwrite: a single latest-value slot holding a frame id, a capture timestamp and the pose. Nothing queues, so a slow consumer skips frames instead of chasing a growing backlog of old targets — the exact failure behind vision result stale every cycle.

The read loop. The robot program runs a plain loop: ask the daemon for the latest target and move only when the frame id has advanced. A variant puts a sync thread on the controller that copies the value into a script variable the main loop reads; same behaviour, one extra copy.

Where the daemon lives matters too. It needs a port for the camera, headroom for your processing and as little competition as possible. When several depth cameras share one host, test one stream at a time before blaming the daemon.

What you need

• One depth camera on the controller's network: a ToF or structured-light model when the loop must see surfaces and volume, a standard area camera when 2D targets are enough.
• A vision controller from the vision controllers range to host the daemon, with a port for the camera, a port for the robot network or a shared switch, and CPU and memory headroom beyond the camera driver.
• The camera SDK on that host, plus an XML-RPC server library in whatever language the daemon speaks.
• Robot-side script access: a program node or a script file that runs the read loop.

The host should have room to grow rather than sit at its limit. The controller below pairs a 12th-generation Intel Core i5 with 32 GB of memory, 5+ LAN ports, PoE camera ports and 8 built-in light channels.

A single-camera cell runs on less: an Intel Core i3 or Intel N-series machine with 8 GB of memory hosts one camera and one daemon without strain.

On the feed side, a ToF camera such as the one below supplies the depth stream the daemon subscribes to.

Once the camera count is fixed, size the host against the ports and headroom above rather than the other way round.

Build the daemon in five steps

1. Subscribe to the camera with its SDK and enable only the stream your loop consumes. Confirm the delivered rate, not the headline rate, by counting frames over ten seconds.
2. Compute the target once per frame: locate the part, transform the result into the robot's base frame with your hand-eye calibration and stamp it.
3. Publish a latest-value structure with at least three fields: an integer frame id that increments per frame, a capture timestamp in milliseconds, and the pose as x, y, z plus the rotation convention your controller expects. Swap the whole structure atomically so a reader never sees a half-written pose.
4. Expose one read function, say get_latest_target(), over XML-RPC bound to the localhost interface on a fixed port, plus a health() function returning daemon uptime and last frame age. Per-frame values stay on this side of the boundary; the pendant's graphical process never touches them.
5. Run the daemon as a service that starts at boot, and prove it answers before any robot depends on it: call health() by hand from the controller host. A daemon that only starts when a program node launches it will eventually fail to start mid-shift, and the node will report a failure nobody can debug under cycle pressure.

Write the read loop on the robot

1. Emit the loop from a program node if you use one: start tracking, run the loop, end tracking on exit. Inspect the emitted script on the pendant to confirm it is the static skeleton you expect, with no camera values inside it.
2. Each pass, call get_latest_target() with a timeout set well under the frame period. When the loop also needs the arm's current state, the script language exposes it directly — get_actual_joint_positions() is the usual name for the measured joint angles.
3. Compare frame ids. If the id advanced, compute the next small move toward the new target and issue it; if not, skip the pass. Small tracking moves are what keep a pass shorter than one frame period.
4. On a timeout, or on two passes with no new frame, leave tracking mode deliberately: hold position or run the planned controlled stop. The loop must never fall into an unbounded wait.

Sizing the read loop for a 30 Hz depth feed

The inputs, with units and where each comes from:

• f_cam = 30 Hz — the depth stream rate of the sensor class this pattern serves; halve the rate and every period below doubles.
• t_read = 2 ms — illustrative localhost XML-RPC round trip; measure yours with a timer wrapped around the call.
• t_decide = 3 ms — illustrative time to compare frame ids and compute the next waypoint.
• t_settle = 25 ms — illustrative time for one short tracking move to finish; the term you control.

The rule in words: the loop tracks every frame when one pass — read, decide, settle — completes before the next frame lands.

The arithmetic:

• Frame period: 1000 / 30 = 33 ms.
• Loop pass: 2 + 3 + 25 = 30 ms.
• Margin: 33 - 30 = 3 ms, so the loop runs at 1000 / 30 = 33.3 Hz, just above the feed.
• Worst-case target age: 33 + 30 = 63 ms.

The result: the robot consumes essentially every frame and never acts on a target older than about 63 ms.

The decision: hold one pass under 33 ms and tracking is one-to-one. A pass between 33 and 67 ms still runs safely — the overwrite slot discards frames and effective tracking halves — but worst-case age climbs toward 100 ms. Above 67 ms, stop stretching the move: drop the feed deliberately to 15 Hz, which re-opens a 67 ms budget, or shorten the tracking move. Measure the pass rather than estimate it; the settle term dominates, and it responds to how you split the motion.

XML-RPC or a raw TCP socket

XML-RPC gets you working fastest: a standard request per update, no framing to invent, and a call that maps straight onto the loop above. A persistent TCP socket trades convenience for control — one connection, push or pull, minimal per-update overhead, and framing plus reconnect logic you own. Both patterns are set out step by step in send camera offsets to a robot over TCP socket messaging; when the socket side goes quiet, fix the send loop that never answers before suspecting the network, and when RPC calls start timing out, work through the XML-RPC timeout causes on camera tools.

Start with XML-RPC at camera rates; move to a socket only when the timing math demands it, not by habit.

Verify before you trust it

1. Log the frame id over 100 passes. Nearly every pass should show a new id; misses cluster once the pass time nears the frame period.
2. Time the passes and compare the mean against the budget from the worked example: 33 ms at 30 Hz, 67 ms at 15 Hz.
3. Kill the daemon mid-run, once, on purpose. The robot must leave tracking within the timeout and hold or stop cleanly. This is the test that matters most.
4. After any change, restart from a known state; leftover settings from an earlier run are a classic source of flaky behaviour, as in start from a known state.

Common mistakes

• Live camera data written into a node's settings store: it saves with the program, never reaches the running script, and per-frame writes near the pendant's graphical thread stall the interface.
• Reading with no timeout: the freeze-on-crash failure from the warning above, and the most common one in the field.
• Queuing frames instead of overwriting: the robot chases a backlog of old targets and the lag grows without bound.
• Starting the daemon per program run: the port may still be bound or the daemon not ready; run it as a boot service and health-check it.
• Doing per-frame processing on the pendant side: that work belongs on the daemon host, where the CPU and memory actually are.
• Trusting the headline frame rate: enable a second stream and the delivered rate drops before your code sees a frame; measure what arrives.

When a loop is the wrong tool

Servo-rate reaction. If the robot must respond within a few milliseconds — fast conveyor tracking, surfacing a live part — a 30 Hz program-loop read is the wrong clock. Robot vendors ship external-control script templates that feed a waypoint to every servo tick over a socket. The daemon side is identical; only the robot-side loop is replaced, and the safety case deserves a fresh look rather than a paragraph.

One-shot pick at a fixed station. Skip the loop: snapshot, locate, send one correction. The full architecture is in vision-guided pick and place that works, with the one-shot X, Y and angle correction the robot applies. Streaming and snapshot capture are different tools, and locate works in teach but fails at run time is what mixing them looks like.

Unexplained lag. When tracking still feels late at a healthy pass time, budget the entire feedback chain — capture, transfer, compute, command — as in image based control latency. The camera link is rarely the largest term.

Parts and the next step

The decision in one line: the camera never talks to a program node. A daemon on the vision controller publishes the latest target, and a timed loop on the robot consumes it, held under 33 ms per pass for a 30 Hz feed. If the cell is still on paper, run camera, controller and cabling through the system configurator; the surrounding patterns live on the systems topic page; and when your timing budget lands where the standard pattern does not reach, the contact page is the direct line to the applications team.