Robot camera calibration lost after restart: diagnose the shift

You run a picking line with three delta robots and three overhead cameras. While the controllers stay powered, every robot picks on target. Power down for the night, start up in the morning, and two of the three robots pick out of position until someone re-calibrates the camera and the base frame again. Restoring a backup helps sometimes and fails at other times, which makes the fault look random. It is not random: something is living in RAM and dying at the power-down, and a fifteen-minute parked-position test will name the layer — robot frames, the camera-to-base transform, or the camera mount. This guide walks that test, the arithmetic that splits a constant offset from a rotation, and the fix for each outcome, in the same spirit as eye-to-hand calibration: verify the camera-to-base result.

The restart breaks what a running day does not

Calibration does not drift on a schedule keyed to power cycles. If the cell picks on target all day and misses only after the controllers switch off, the camera, the robots and the calibration maths are fine — the data describing them is not loading back. A running controller holds everything in memory: frames taught at the pendant, transforms uploaded by the vision tool, parameters typed during commissioning. A cold start rebuilds the cell from whatever was written to persistent storage. Anything that lived only in RAM returns as a default or a stale value, and the picks land somewhere else.

That is why re-calibration appears to fix it: you re-teach the relationship into the running system. If the save that should follow is skipped, or writes to a layer the start-up sequence never reads, the fix dies at the next power-down. The fault returning even after restoring a tested backup tells you the backup and the live system disagree about where that one piece of data lives.

The parked-position test, step by step

1. Park one robot mid-workspace, in the tool and work object the project actually uses — for example X 450.0, Y 0.0, Z 320.0, Rz 180.0.
2. Write all six values on paper, with the frame name and tool number. The record must live outside the controller; a screenshot stored on it helps nobody. It is the same discipline as setting a repeatable camera focus position during calibration: a number you can re-check beats one you remember.
3. Take the backup you currently trust, and label it.
4. Reproduce the real failure: a full power-down of the controller and the vision equipment, for the same interval the line actually fails after — not a warm software restart.
5. Power up, let the start-up sequence finish, then read the same pose from the same frame, jogging at reduced speed. Record the difference on every axis, orientation included.
6. Run the picking project without re-calibrating anything, and record which robots miss and by how much.

Three outcomes, three different layers:

• The parked pose reads differently by more than repeatability noise: robot-side frame loss.
• The parked pose is identical, and picks miss by the same amount everywhere: the camera-to-base transform is not surviving the boot.
• The parked pose is identical, and the miss is small near one point but grows toward the field edges: a rotation — camera pose or frame orientation.

The arithmetic that separates a lost frame from a moved camera

Say the failing robots pick about 5 mm off after a cold start. The camera images a 400 mm wide field on 1600 pixels, so the scale is 400 / 1600 = 0.25 mm per pixel, and 5 mm is 5 / 0.25 = 20 pixels. Where those 20 px live names the fault:

• The same 20 px at the centre and at every corner is a pure translation: the loaded frame or transform differs from the calibrated one by 5 mm.
• Zero at one point and growing to 20 px at the far edge is a rotation about that point. A 0.5° error on a 300 mm lever moves a point 300 × sin(0.5°) = 2.6 mm.
• The parked pose reading X 452.6 instead of 450.0 with Rz 179.5° instead of 180.0° is the same rotation counted twice: the 2.6 mm and the 0.5° are one frame error seen on two axes.

A delta picker in good condition repeats a taught pose within a few hundredths of a millimetre, so millimetre-level changes after a restart are loaded data, not loosened mechanics — the reasoning behind repeatability testing and Gage R&R camera calibration errors. For the full derivation of turning pixels into robot base coordinates, work through the pixel-to-robot-base transform arithmetic.

If the parked position changed: the frame is not persistent

A frame taught interactively at the pendant exists in RAM until it is written into a module or parameter file that the start-up sequence loads. On a line commissioned under time pressure, the robot that never needed re-aiming keeps its original start-up-loaded frame, and the ones that were re-taught keep nothing. Fix it once, properly:

1. Re-teach the base or user frame.
2. Declare it in the module your start-up loads, so the value is read at every boot rather than typed once.
3. Cold-start three times in a row; the parked pose must read identical each time before you release the cell.

When a later change moves a camera or a robot, the same recovery discipline applies — vision calibration recovery after camera or robot changes covers the order of operations.

If the parked position held: the transform is not loading

The picking project and the vision tool each hold a copy of the camera-to-base transform. Calibration updates the copy in use; a cold start loads the copy on disk. If those are different files — the result uploaded at runtime, saved to a volatile location, or kept on the vision computer instead of the controller — the line runs on yesterday's numbers every morning. Two fixes, in order of preference:

1. After calibrating, write the result to the persistent location the start-up sequence reads, in the format it loads.
2. Or publish the calibration as a static frame the robot loads with its other frames: publishing an eye-to-hand calibration as a static camera transform shows the pattern.

Then check the stored copy offline — using eye-to-hand calibration results to compute robot-base picks gives the transform arithmetic — so you know the saved numbers pick correctly before Monday does.

If the offset grows toward the field edges: the mount

A constant offset is data. An offset that is zero near one point and grows toward the edges is geometry: the camera is not where the calibration found it. Cables pulling tight, a focus ring creeping, a bracket on unbraced extrusion, or a stand bolted to a frame member that shifts as the hall cools through the night — all of these move the camera between the calibration and the cold morning.

Fix a dot grid rigidly to the machine frame in the camera's view and compare a warm image with a cold-start image; the shift of the grid centre settles mount movement in one exposure, independent of the robot. To map how an offset varies across the field, place the same grid at the centre and at each corner — LS-LXR-0029, a dot grid with 0.5 mm dots on a 76 x 76 mm board, and a small stage such as LS-MNT-0220, a manual XY + rotation stage (60 mm, dia. 60 mm base), make those positions repeatable.

For the camera itself, mass and a wide bolted base win: the LS-MNT-0040 machine vision stand gives a 700 mm height on a 400 x 450 mm base, and its slide table carries the alignment instead of a bent bracket. Lock every screw that was used for focusing — lens locking screws: focus, aperture and enclosure clearance — and strain-relieve the cables so they cannot steer the camera. Browse camera stands or start at the stand step of the configurator.

Why the backup only sometimes helps

A backup restores exactly what it contains. If it was captured before the last calibration, it replays stale transform numbers; if it was captured after a re-calibration while the robot frame was still unsaved, it replays half the fix. "Sometimes helps" is two backups with different vintages of one file. Scope matters too: a project backup, a controller system image and the vision tool's own files cover different layers, and the one that matters may sit in none of them.

Build one golden backup and prove it:

1. Fix the layers above first; never archive a fault.
2. Run a full production cycle clean.
3. Back up every layer in the same hour — controller system, picking project, vision tool files, camera settings.
4. Label it with the date and which calibration it contains.
5. Restore it, cold-start, run the parked-position test, then one production hour.

The same rule applies before any intervention, not only after it: vision reader configuration backup before troubleshooting.

Why robots 2 and 3 fail and robot 1 does not

Because robot 1 has what the others lack. When two robots fail and one survives the same power cycle, the difference is in their configuration, not their hardware. Compare, line by line: which module declares each frame, the order of the start-up routine, where each camera-to-base transform is stored, and which vision-tool references each project loads. Copy robot 1's storage pattern onto robots 2 and 3 — same file, same load order, same persistence — and the weekend stops mattering.

Prevention checklist

• Every base and user frame declared in a module the start-up loads — a frame that exists only as a taught value is a fault on a timer.
• The camera-to-base transform written where the cold start reads it, and checked offline once.
• One golden backup per released state, restore-tested, labelled with date and calibration identity.
• A paper parked-pose card at each controller, signed at every cold start; a two-line readback catches the fault before the first pick does.
• The camera on a rigid stand, screws locked, cables strain-relieved, so geometry cannot move between calibrations.
• Three consecutive clean cold starts before the cell is released after any change.

When the restart theory is wrong

If the picks also drift while the line runs, persistence is not your problem. A camera that moves with vibration shows as shifting frames during production — detecting camera vibration with OpenCV frame tracking gives the measurement. If the error appears only when the part is rotated, the orientation in the vision result is the suspect: vision pick error under rotation: diagnose position drift. And if the image itself is wrong after a power cycle — dark, corrupted or at the wrong exposure — chase camera settings before calibration: keep camera bitrate settings after power cycle and GigE Vision image corruption after a camera power cycle cover that path.

The rule to commit to: a fault that needs a power cycle to appear is a loading problem; a fault that appears during running is a physical or settings problem.

Run the parked-position test at the next controlled shutdown, fix the layer it names, and log the readback at every cold start. For the hardware side, start from calibration targets and lens accessories or the accessories finder; if a layer still refuses to survive a cold start, send the controller and picking software versions through the contact page and we will work the persistency chain with you.