Halfway through a hand-eye calibration run, the solver stops and tf, the ROS transform library, reports extrapolation into the past. The message means what it says: the timestamp your code asked for is older than the oldest transform tf still holds, so tf refuses to invent the missing data. The tempting fix — calling the transform lookup with rospy.Time(0) so tf returns the newest value regardless of age — finishes the run and quietly poisons it. This troubleshooting guide walks the failure in order of likelihood, gives you the log lines that isolate each cause, and replaces the workaround with stamp-matched sampling plus a visibility gate. You also get a latency budget with the arithmetic, a skew rule in millimetres, and a prevention list that keeps the error from coming back.
What the error actually says
tf keeps every transform as a rolling history of stamped samples. A lookupTransform names a target frame, a source frame and a time; tf picks the two samples bracketing that instant and interpolates. Ask for a time older than the oldest stored sample and tf answers extrapolation into the past. Ask for one newer than the newest and you get the mirror-image refusal about the future. Both say the same thing: the buffer does not cover the instant you asked about, and tf will not guess.
During calibration the requested instant is the image stamp or a nearby "now", so the error reduces to one question: why does the buffer not cover the moment your image was taken?
One special case first. A static link in the chain — camera to flange, marker plate to table — broadcast once with no usable stamp shows a time of 0.000 in tf_echo. That case is covered in tf_echo time 0.000: why static camera transforms show no timestamp; until it is fixed, lookups naming the image stamp keep failing on that link.
The likely causes, tested in order
| Cause | Typical signature | Test that isolates it |
|---|---|---|
| Lookup arrives late | Fails on some samples only; worse with long exposures or slow detection | Log requested time minus image stamp, per sample |
| Request lands ahead of the data | Fails at start-up or right after each pose change | Log the newest stored stamp for every frame in the chain |
| Two clocks in the chain | Every lookup fails, or errors cluster soon after start-up | Compare an image stamp and a joint-state stamp captured the same moment |
| Unstamped static link | Works with Time(0), never with the image stamp | tf_echo shows 0.000 on that link |
Late lookup. The stamp is right, but your node reaches the lookup after the buffer's history has moved past it. Marker detection, a backlog of callbacks, or a pause in the runtime each eat the window. In a stopped-pose set-up, latency costs accuracy only when it outruns the buffer; in a moving set-up every millisecond costs motion, priced below.
Request ahead of the data. You asked for "now" before the joint states describing that instant arrived. Common at start-up — the first image can beat the first state message — and right after each move. Check the newest stored stamp per frame before sampling, and drop the sample if your requested time runs ahead of it.
Two clocks. The camera stamps images on one clock, the robot controller on another, and a lookup for a time one side has never produced fails every time. Put a stamped image and a stamped joint-state message from the same moment side by side and compare. Timestamp faults upstream surface elsewhere as well; camera recording faster than real time: fix timing errors covers that diagnosis.
The fix: stamp-matched sampling with a visibility gate
- Command the robot to a calibration pose and wait for it to stop: zero commanded velocity plus a short settle, not a fixed sleep.
- Capture the image and read the timestamp from its header. That instant — not the time your code runs — is the lookup time.
- Confirm the buffer covers that instant for every frame from base to camera. If it does not, drop the sample. A dropped sample costs one pose; a substituted "now" costs the calibration.
- Run marker detection and record the sample only when detection succeeds and the corner fit passes the threshold you already use.
- Log three numbers per sample: image stamp, requested time, newest buffered stamp. The skew column is your early warning.
- Spread the poses with real rotation, not just translation; rotation is what pins a hand-eye solution.
If you broadcast the marker pose as a frame for the solver, stamp that broadcast with the image time so downstream lookups stay honest; the pattern is in publish an ArUco marker pose as a TF frame for hand-eye calibration. Consistent inputs matter as much as timing — the pairing discipline in eye in hand calibration inputs: the same robot pose and frame pairs is the other half of a clean sample set.
Why rospy.Time(0) poisons the result
The workaround that circulates for this error is to pass rospy.Time(0) to the transform lookup. That argument tells tf to ignore the instant you asked about and return the newest stored sample, whatever its age. The error disappears — and so does the warning that data was missing.
Think about what "newest" means mid-calibration. Every link between robot base and camera changes at each pose. If the arm swings out of the marker's field of view, the tracking frame stops updating and freezes at its last position. A lookup with rospy.Time(0) hands back that frozen pose without complaint; the solver records a fresh image against a stale robot pose, and the run finishes with a transform that misses by centimetres at runtime. The error was doing its job. Silencing it converts a visible failure into an invisible one.
Weak or stale marker data also produces solver symptoms that look like bugs elsewhere; ArUco Z axis distance error: fix the marker solve first walks that diagnosis.
A latency budget with the arithmetic
Worked example, eye in hand, numbers you can rescale:
- image stamp: 20.480 s, set at capture
- exposure: 8 ms
- sensor readout plus transport to the host: 15 ms
- marker detection: 12 ms
The pose is available 8 + 15 + 12 = 35 ms after the stamp. That is fine while the lookup asks for 20.480 s and the buffer still covers it: tf interpolates the robot pose between the joint-state messages bracketing the stamp.
Two ways to get it wrong, priced at a TCP speed of 250 mm/s:
- Requesting "now" instead of the stamp pairs the image with a pose 35 ms later: 250 mm/s × 0.035 s = 8.75 mm baked into every sample.
- Joint states arriving at 100 Hz leave the stamp up to 5 ms from the nearest message; interpolation error while moving is bounded by 250 mm/s × 0.005 s = 1.25 mm.
The decision follows. Stop the arm at each pose and both terms collapse, because a stationary tool has no motion to smear. If the set-up must capture while moving, budget backwards: skew_budget = acceptable_error / TCP_speed. Accepting 0.5 mm at 250 mm/s allows 0.5 mm ÷ 250 mm/s = 2 ms; slowing the pass to 50 mm/s buys 10 ms for the same error.
And when a runtime pause delays the lookup until the stamp falls behind the buffer's history, tf answers extrapolation into the past — the exact failure the gate in step 3 catches by dropping the sample. Data-efficiency is now a research target in hand-eye calibration itself, with a 2025 journal method built around doing more with fewer samples; the per-sample gate is where that gain is won in your cell.
Hardware that shrinks the timing window
- Global shutter keeps the marker square while the arm settles; a rolling shutter can shear it mid-readout and cost you detection time.
- A bright, even light on the plate shortens exposure and detection together — the 8 ms exposure in the example is what a dim scene forces, not a requirement.
- A hardware trigger from the controller ties the exposure to a commanded instant, so the stamp means what you think it means.
- On a wrist-mounted camera, choose the interface for the cable route first; the timing argument is wasted on a link that drops frames.
A 2.5 MP global shutter GigE camera such as the MV-CU020-90GC covers a marker plate at working distance with a cable run that survives a wrist mount. When the target is small or the plate is far, the MV-CH120-11UC carries 12 MP on a global shutter over USB3. For marker plates and fine fiducials, global shutter options from the area scan cameras range:
- Sensor format
- 1/1.7"
- Lens mount
- C-mount
- Sensor format
- 1.1"
- Lens mount
- C-mount
The same stamp rule governs a 3D sensor on the wrist or over the table: a point cloud is an image with a longer capture, and it pairs with the kinematic chain at the instant it was stamped. Profile sensors and structured-light cameras from the 3D sensors range obey the budget above.
Laser profilers and stereo 3D cameras
170 models · from $699.00The configurator matches a camera, lens, light, light controller and cables to each other, step by step, and lists only parts that fit what you already picked.
Checks before you trust the calibration
- Re-run the same pose set with stamp-matched lookups and compare solver residuals against the Time(0) run. The timed run converges tighter; if it does not, the marker solve is the next suspect.
- Hold poses out of the calibration set, command them afterwards, and measure where the robot lands against the marker's known position — the method in hand-eye calibration accuracy verification: test before you trust it.
- Repeat the whole calibration three times. Timing noise is what scatters the results; three runs agreeing within your repeatability target is the pass condition.
- If the solver only converges after you loosen the marker threshold, the timing is still wrong. Tighten it back and find the delay.
Clean, time-matched samples are also what make the downstream transform trustworthy at runtime; a worked pixel-to-robot coordinate transform walk-through shows the payoff once the data is solid.
When the stopped-pose answer is wrong
- Continuous capture — conveyor, spinning dial — cannot stop. Use the budget: slow the motion, shorten exposure, or both, and gate harder on skew.
- Part of the chain on simulated or controller-local time fails identically, and no gate fixes a clock offset. Align the clocks first.
- If the gate drops every sample, latency outruns the buffer's history. Shorten the path — decode less, detect faster — or give the buffer a longer history when you build it.
Prevention checklist
- Stamp every lookup with the image header time — pairs image and pose at the same instant.
- Stop at each pose and capture on a settle condition — zeroes the motion term.
- Gate on detection success and buffer coverage, and drop rather than substitute — a dropped sample is free.
- Log skew per sample and abort above budget — the visible error is the one that already happened.
- Publish static links with a real stamp — keeps one rule for the whole chain.
- Run the pose set in reduced-speed mode with the cell clear: a calibration script jogging a tool over a marker plate is still robot motion and gets treated like any program.
If the skew column still misbehaves once the gate is in, send the per-sample log and pose set through the contact page and we will read it with you.
Questions
What does the extrapolation into the past error actually mean?
tf holds each transform as a short, stamped history. Your lookup asked for an instant older than the oldest sample still stored, usually because detection or queuing delayed the request past the buffer's history window. Rather than guess a pose, tf raises the error. The stamp fell off the back of the buffer, so the fix is a faster path from image to lookup or a request made at capture time, not a looser lookup.
Is rospy.Time(0) ever the right call?
Only on a link that never moves. Time(0) returns the newest stored sample for that transform, whatever its age. On a truly static transform the newest sample equals every other sample, so nothing is lost. During calibration the robot links move at every pose, so newest pairs a fresh image with an old pose. Some calibrators use it for the fixed camera-to-flange bolt-on while sampling the moving links at the image stamp; that split is legitimate.
How much timestamp skew can a hand-eye calibration tolerate?
Budget it backwards from motion: acceptable error divided by tool speed. At 250 mm/s, 2 ms of skew moves the tool 0.5 mm; at 50 mm/s the same 0.5 mm allows 10 ms. If you stop the robot at each pose, skew stops mattering, because a stationary tool has no motion to smear and even 35 ms costs nothing. Decide your case first: stopped poses need a gate, moving capture needs a millisecond budget.
Why can't I just use the robot pose when the image arrives?
Because the image describes an earlier instant. Exposure, sensor readout, transport and detection sit between capture and processing, 35 ms in the worked example, and every millisecond of robot motion in between changes the true pose. The stamp in the image header is the only record of when the pixels were real. Pairing them with the pose at processing time bakes the whole latency into every calibration sample.
How do I prove the timing fix worked?
Run one calibration with Time(0) as a negative control and one with stamp-matched lookups; the residual gap between them is the size of your timing error. Then command the robot to fresh poses the solver never saw and measure the landing against the marker's known position. If a run only converges once the marker threshold is loosened, treat it as mistimed and find the delay rather than accepting the looser fit.

