Rviz crash changing image topic: fix the mutex teardown order

Changing the image topic in Rviz should be a two-second action. Instead, the whole process sometimes dies with a Boost mutex error, taking your visualization and every tool attached to it down with it. The crash is intermittent, which pushes most teams to blame the camera, the driver or the config file. It is none of those: the panel is being destroyed while its image callback is still running. This guide is for engineers whose custom Rviz panel or plugin displays camera frames. You will get the teardown order that removes the crash, the arithmetic that predicts when it fires, and a repeat count that proves the fix.

What the mutex error is telling you

The process dies with two lines on the console:

terminate called after throwing an instance of 'boost::wrapexcept<boost::lock_error>'
what():  boost: mutex lock failed in pthread_mutex_lock: Invalid argument

Read them from the bottom up. pthread_mutex_lock refused the call with EINVAL, the POSIX 'Invalid argument' error, and in this failure the invalid mutex is one that has already been destroyed. Boost wraps that in a lock_error, nothing catches it, and std::terminate ends Rviz. One bad lock kills the whole visualization, not just the panel.

The same message appears for any Boost mutex misuse anywhere in the process, so the text alone does not prove which lock failed. The instrumentation in the verification section below confirms it is the panel teardown path.

The root cause: a callback holding a lock on a dying object

Your image callback does not run on the GUI thread. Rviz delivers frames on a background spinner thread, and the callback locks a mutex that lives inside the widget, usually guarding the frame buffer the panel draws. Deleting the widget destroys that mutex with it. A callback still running, or arriving a moment later, then locks memory that is no longer a mutex, and pthread_mutex_lock answers EINVAL.

Changing the topic runs the same teardown inside the display, which is why a topic change crashes exactly like deleting the panel. The sequence:

1. GUI thread: you change the topic, and the panel starts tearing the old subscription down.
2. Spinner thread: a frame arrives and the callback starts.
3. GUI thread: the teardown completes and destroys the widget, mutex included.
4. Spinner thread: the callback tries to lock, and the process dies.

The window is a few milliseconds wide, so it only opens when a frame happens to be in flight. That is the whole story of 'sometimes'.

Likely causes, in order of probability

1. Widget deleted while the subscription is live (most likely). This covers both repros: deleting a panel and changing a topic. Test: log an entry and exit timestamp in the callback and in the destructor. Every crash shows the destructor starting before the callback exits.

2. The topic change rebuilds the subscriber mid-flight. Some panels construct the new subscription before the old one has finished. Test: set the image topic to empty first, wait a beat, then set the new topic. If the crash vanishes, this ordering is your bug.

3. Config reload tears down displays during startup. Rviz restores saved displays on launch, and a panel that is not yet ready gets its subscription reset at once. Test: does it crash on opening the saved config but never on a manual change?

4. The mutex was never initialized (rare). If the crash also fires on the very first subscribe, the mutex may be constructed after the subscriber starts. Test: move the mutex construction earlier and retest.

Why it only crashes sometimes: the race window

The odds per delete are the fraction of time the callback is busy:

Worked example: take an 8 ms callback, the time to copy a frame and hand it to the widget. That figure sits in the single-digit milliseconds on a desktop-class machine; measure yours once with two timestamps, because the fraction scales linearly with it.

• At 30 fps: period = 1000 / 30 = 33.3 ms. Busy fraction = 8 / 33.3 = 0.24. One delete in four lands inside the callback.
• At 60 fps: period = 1000 / 60 = 16.7 ms. Busy fraction = 8 / 16.7 = 0.48. Every second delete crashes.
• At 5 fps: period = 1000 / 5 = 200 ms. Busy fraction = 8 / 200 = 0.04. One crash in 25 deletes, which is why a slow bench hides the bug for days.

This is why the bug reproduced a few times and then refused to: the dice moved, not the config. For bench work you want the window wide, and a steady 30 fps source makes one delete in four land mid-callback. The LS-CMR-0166 provides that rate: 4000 × 3000 at 30 fps over USB3.

The fix: flag, unsubscribe, drain, then delete

Tear the subscription down in this order, in the destructor and on every topic change:

1. Set the flag. A boolean the callback checks at entry, under the mutex; once it is true, every callback exits immediately.
2. Shut the subscription down. Call the subscriber object's shutdown or unsubscribe method and let it return. New frames stop here.
3. Drain the delivery thread. If the panel owns the spinner thread, join it. If not, wait until the running callback has returned; a condition variable signalled at the callback's exit works.
4. Reset the subscriber object, so nothing resubscribes through a stale handle.
5. Delete the widget. Only now. On a topic change, run the same five steps, then subscribe to the new topic.

In code, the shape is:

// Names below are ours, not library calls — substitute your panel's own members.
bool m_shutting_down = false;   // bool, init false: guards callback entry

void ImagePanel::onImage(const FrameMsg& msg)         // your subscriber's callback
{
    std::lock_guard<std::mutex> lock(m_frame_mutex);  // std::mutex, member of the panel
    if (m_shutting_down) { return; }                  // late frames exit here
    m_last_frame = msg;                               // copy the payload, never reference it
}

void ImagePanel::teardown()                           // destructor and topic change both call it
{
    m_shutting_down = true;                           // step 1: flag first
    m_subscriber.shutdown();                          // step 2: stop delivery
    wait_for_callback_exit();                         // step 3: drain, your condition-variable wait
    m_subscriber.reset();                             // step 4: drop the handle
    // step 5: only past this line is deleting widgets safe
}

Prove it with the twenty-switch test

Run the repro that used to crash it, with the stream publishing continuously:

1. Save the config with the panel open and the image topic set.
2. Reopen Rviz, close the panel, open the panel, set the image topic — one switch.
3. Repeat for twenty switches and count the crashes.
4. Add two counters to the panel, frames started and frames finished, incremented at the callback's entry and exit. Assert the two are equal whenever the panel closes.

The pass mark is twenty consecutive clean switches at a 30 fps stream. The arithmetic that says twenty is enough sits in the first FAQ below. If any switch still crashes, the counters and timestamps from the causes section tell you which thread was where.

When it is not the teardown order

• Crash while loading a saved config, with a truncated display-config value. The Rviz changelog records a separate, since-fixed crash when a display-config parameter was shorter than four characters. Update Rviz, or repair the value in the file.
• Crash when selecting a frame, reporting a double free. That is a plugin-library fault, not a mutex race. Follow the double-free fix for frame selection in Rviz plugins.
• Crash at startup with a robot configuration. Startup crashes that trace back to a MoveIt configuration follow a separate trail; the RViz startup crash walkthrough for MoveIt configurations covers that path.
• The camera fails, not Rviz. A camera that dies when acquisition stops has its own ordering problem — see camera crash when stopping acquisition. A camera that never comes up is usually a start-sequence fault — see camera failed to initialize: fix the start sequence.

Prevention checklist for panels that subscribe

• Subscribe last, unsubscribe first. Write the teardown before you write the subscribe, so review catches a wrong order before the bench does.
• Copy frames inside the callback. Never keep a reference to the delivery buffer; the torn-frames fix for camera callbacks shows what else that mistake costs.
• Give the delivery thread a longer life than every widget that uses it. Stop the subscription, then stop the spinner, then destroy the GUI.
• Treat a config reload as a fresh subscribe. Clear the topic to empty before the panel restores its saved one.
• Keep a fast test stream on the bench. A 30 fps source exposes a teardown race within twenty switches; a 5 fps one can hide it all afternoon.

For a bench stream you can leave running, pick from the area scan cameras or the camera finder, or wire one up in the camera configurator. More fixes at this level sit on the systems topic page, and questions about a specific set-up go through the contact page.