Visual servoing detection messages: Python to C++

A detector and a visual-servoing application do not need to run in the same process or language. You can publish detections from Python and receive them in a C++ subscriber, provided both sides agree on geometry, timing and validity. This procedure defines that interface, shows a small receiver pattern, and explains how to test stale-data rejection before connecting detections to a motion command.

1. Separate detection transport from motion control

Keep inference inside the detector process. Its publisher sends observations, not velocity commands. The C++ application validates each observation before making it available to the existing servo controller.

Use this integration order:

1. Define and test the detection-message contract using recorded images.
2. Connect the subscriber and inspect accepted observations without commanding motion.
3. Add freshness checks, target selection and explicit loss-of-target handling.
4. Connect the validated observation to the controller through its existing command gate.

A subscriber callback should not run a complete control iteration. Inference and message arrival are asynchronous; the control application needs its own scheduled update and timeout checks.

For adapting camera buffers and image formats, use the separate guide to visual-servoing camera integration. The interface here begins after inference has produced a detection.

2. Define one message contract before writing callbacks

A useful envelope identifies the source image, describes its geometry and explicitly distinguishes a detection from an empty result. Do not publish only four unexplained coordinates.

For this example, coordinates refer to the full acquisition image, with the origin at its upper-left edge. Horizontal coordinates increase rightward; vertical coordinates increase downward. Use continuous edge coordinates and half-open boxes: the right and bottom edges are excluded.

State whether those coordinates describe the distorted acquisition image or a rectified image. A frame identifier alone does not communicate distortion processing, crop offsets or calibration revisions.

Publish an explicit empty result when a processed image contains no acceptable target. Silence could instead mean a stopped detector, broken transport or stalled acquisition. The receiver must handle both empty results and missing messages.

If several objects are present, add class and tracking identifiers, or publish an array and select the target downstream. The receiver below assumes that an upstream selector has already chosen one target. A high score does not prove that successive boxes belong to the same object.

3. Convert Python boxes into the declared coordinates

Resize and letterbox operations change coordinate meaning. Convert detections back to the declared image space before publishing, unless the contract explicitly describes the transformation.

For uniform scaling, padding and cropping, the inverse mapping is:

Apply the transformation to both box corners. Nonuniform resizing requires separate horizontal and vertical scale factors. Rectification requires its own geometric mapping; this affine expression does not reverse lens distortion.

Consider a hypothetical full image measuring 1280 × 720 pixels. A 640 × 480 pixel crop begins at offset (160, 120). Scaling that crop by 0.8 produces 512 × 384 pixels, with 64 pixels of padding above and below inside a square network input.

A network box from (80, 104) to (240, 264) becomes:

The following Python helper illustrates the publisher-side adapter. publish is your middleware-specific sending function, and image_meta contains metadata preserved from the source image.

def publish_result(publish, image_meta, box, quality,
                   scale, pad_x, pad_y, crop_x, crop_y):
    message = dict(image_meta)
    message.update(found=False, quality=0.0,
                   x0=0.0, y0=0.0, x1=0.0, y1=0.0)

    if box is not None:
        x0, y0, x1, y1 = box
        message.update(
            found=True,
            quality=float(quality),
            x0=(x0 - pad_x) / scale + crop_x,
            y0=(y0 - pad_y) / scale + crop_y,
            x1=(x1 - pad_x) / scale + crop_x,
            y1=(y1 - pad_y) / scale + crop_y,
        )

    publish(message)

Validate preprocessing parameters before calling this helper. Preserve the capture timestamp and sequence through inference; do not replace them with publication time.

Deciding whether the box center is the correct control feature is a separate task. See feature selection for perpendicular approach before treating a detector's rectangle as a physical alignment feature.

4. Put a validated mailbox behind the C++ subscriber

Decode each incoming middleware message into an application-owned structure. Avoid keeping pointers into temporary receive buffers. The example below uses a mutex-protected latest-observation mailbox rather than a queue of historical detections.

It is middleware-independent C++17 application code. A generated subscriber callback calls receive(), while the scheduled control task calls read() with the current time from the validated capture-clock domain.

The configured dimensions, identifiers and thresholds are illustrative test settings, not hardware specifications.

#include <chrono>
#include <cmath>
#include <cstdint>
#include <mutex>
#include <optional>
#include <string>

struct Detection {
    std::string source_id, session_id, frame_id;
    std::uint64_t sequence;
    std::int64_t capture_time_us;
    int width, height;
    bool found;
    double x0, y0, x1, y1, quality;
};

class DetectionMailbox {
    using Clock = std::chrono::steady_clock;
    struct Sample {
        Detection detection;
        Clock::time_point received;
    };

    std::mutex mutex_;
    std::optional<Sample> latest_;
    std::uint64_t last_sequence_ = 0;

public:
    void receive(const Detection& d) {
        std::lock_guard<std::mutex> lock(mutex_);
        if (d.source_id != "camera_main" ||
            d.session_id != "run_a" ||
            d.frame_id != "camera_main_optical") return;
        if (d.sequence <= last_sequence_) return;
        last_sequence_ = d.sequence;

        // A newer empty or invalid result clears the old target.
        latest_.reset();
        if (!d.found || d.capture_time_us <= 0) return;
        if (d.width != 1280 || d.height != 720) return;
        if (!std::isfinite(d.quality) || d.quality < 0.7) return;
        if (!std::isfinite(d.x0) || !std::isfinite(d.y0) ||
            !std::isfinite(d.x1) || !std::isfinite(d.y1)) return;
        if (!(0 <= d.x0 && d.x0 < d.x1 && d.x1 <= d.width &&
              0 <= d.y0 && d.y0 < d.y1 && d.y1 <= d.height)) return;

        latest_ = Sample{d, Clock::now()};
    }

    std::optional<Detection> read(std::int64_t capture_now_us) {
        std::lock_guard<std::mutex> lock(mutex_);
        if (!latest_) return std::nullopt;
        const auto& sample = *latest_;
        const auto age_us =
            capture_now_us - sample.detection.capture_time_us;
        if (age_us < -2000 || age_us > 50000) return std::nullopt;
        if (Clock::now() - sample.received >
            std::chrono::milliseconds(50)) return std::nullopt;
        return sample.detection;
    }
};

The score threshold is an application setting, not a statement that the score represents a calibrated probability. Select it using representative validation images.

At every control update, an empty return must withdraw observation validity. Do not leave the previous motion command active merely because the latest read failed. Choose the hold, controlled-stop or reacquisition behavior through the machine's control and risk assessment.

The sample intentionally rejects a new session. Production restart handling should explicitly reset sequence state, clear the target and validate the replacement session before rearming. Do not automatically accept any unfamiliar source identifier.

5. Set freshness limits from the error budget

Capture age and time since receipt answer different questions. Capture age measures how old the scene information is. A local monotonic watchdog detects missing arrivals without depending on the source clock.

The sample permits a small future-timestamp tolerance. That tolerance must come from a measured clock-error bound, not an arbitrary allowance for badly synchronized clocks. If capture and receiver clocks are unrelated, their timestamps cannot be subtracted directly.

Use documented synchronization or a validated clock mapping. If only host arrival time is available, label it accordingly and account for the unmeasured acquisition and transport delay. An arrival watchdog alone cannot prove image freshness.

For a hypothetical pipeline:

If the allocated stale-image displacement is 6 pixels at that speed, the corresponding age limit is:

This is a constant-image-speed estimate, not a stability proof. Clock uncertainty, acceleration, localization error and actuation delay consume additional margin. The separate visual-servoing loop design guide addresses the broader control problem.

Configure bounded middleware queues. A receiver mailbox cannot remove latency already accumulated upstream. Select reliability and queue behavior using measured loss and delay; reliable delivery does not guarantee fresh observations.

6. Verify rejection paths before enabling commands

Run the interface with motion commands inhibited through documented controls. Record the received message, acceptance decision and rejection reason. Keep image sequence, capture time and session identifiers in the same log entry.

Execute these tests:

1. Geometry: Publish the worked-example box and verify its overlay on the original image. Check every crop and resize setting used in production.
2. Empty result: Send a newer message with found=false. Confirm that the previous target becomes unavailable immediately.
3. Stale result: Delay a valid message beyond the capture-age limit. Confirm that recent arrival does not make it acceptable.
4. Stopped publisher: Stop message delivery. Confirm that the scheduled task invalidates the observation without needing another callback.
5. Reordering: Deliver an older sequence after a newer one. Confirm that it cannot replace the accepted observation.
6. Malformed geometry: Send nonfinite coordinates, inverted corners and incorrect dimensions. Confirm rejection without an exception escaping the callback.
7. Restart and identity: Change the session, source or coordinate frame. Confirm that unexpected messages cannot refresh the active target.
8. Target switch: Present two similar objects. Verify the target-selection policy rather than assuming sequence continuity proves object identity.

Measure age distributions under processing load, not just while displaying a preview. Report missed deadlines and rejected observations alongside throughput.

If acquisition itself stalls, separate that fault from subscriber behavior using camera acquisition recovery checks. If image ownership is unclear, review camera buffer ownership before passing image data between asynchronous tasks.

7. Choose acquisition hardware without changing the contract

The message interface can remain stable when acquisition hardware changes, provided its declared geometry and timing remain correct. Start with the required image quality, exposure behavior, interface and measured latency, rather than selecting hardware from detector throughput alone.

Compare area scan cameras through the camera finder. Check timestamp semantics and software compatibility in the applicable documentation before ordering.

For a GigE acquisition path, consider LS-CMR-0053.

For a USB3 acquisition path, consider LS-CMR-0166.

These are acquisition candidates, not guarantees of inference speed or subscriber compatibility. Verify the complete acquisition-to-control path with representative workloads.

If you need consolidated camera acquisition and processing, review vision controllers. Confirm runtime dependencies, camera connections, middleware support and any required software licenses separately.

8. Complete the integration checklist

Before enabling the commissioned operating mode, confirm:

• Coordinates match the declared full-image or rectified-image space.
• Capture timestamps survive inference unchanged, with a validated clock relationship.
• Empty results clear the selected target rather than preserving its last box.
• Queue depth, callback work and message size remain bounded.
• Freshness is checked on every control update, not only on message arrival.
• Restarts and target changes require explicit state handling.
• Loss of valid observations invokes the documented command-gating behavior.
• Independent machine-safety measures remain active.

For the acquisition build, start with the camera configurator. For interface review, send the message schema, preprocessing transform, clock assumptions and measured latency distribution through application support.