Embed a camera image in your HMI panel with cursors and zoom

A vendor viewer window fights your HMI: it grabs the mouse, spawns its own menus, and hides the frame buffer your inspection reads. The fix is to own the display. Acquire frames as raw buffers from the camera SDK, paint each one into a panel control that belongs to your application, and convert every cursor position back to image pixels with one mapping that carries the current zoom and pan. The worked example below runs a 12 MP stream on a 1024 × 768 panel, and the same arithmetic scales to any sensor and viewport. You finish with cursors that read true pixel coordinates, zoom that stays put under the mouse, and a repaint loop that keeps up with the camera.

What you need

Four pieces cover it:

• An area scan camera whose SDK hands you raw frame buffers, not only a viewer window. The area scan cameras page covers GigE, USB3, 10GigE and CoaXPress models, and the camera finder filters by resolution, interface and shutter.
• A GUI toolkit with an owner-draw panel, a paint callback and mouse plus scroll events. Any native toolkit qualifies; the display path is read and written by your code, with no vendor window inside the event chain.
• One machine for acquisition, inspection and display. A vision controller with built-in light controller channels keeps the camera, the lights and the panel in one box.
• Frame buffers in Mono8 for measurement work; request Bayer only when the operator needs color.

The mapping that drives everything

Keep three doubles as the whole display state: zoom, the screen pixels per image pixel, and offset_x and offset_y, the screen-pixel distances the image origin sits left of and above the viewport's top-left corner. Everything else follows:

The y formulas run in parallel against the viewport height. The offset has a second reading: offset_x = zoom × u, where u is the image x coordinate shown at the viewport corner, so a pan is nothing more than storing a new u. Run the conversion in the same coordinate space your paint code draws in. If the toolkit reports logical pixels while scaling for display DPI, convert cursor and offsets to device pixels first, or at 150% scaling every result lands 1.5× away from the truth.

Worked example: 4000 × 3000 on a 1024 × 768 panel

The LS-CMR-0166 delivers 4000 × 3000 at 30 fps over USB3, which makes it a concrete stand-in for any 12 MP Mono8 stream. One Mono8 frame is 4000 × 3000 = 12,000,000 bytes, about 12 MB.

Fit-to-window zoom is min(1024 / 4000, 768 / 3000) = min(0.256, 0.256) = 0.256, with offset (0, 0). The frame fills the panel edge to edge, the clamp range collapses to zero, and the fit view cannot pan.

Step the zoom to 4× and drag the image 400 screen pixels left and 300 up, so the offset is (400, 300). A click lands at screen (800, 600):

Two checks before trusting the number. Forward: 300 × 4 - 400 = 800 and 225 × 4 - 300 = 600, exactly the click coordinates. Bounds: 0 ≤ 300 < 4000 and 0 ≤ 225 < 3000, inside the frame.

Scroll one step to 8× with the cursor held at (800, 600). The image point under it must stay put, so offset_x = 300 × 8 - 800 = 1600 and offset_y = 225 × 8 - 600 = 1200; the check runs (800 + 1600) / 8 = 300. Clamps at 4×: offset_x runs from 0 to 4 × 4000 - 1024 = 14976 and offset_y from 0 to 4 × 3000 - 768 = 11268, and both example values sit inside.

Build the display loop

1. Start acquisition and take frames as raw buffers from the SDK: Mono8 for measurement, Bayer only if the operator needs color.
2. Hold the display state in doubles — zoom, offset_x, offset_y — initialized at fit to window: zoom = min(W / img_w, H / img_h), offset (0, 0).
3. On each frame callback, scale the frame into a back buffer, nearest-neighbor at zoom ≥ 1 and area average below 1, then swap buffer pointers. Never scale inside the paint handler.
4. On paint, blit the back buffer and draw the crosshair, ROI box and labels on an overlay above it, leaving the frame buffer untouched, exactly as when you overlay text on a live camera image.
5. On click and mouse move, convert with image_x = (screen_x + offset_x) / zoom, carry doubles all the way, and floor only the integer index you hand to a tool.
6. Reject the point when x falls outside 0 to img_w - 1 or y outside 0 to img_h - 1; a silent out-of-range cursor corrupts measurements downstream.
7. On scroll, step zoom in ×2 jumps (0.25, 0.5, 1, 2, 4, 8) and set the offset about the cursor with offset' = image × zoom' - screen.
8. Clamp each axis to the range 0 through zoom × image - viewport; when that range is negative, center the image instead.
9. Push accepted coordinates through your pixel-to-millimeter calibration before they reach HMI tags; the defect location with pixel coordinates and calibration guide picks up from there.

Prove the mapping end to end at 1×. With offset (0, 0), a click at viewport (0, 0) must read image (0, 0). Pan to offset (4000 - 1024, 3000 - 768) = (2976, 2232), click viewport (1023, 767), and the result must be (1023 + 2976, 767 + 2232) = (3999, 2999), the last pixel of the frame. Both corners agreeing means the loop is correct.

Choosing the zoom the task needs

Derive the top working zoom from the smallest feature an operator must judge by eye, and spend about 10 screen pixels across it — below roughly 8, a feature stops reading as shape on a shop-floor monitor. If the defect tool flags scratches 2 image pixels wide, the requirement is 10 / 2 = 5, and the next ×2 step, 8×, shows them at 16 screen pixels; at 4× they span 8, visible but thin, enough for a pass or fail glance and not for shape judgment. Set the bottom of the range just as deliberately. The fit view at 0.256 puts one image pixel into 0.256 screen pixels, so a 2-pixel scratch is half a screen pixel — orientation only, and useless for focus, because downsampling averages away the blur you are looking for. Cap the top too: past roughly 6 to 8 screen pixels per image pixel the view is blocky, and interpolation beyond that pretties the squares without adding information.

Keep the repaint inside the frame budget

The example stream moves 12 MB × 30 = 360 MB/s of pixel data. USB3 carries 350 to 400 MB/s in practice, so a full-rate 12 MP panel view fits with a little headroom. GigE's practical 110 MB/s holds the same frame near 110 / 12 ≈ 9 fps, so on GigE either repaint every third frame or give the panel a reduced-size copy while the inspection works on the full frame.

Split the work across threads: acquisition scales the newest frame into a back buffer and bumps a counter, the panel blits the latest buffer on a paint timer at display refresh around 60 Hz, and nothing in the paint path touches the camera or the disk. When the panel comes up blank, the blank image window troubleshooting guide sorts the usual causes; judder in the live view points to screen tearing diagnosis. Exposure and white balance belong in the same runtime loop — the live camera parameter control guide shows the pattern.

Cursor traps that shift the result one pixel

• Integer state. (801 + 400) / 4 = 300.25, and truncating early compounds the error across every pan and zoom. Carry doubles through the state and floor once, at the last step.
• Row and column order. Frame buffers index [row][column], which is [y][x], so a swap mirrors the cursor across the diagonal. The corner test in the display loop catches it in seconds.
• Pixel centers. A screen pixel at index i covers i through i + 1, so the click at (800, 600) touches the left edge of pixel 300's footprint while its center reads (300.5, 225.5). Pick the convention your measurement tools expect and hold it everywhere; the subpixel corner coordinate convention guide and the blob centroid half-pixel offset article map the two common choices.
• DPI scaling. At 150% scaling the toolkit can hand the mouse logical pixels while the back buffer lives in device pixels, and every conversion then lands 1.5× off. Do both sides of the mapping in one space.
• Gray frames from a color camera. That is the SDK handing over Mono8; if the operator needs color, request Bayer and decode it, as the encoding fallback fix describes.

When a viewer window or a web embed wins

Owner-drawn panels earn their keep when cursors, ROIs or measurements must live inside your HMI. For pure process monitoring, a viewer on a second screen is less code, and the second monitor setup without a frame grabber shows how little. For SCADA-style dashboards, serve the stream over the network and embed it: the camera stream in a web HMI guide walks that path, and the smart camera image in a PLC HMI article explains what actually displays when the camera thinks for itself. Packaged HMI software splits the same way: some packages let you embed live video directly in an HMI display, and others expect the camera to serve a page you drop into the HMI as a browser object.

Cameras and the controller behind the panel

Three cameras cover the display cases above, and one controller runs the loop.

For moving parts, the MV-CH120-11UC adds a global shutter and a high-speed rating in the 12 MP class over USB3. For the review bench, the MV-CS200-10UM raises the detail to 20 MP mono.

One box can run acquisition, inspection and the panel together.

The MV-VC3301H-128G66 pairs an Intel Core i3 with 8 GB of RAM and a 128 GB SSD, carries four built-in light controller channels with no PoE, and supports PCIe expansion.

Decision and next step

• Review station, static parts, your own UI: owner-drawn panel with the nine steps, a 12 to 20 MP mono camera on USB3, working zoom to 8×, budget 12 MB per frame at 30 fps.
• Moving line with an operator display: the same panel behind a global-shutter camera, with repaint tied to the trigger rate instead of a free-running timer.
• Dashboard or SCADA front end: skip the owner-drawn panel and embed the stream, as the web HMI guide shows.

Start the camera configurator to match sensor, interface and shutter to your panel budget, browse the other display and integration guides on the systems topic page, or bring your frame sizes and zoom targets to us through the contact page.