Map an EGL frame to a cv::cuda::GpuMat with the right pitch and stride

Frames that a camera pipeline produces directly in GPU memory arrive with their own row layout, and it is almost never the packed layout OpenCV expects. Wrap one as a cv::cuda::GpuMat with only rows, columns and type, and OpenCV walks the wrong distance between rows: the image comes out sheared. Convert the same buffer with the wrong channel code and red and blue trade places. This guide gives you the three fixes: pass the buffer's stride as the step argument, pick the right cvtColor code for an ABGR32 frame, and prove both with a downloaded test image before the pipeline runs live. The same habits apply to any hardware buffer — camera ISP output, decoder output, an SDK device pointer — that you want inside OpenCV CUDA without a host copy.

Why does an EGL frame come out warped in a cv::cuda::GpuMat?

The warp is a stride mismatch: the GpuMat was built without a step argument, so OpenCV reads row n at n × width × 4 bytes while the buffer stores it at n × pitch — the error grows by each row's padding, and the picture shears diagonally.

A pitch-linear EGL buffer pads every row out to an alignment boundary the producer chose. The true distance between row starts is the pitch field of the CUeglFrame, not width times the element size. A header built without an explicit step resolves its stride to width × elemSize — 4 bytes per pixel for CV_8UC4. The difference accumulates quietly: row 0 lines up, row 1 is a few pixels late, row 1000 is thousands of pixels late. Nothing faults, because the reads normally stay inside the allocation. You get a clean, confidently wrong image. The variant that does fault — mapping with the wrong plane count — is its own failure, covered in GPU frame mapping illegal memory access: check pitch and planes.

Three symptoms, three causes:

• Diagonal shear that grows toward the bottom of the frame — the missing step argument, fixed in the next section.
• Red and blue swapped — the wrong cvtColor code, fixed under the conversion section below.
• A conversion call that rejects planar YUV input — a producer format problem, fixed by requesting ABGR32 output.

Which GpuMat constructor argument holds the buffer's stride?

The step parameter — fifth argument of the external-data constructor, cv::cuda::GpuMat(rows, cols, type, data, step) — takes the EGL pitch, and with it OpenCV advances exactly one stride per row, so the geometry is exact however wide the padding is.

With eglFrame as the CUeglFrame returned by the mapping call:

CUeglFrame eglFrame = {};                  // filled by cuGraphicsResourceGetMappedEglFrame

cv::cuda::GpuMat abgrMat(eglFrame.height, eglFrame.width,
                         CV_8UC4,                  // 4 bytes per pixel
                         eglFrame.frame.pPitch[0], // device pointer, plane 0
                         eglFrame.pitch);          // row stride, in BYTES

cv::cuda::GpuMat bgrMat(eglFrame.height, eglFrame.width, CV_8UC3);
cv::cuda::cvtColor(abgrMat, bgrMat, cv::COLOR_RGBA2BGR);

The constructor stores a header — pointer, size, type and step — and allocates nothing, so wrapping costs no memory and no copy. Step and eglFrame.pitch are both in bytes: pass the pitch through untouched, because dividing it by 4 "to get pixels" is the second most popular way to shear this frame. Size the three-channel destination the way you would size any multi-channel buffer — Check multi channel image buffer size before processing.

The CPU-side twin of this trick, wrapping a host SDK buffer in a cv::Mat without copying, is Wrap an SDK frame buffer in an OpenCV cv::Mat without copying.

How do I register the EGL image and read the pitch in CUDA?

Four documented calls move an EGLImageKHR to a usable CUeglFrame: register the image with cuGraphicsEGLRegisterImage and CU_GRAPHICS_MAP_RESOURCE_FLAGS_NONE, map it with cuGraphicsResourceGetMappedEglFrame, synchronize the context, and release the registration with cuGraphicsUnregisterResource when the frame is consumed.

Declare the three objects once and reset them per frame:

CUresult           status    = CUDA_SUCCESS;
CUgraphicsResource pResource = NULL;
CUeglFrame         eglFrame  = {};

1. Establish the CUDA context once per process with cudaFree(0), the handshake this path needs before its first registration.
2. Register the image: status = cuGraphicsEGLRegisterImage(&pResource, image, CU_GRAPHICS_MAP_RESOURCE_FLAGS_NONE); where image is the EGLImageKHR produced upstream — NvEGLImageFromFd on a dma-buf fd, for example. Compare status against CUDA_SUCCESS and stop on anything else.
3. Map the resource: status = cuGraphicsResourceGetMappedEglFrame(&eglFrame, pResource, 0, 0); this fills width, height, pitch and frameType.
4. Wrap only on the pitch path: if (eglFrame.frameType == CU_EGL_FRAME_TYPE_PITCH) build the GpuMat from frame.pPitch[0] and pitch, exactly as the previous section shows.
5. Synchronize: cuCtxSynchronize() after the mapping and again after your kernels, so producer and consumer stay ordered on the device.
6. Release before the next acquire: cuGraphicsUnregisterResource(pResource); — the registration lives and dies inside the frame-acquire scope.

Upstream, the producer side of the EGL stream writes the frames this consumer maps; destroy the EGL image with NvDestroyEGLImage once the resource is unregistered.

Which cvtColor code converts an ABGR32 pitch frame to BGR?

Use cv::COLOR_RGBA2BGR: on this path each pixel's four bytes sit in R, G, B, A order in memory, so channel 0 of the CV_8UC4 header is red — exactly what COLOR_RGBA2BGR expects, while COLOR_BGRA2BGR reads channel 0 as blue and the two colors trade places.

The naming trap explains the bug. The producer labels the format by its 32-bit word — createNvBuffer calls it NvBufferColorFormat_ABGR32 — while the bytes in memory run R first, which is what the working conversion implies. The code that "sounds right" for a label ending in ABGR is the one that swaps red and blue. A five-second check settles any doubt: point the rig at something red and something blue, run both conversion codes once, keep the one that matches the world.

How big is the pitch error on a 2616-pixel frame?

A 2616-column CV_8UC4 row is 2616 × 4 = 10,464 bytes packed; a mapped frame reporting pitch = 10,752 carries 288 padding bytes per row, so each row is read 72 pixels ahead of the one before it — row 100 sits 7,200 pixels off.

The 10,752 value is illustrative — the real number is whatever eglFrame.pitch prints, and the arithmetic is identical either way:

By the last row the displacement has passed 54 full row widths, so the bottom of the picture is drawn from content dozens of rows earlier, folded along one hard diagonal seam. The failure also stays silent: the allocation holds 1964 × 10,752 = 21,116,928 bytes, while the wrong mapping never reads past 1964 × 10,464 = 20,551,296 bytes. Inside the buffer, no fault, all wrong — which is what separates this bug from the plane-count bug that faults outright.

What do I do about a YUV420 semiplanar EGL frame?

Do not try to convert it in OpenCV: the CUDA cvtColor set has no YUV420-planar-to-BGR entry, so the call fails at runtime — request a pitch-linear ABGR32 frame from the producer instead, which maps as a single CV_8UC4 surface with one stride.

The semiplanar layout — cudaEglColorFormatYUV420SemiPlanar_ER in the frame comment — keeps Y in one surface and interleaved V/U in a second, with chroma at half the Y width and half the Y height: two plane pointers under one pitch. You can wrap both surfaces as headers, single-channel for luma and two-channel for chroma, but no OpenCV kernel turns them into BGR.

The fix belongs upstream, where the output format is chosen: create the buffer as NvBufferColorFormat_ABGR32 with NvBufferLayout_Pitch — createNvBuffer(resolution, NvBufferColorFormat_ABGR32, NvBufferLayout_Pitch) in the native-buffer path — and every section of this guide applies unchanged. If a producer you cannot change insists on YUV420, convert in your own kernels and go through packed 4:4:4 on the way: 4:2:0 to 4:4:4 to RGB is the recommended sequence in Recommended 8-bit YUV formats for video rendering. It stays on the GPU with no host copies, at twice the kernels of the ABGR32 route.

One habit keeps the per-frame loop clean: create the dma-buf once at stream resolution, then copy each acquired frame into it with copyToNvBuffer — a device-side copy that keeps allocation out of the frame path.

How does this stay zero-copy in a four-camera pipeline?

Run register–map–convert–unregister inside each consumer's frame-acquire scope and the frame never touches host memory; at 20 fps one camera has a 50 ms budget, and four cameras sharing one GPU have to fit their conversions inside it.

The wrapping itself is free — it is a header. The budget goes to conversion kernels and the per-frame registration block, so at 80 combined frames per second, kernel time, not mapping, sets the ceiling. When decode or conversion starts starving the CPU instead, that is the offload problem, covered in RTSP decode CPU overload: offload decode and conversion to the GPU. When the views need tighter capture-side alignment than software acquire order gives, Synchronized camera bank switching for multi-view capture handles that half.

Assembling the node rather than adapting one puts the controller first: LAN port count for the cameras, PoE where the cameras draw power, memory for queue depth, and PCIe expansion when the GPU plan outgrows the onboard one. These controller models span that range:

All 3 share: Certification CE, KC. Values as printed in each product's data sheet; open a part for its drawing and full specifications.

MV-VC3301H-128G66 includes a VM6200Pro license, so vision software arrives with the node. MV-VC4712-256G10-2TH adds PCIe expansion, PoE camera ports and 6 built-in light channels for a rig that also drives lighting. MV-VC5525-256G70-NN carries 5+ LAN ports, PoE camera ports and 32 GB of memory. That port count is what lets four GigE cameras connect directly, no switch in the path; the vision controllers category filters on the same numbers.

The capture side starts with the cameras:

MV-CU020-90GC pairs a 2.5 MP global shutter color sensor with GigE and a C-mount. Global shutter matters under software sync: the whole frame is exposed at once, so motion stays comparable across the four views. When 2.5 MP runs short of the field of view, the area scan cameras shelf steps up to sensors beyond 25 MP.

To size the node end to end — cameras, controller, light channels — start at the system configurator; expert support comes with every order.

How do I verify the mapping before the line goes live?

Wrap one frame, download it, and look at it: cv::cuda::GpuMat::download into a cv::Mat, cv::imwrite at a fixed frame number, and a test target with straight lines plus known red and blue patches — shear means the step is wrong, swapped colors mean the cvtColor code is wrong.

cv::Mat host;            // destination, allocated by download()
bgrMat.download(host);   // one device-to-host copy, diagnostics only
cv::imwrite("frame_check.png", host);

Hold the frame index fixed — frame 20, say — so successive code changes compare the same picture. Then work the checklist:

• Print eglFrame.pitch and width × 4 once at start-up. Equal means rows are packed, and the warp lives somewhere else.
• Print eglFrame.frameType. It decides whether the mapping hands you plane pointers at all (the FAQ covers the array case).
• Shoot a straight-line target. Residual shear means the step is still wrong; the classic variant is a pitch passed in pixels instead of bytes.
• Keep red and blue patches in view. Swapped colors point at the conversion code, not the stride.
• Run a hundred frames or more before declaring victory. A mapping that is right once can still tear under load — Torn frames in a camera callback: copy the buffer before you queue it is the capture-side twin of that failure.

Once the check image comes out clean — lines straight, red red, blue blue — the mapping is proven and everything downstream runs on real geometry. If a frame still refuses to cooperate, contact the application team with the downloaded test image and the printed pitch; the fix is usually one line. More GPU-side buffer handling sits on the systems topic page.