KLT tracker custom frame buffers: why init works and process fails

You bypassed the framework pipeline, called NvMOT_Query, and honoured what it reported: NVBUF_COLOR_FORMAT_GRAY8 frames in NVBUF_MEM_CUDA_UNIFIED memory with NVMOTCOMP_CPU compute. NvMOT_Init returned NvMOTStatus_OK. Then NvMOT_Process returned NvMOTStatus_Error with no log line to point at, and every field you typed looks plausible. Two rules of the library's buffer contract are broken, not one. The mapped host address must point at your unified-memory allocation, and the output tracked-object batch must arrive zeroed so the library can create its lists. Fix both and the same call sequence passes. Here is where each rule bites, the arithmetic behind the buffer sizes, and the host hardware that keeps a tracker like this fed.

The symptom and the two changes that fix it

Value-initialised NvBufSurfaceParams leaves mappedAddr.addr[0] at nullptr, and on a discrete GPU card the tracker dereferences that member — dataPtr alone is not read. The second break is the output parameter. A raw NvMOTTrackedObjBatch on the stack carries garbage in its counters, and the library allocates its per-stream lists only when those counters read zero. Apply either change alone and NvMOT_Process still returns NvMOTStatus_Error. Apply both and the same call returns NvMOTStatus_OK.

The pitch row is the trap, not your bug: the original value was already correct for gray, and the four-byte variant is a misfix that quietly describes a different pixel format. The two rows above it are the ones that break the call.

Why the tracker reads mappedAddr, not dataPtr

NvBufSurfaceParams describes the plane twice. dataPtr says where the allocation lives; mappedAddr.addr[0] says where the library is allowed to dereference it. On a discrete GPU card the KLT path goes through the mapped address, so set it to the same pointer and leave the EGL field empty:

bufferParam->dataPtr = yDevMem;
bufferParam->mappedAddr.addr[0] = yDevMem;   // the fix: was nullptr
bufferParam->mappedAddr.eglImage = nullptr;  // discrete card: no EGL image

On an embedded GPU module the mapped address must come from a real EGL mapping of the buffer, so the pointer shortcut above is strictly the discrete-card case. Keep layout at NVBUF_LAYOUT_PITCH, because cudaMallocManaged returns linear memory and the plane descriptor should say so.

Zero the output batch before the first call

The output parameter is caller-owned: you declare the struct, the library fills it. Zero it once, before the first NvMOT_Process call for a context:

NvMOTTrackedObjBatch outputTrackedBatch;
memset(&outputTrackedBatch, 0, sizeof(outputTrackedBatch)); // the fix: was raw stack memory

After the call, read the filled entries for that streamID and overlay them on the frame you just submitted. The next call takes fresh detections for the next frame, not the previous output boxes. Give each stream its own context so the lists stay separate — the same discipline as running multiple GenTL cameras in one process with one acquisition object each.

Pitch is bytes per row, and gray is one byte per pixel

planeParams.pitch[0] is bytes per row. With the NVBUF_COLOR_FORMAT_GRAY8 the tracker asked for, bytesPerPix[0] is 1, so a 2560-pixel row is 2560 bytes and the whole plane is 2560 × 720 = 1,843,200 bytes, about 1.76 MiB. Set dataSize and planeParams.psize[0] to that same product, and planeParams.offset[0] to 0 for the single plane. The width-times-sizeof(uint32_t) form belongs to four-byte pixel formats and overstates a gray plane fourfold; leave it out.

At init, the maxima you declare must cover the frames you submit: maxHeight, maxWidth, maxPitch and maxSize in the per-transform batch config, set equal to the frame values for a fixed-resolution stream. Point customConfigFilePath at the tracker's YAML (the sample tracker_config.yml) with customConfigFilePathSize covering the terminator.

cudaMemAttachGlobal, not cudaMemAttachHost

NVMOTCOMP_CPU compute suggests pinning the allocation to the host. Do not. Host-attached managed memory cannot be reached from the GPU on devices that lack cudaDevAttrConcurrentManagedAccess until you re-attach it with cudaStreamAttachMemAsync. The global attach is reachable from both sides and costs nothing here:

void* yDevMem = nullptr;
const uint32_t width = 2560;              // pixels
const uint32_t height = 720;              // pixels
const uint32_t pitch = width * 1u;        // GRAY8: 1 byte per pixel
const uint32_t fullSize = pitch * height; // 1,843,200 bytes
cudaMallocManaged(&yDevMem, fullSize, cudaMemAttachGlobal);

The struct and function names are the library's; the values are this worked example. Swapping the flag alone does not clear the process error — the mapped address and the zeroed batch are the two changes the call needs.

A 12 MP gray stream, in bytes

Take the camera end first. The LS-CMR-0166 is a 12 MP USB3 camera with a 4000 × 3000 sensor at 30 fps.

One GRAY8 frame from that sensor is 4000 × 1 = 4000 bytes per row and 4000 × 3000 = 12,000,000 bytes per plane — about 11.4 MiB. A four-deep ring for the tracker holds about 45.8 MiB: allocate once and reuse the buffers instead of allocating per frame. At 30 fps the stream is 12,000,000 × 30 = 360 MB/s, or 2.88 Gbit/s of payload — nearly three times a 1 GbE line rate, comfortable on a 5 Gbit/s USB3 link.

If you copy each captured frame into the unified allocation before the call, push the source buffer back only after the copy. When a live preview shares the same callback, copy before you queue, or you get torn frames in a camera callback.

The call sequence and the two-frame test

Keep the init side as you have it: one stream, one transform, NVBUF_MEM_CUDA_UNIFIED with NVBUF_COLOR_FORMAT_GRAY8, maxima equal to the frame. The process side then looks like this:

NvBufSurfaceParams* bufferParamPtr{ bufferParam };

NvMOTObjToTrack objects[1] = {};
objects[0].classId = 0;
objects[0].bbox.x = 40;      objects[0].bbox.y = 40;
objects[0].bbox.width = 40;  objects[0].bbox.height = 40;
objects[0].confidence = 1.f;
objects[0].doTracking = true;

NvMOTFrame frame = {};
frame.streamID = 0;
frame.frameNum = 0;                  // increment on every frame
frame.numBuffers = 1;
frame.bufferList = &bufferParamPtr;
frame.objectsIn.detectionDone = true;
frame.objectsIn.numAllocated = 1;
frame.objectsIn.numFilled = 1;
frame.objectsIn.list = objects;

NvMOTProcessParams processParams = {};
processParams.numFrames = 1;
processParams.frameList = &frame;

const auto status = NvMOT_Process(pContextHandle, &processParams, &outputTrackedBatch);
// pass: status == NvMOTStatus_OK on frame 0 and frame 1

Prove it with two frames: submit frame 0 with the seed box at (40, 40), then frame 1 with the image content shifted 8 pixels right and frameNum incremented. The status must be NvMOTStatus_OK on both, and the tracked box for streamID 0 must land near (48, 40). If the status is fine but the box never moves, chase the pixels the way a blank image with no pixels is probed stage by stage: write a known gradient into the unified buffer and confirm the tracker reports motion.

What not to change

• Do not multiply pitch by sizeof(uint32_t) for gray. That form belongs to four-byte pixel formats; GRAY8 rows are one byte per pixel.
• Do not set eglImage on a discrete card. nullptr is correct there; EGL mappings are the embedded-module path.
• Do not switch computeConfig away from what NvMOT_Query reported. This build of the KLT tracker runs NVMOTCOMP_CPU and asks for gray unified memory.
• Do not expect the attach flag alone to rescue the call. It is hygiene; the mapped address and the zeroed batch are the fixes.
• Do not leave layout anywhere but NVBUF_LAYOUT_PITCH for linear cudaMallocManaged memory.

Host hardware that keeps the tracker fed

Every frame crosses a link and a host buffer before the library sees it. Area scan cameras set the pixel rate, vision controllers supply the ports and RAM, and frame grabbers set how many cameras one box can ingest. For a one-box build, these two cover the common shape:

The MV-VC3301H-128G66 brings 5+ LAN ports, 3-4 USB 3 ports, 8 GB of RAM and 4 built-in light channels, so a GigE camera, a USB3 camera and a light each get their own connection without a switch. The MV-GE2002P dual-port PoE NIC carries power and frames for two cameras over one card. When several high-resolution cameras share the host, push the ingestion onto 10GbE:

The MV-GT2104 adds 4 ports of copper 10GbE for exactly that. Two host-buffer traps sit nearby: two USB3 cameras at full resolution can starve the host USB buffer before the tracker ever runs, and when ingestion dies at the card, diagnose frame grabber buffer allocation errors first. Filter area scan cameras by sensor and interface, or start the configurator at the camera step to build the chain in order.

When KLT is the wrong tracker anyway

KLT follows sparse features by aligning small image patches from frame to frame, so it lives on texture. A flat shiny surface, a part that turns to a new face, or a frame gap long enough for fast motion will drop the tracks, and no buffer fix recovers that. Where parts look alike or appearance changes faster than geometry, move to a detection-association tracker or a 3D sensor — the trade-offs in choosing a 2D or 3D camera for conveyor pick and place are the right frame for that call. Keep KLT for textured targets, high frame rates and modest hardware.

Checks before you call it done

• NvMOT_Query honoured: color format, memory type and compute match what it reported.
• mappedAddr.addr[0] equals your unified pointer on a discrete card; eglImage is nullptr.
• The NvMOTTrackedObjBatch output struct is zeroed before the first process call for the context.
• planeParams.pitch[0] equals width × bytesPerPix; dataSize and psize[0] equal pitch × height.
• Init maxima — maxHeight, maxWidth, maxPitch, maxSize — equal or exceed the frame values.
• Allocation uses cudaMemAttachGlobal.
• The two-frame shift test passes: status NvMOTStatus_OK, box moves to the shifted position.
• frameNum increments every frame; streamID is stable per camera; one context per stream.

Still failing with both changes in place? Capture the exact call arguments to a file and send them to us — a bare NvMOTStatus_Error at that point points at the config file path or the YAML contents, not the buffers.