6D pose ground truth: where per-image object labels come from

You are training or evaluating a 6DoF pose-estimation model, you open the dataset, and the scene metadata lists 'a rotation matrix and translation vector' — with no sign of labels for the objects in each image. The labels exist. They sit in a separate per-image annotation file, one pose per object instance. The test split leaves them out on purpose, because predicting those poses is the scored task. This page walks through the file layout, the pose convention and units, a worked projection with the arithmetic shown, and the checks that catch broken labels before they reach a training run.

One pose per object instance, stored in scene_gt

Datasets that follow the BOP annotation layout — the structure most public 6DoF pose benchmarks reuse — store object labels in a JSON file named scene_gt.json. Keyed by image ID, each entry is a list. Each list item is the ground-truth pose of exactly one physical object in exactly that image. When a scene was captured by more than one camera, the poses are split per camera into files named scene_gt_<cam_id>.json, so a pose can never drift away from the camera that saw it.

The camera's own rotation and translation are not in that file. They sit in scene_camera.json beside the intrinsics. Two files, two jobs: scene_gt answers where each object is relative to this camera, scene_camera answers how this camera projects and where it sits in the world. Keep that separation in mind and the rest of the format resolves quickly.

What a ground-truth entry contains

Each entry carries three fields:

• obj_id — which object this instance is; it indexes a mesh file (usually PLY) shipped with the dataset.
• cam_R_m2c — the 3×3 rotation from the object's model frame into the camera frame.
• cam_t_m2c — the 3×1 translation from the model origin into the camera frame, in millimetres in BOP-format files.

The m2c suffix reads model-to-camera, and the convention behind it is one line of matrix math:

Annotations are per instance, not per class: a bin holding three of the same gear produces three entries with the same obj_id, each with its own R and t. Loaders that assume one pose per image are the classic source of silently shifted labels; 6D pose dataset annotation mismatch: objects outside the view works through what that looks like when instances land outside the frame.

Where the camera matrix and extrinsics fit

scene_camera.json lists, per frame, the values that turn camera-frame millimetres into pixels: fx, fy, cx, cy. Rigs also store a timestamp (for example time_us), and moving-rig datasets add each frame's world-to-camera extrinsics. Those camera-side R and t are what a dataset description means by 'rotation matrix and translation vector for the cameras' — they place the camera in the scene, which makes them calibration, not labels.

The useful chain when you need an object pose in the world or robot frame is: object-to-camera from scene_gt, then camera-to-world from the frame extrinsics or from your own hand-eye calibration. Where Z is measured from changes that second step; stereo triangulation world origin: choosing where Z is measured from covers the choice. If your numbers arrive as a rotation vector rather than a 3×3, camera pose: rotation vector to rotation matrix, tvec units covers the conversion and the unit traps. To recover fx, fy, cx, cy from a full 3×4 projection matrix, use decompose a projection matrix into camera intrinsics.

Why the test split ships without poses

For training scenes the poses ship alongside the images. For test scenes they do not: predicting those poses is the benchmark, so the labels stay with the organisers and your submission is scored against them after upload. The habits that follow:

• Treat test images as label-free, and make sure a loader that reads every scene cannot pull test frames into training.
• Validate on a held-out slice of the training split; its labels use the same files and conventions.
• Before trusting a pipeline, render every ground-truth pose over its own training image once. If the overlays land on the objects, your loader reads the convention correctly.

Worked example: from a stored pose to a pixel

Take one annotated instance with these illustrative values, declared here rather than copied from any dataset:

• fx = fy = 1200 px, cx = 640 px, cy = 512 px (a 1280 × 1024 RGB frame)
• R = a 90° rotation about the camera z axis: R = [[0, −1, 0], [1, 0, 0], [0, 0, 1]]
• t = [500, −80, 1200] mm
• mesh corner x_m = [60, 0, 0] mm

Rotate the corner into camera orientation: R · x_m = [0·60 + (−1)·0 + 0·0, 1·60 + 0·0 + 0·0, 0·60 + 0·0 + 1·0] = [0, 60, 0] mm.

Translate into the camera frame: x_c = [500 + 0, −80 + 60, 1200 + 0] = [500, −20, 1200] mm. Z is positive, so the corner sits in front of the camera.

Project to pixels: u = 1200 × 500 / 1200 + 640 = 500 + 640 = 1140 px. v = 1200 × (−20) / 1200 + 512 = −20 + 512 = 492 px.

The stored pose therefore puts that corner at pixel (1140, 492). Repeating the three steps for every mesh vertex is exactly what a label-check overlay computes.

How ground-truth poses are produced

Public datasets label their frames four ways, and each sets the accuracy ceiling you can expect:

• Motion capture. Markers on the object and a tracked camera give both poses per frame; a drone-pose study builds its 6D ground truth exactly this way, from mocap-measured poses of drone and camera. After marker-to-model offsets are calibrated out, expect labels within a few millimetres and a few degrees.
• Robot kinematics. Fix the object to a calibrated flange, or move the camera on the robot, and every saved robot pose plus hand-eye calibration yields one label per frame. The capture discipline lives in hand-eye calibration dataset acquisition with saved robot poses, and the intrinsics to freeze first in camera intrinsics for hand-eye calibration: fix the pose first.
• Rendering. Synthetic pipelines drop textured objects into scenes with physics and write the exact pose at render time — labels are perfect by construction, and the sim-to-real gap is the price.
• Semi-automatic annotation. Keypoint clicks on the image, then a pose solve against the mesh; the intrinsics choice that governs accuracy is the subject of rectified image pose estimation: choosing PnP intrinsics.

For RGB-D sets captured with 3D sensors, the depth map gives a second route: backproject it with the matching intrinsics (aligned depth point cloud intrinsics: which camera to deproject with) and compare each label's t against the object's measured depth.

Recording your own labelled frames

Fine-tuning on your own parts means generating the annotation file yourself. Keeping the BOP field names costs nothing and lets public loaders, error metrics and visualisers run against your data unchanged. A rigid bench keeps every capture session comparable.

For the colour stream, keypoint annotation depends on resolving fine features. A 20 MP GigE colour camera covers that:

For depth-based methods, a projected structured-light 3D camera in the same bench supplies calibrated depth that each label's translation can be checked against.

Fix the camera to a slide-table stand so distance and angle repeat between sessions. A stand with a 700 mm column and a 400 × 450 mm base holds the geometry while parts move underneath.

Capture raw frames to disk and label offline rather than annotating live; 3D camera recording to H5 files for offline processing covers the capture side, and the camera configurator narrows sensor, resolution and interface choices against your targets. More reading on the subject sits on the 3D topic page.

Checks before a label reaches training

1. Reprojection overlay. Render the mesh with fx, fy, cx, cy, R and t over the image; landing on the object within a pixel or two confirms the whole chain at once. A constant shift points at cx, cy; a shift that grows with distance points at fx, fy.
2. Unit audit. BOP-style translations are millimetres. A solver working in metres is off by 1000×, and projection alone will not expose it, because u = fx·X/Z uses ratios only. Depth fusion and robot-frame math will.
3. Direction audit. Model-to-camera versus camera-to-model: the inverse needs Rᵀ(x_c − t), not a bare transpose. Symmetric parts project plausibly under the wrong direction, so audit on an asymmetric corner as in the worked example.
4. Count audit. One annotation list per image, one entry per physical instance, and a key for every frame — scenes with per-camera files are where counts drift.
5. Depth cross-check. For RGB-D frames, backprojected object depth within about 10 mm of t's Z component corroborates the label; a larger gap suggests a depth-scale or frame mismatch.

None of this needs exotic tooling: a JSON parser, the mesh, the camera matrix from scene_camera and one overlay render settle whether your labels are sound. If you are specifying the hardware side of a capture bench, the area scan cameras and camera stands pages filter by exactly those parameters — resolution, interface, stand height — or contact us with your part sizes and accuracy target and we will help you spec the rig.