6D pose dataset format: read BOP scene annotations in your script

A pose-estimation project rarely starts from a blank page. BOP-format datasets ship thousands of annotated scenes with color, depth, masks and ground-truth 6D poses, ready for training or evaluating a model on your own parts. The annotations live in a handful of JSON files whose conventions are easy to misread, and the benchmark's starter toolkit does not wire itself into your code. This guide walks each file in a scene folder, states the units and reference frames exactly, and builds a thirty-line loader that checks itself before anything downstream consumes a pose. You finish able to read obj_id, cam_R_w2c and cam_t_w2c for any instance, project it into the image, and prove the parse against the depth map.

The five files that hold a scene

A BOP dataset splits into image sets such as test and train, each holding numbered scene folders (000001, 000002, and so on). Every scene folder has the same layout: four JSON files and four image folders. File names inside are zero-padded to six digits, so frame 17 on disk is rgb/000017.png and depth/000017.png. Fix the layout once and one loader reads any scene of any set.

The shared layout is why so many 6D pipelines can exchange data: benchmarks built on the format cover bin picking, industrial parts and household objects, and the same reader ingests scenes you capture yourself.

scene_gt.json: one row per object instance

scene_gt.json is a single JSON object for the whole scene. Its keys are image ids as strings ('0', '1', up to the last frame), and each value is a list with one entry per annotated instance. An entry carries three fields:

• obj_id: integer, an index into the dataset's models folder.
• cam_R_w2c: nine floats, a 3×3 rotation matrix flattened row-major, so the first three numbers are the matrix's first row.
• cam_t_w2c: three floats, the object's model-frame origin expressed in the camera frame, in millimeters.

R is cam_R_w2c and t is cam_t_w2c; x_model in millimeters gives x_cam in millimeters, in the camera frame. Where those per-image labels come from in the first place is its own subject — the ground-truth provenance guide covers it.

Intrinsics and depth scale: camera.json versus scene_camera.json

Pixel math needs the camera matrix, and the format stores it in one of two places. Newer datasets ship a camera.json per image set with cam_K (nine floats, row-major, reading [fx, 0, cx, 0, fy, cy, 0, 0, 1]) and a depth_scale shared by the set; some converted sets drop a copy into each scene folder, so have the loader look in the scene folder first, then the set folder. Older datasets repeat cam_K inside every scene_camera.json entry instead, alongside per-image cam_R_w2c, cam_t_w2c and depth_scale. The loader rule is one line: use a per-image cam_K when the entries carry it, otherwise load camera.json once.

Depth maps are 16-bit PNGs whose pixel values are not millimeters until scaled:

depth_scale is part of the data, so the loader reads it exactly as it reads fx. Capture with your own depth camera and the same conventions apply: the intrinsics and distortion explainer shows how to read them rather than guess, and exporting intrinsics to JSON keeps your own captures in the same shape.

scene_gt_info.json and the mask folders

A pose alone does not say what the annotator could actually see. scene_gt_info.json mirrors scene_gt.json row for row and adds visibility: bbox_obj is the [x, y, w, h] pixel box of the full silhouette, bbox_visib the box of the visible part only, px_count_all and px_count_visible count those silhouette pixels, and visib_fract is their ratio. px_count_valid counts pixels that also carry valid depth.

Next to the images sit mask/ with the full silhouette of each instance and mask_visib/ with the visible part, one PNG per instance. Treat the boxes as the cheapest sanity check on a parsed pose — and when annotated objects fall outside the frame entirely, that is a dataset-scope problem rather than a parse bug; the annotation mismatch fix walks that failure mode.

A loader in thirty lines

The files are JSON and the matrices are slices, so the standard library is enough. Variable names below are illustrative; the field names are the format's. The benchmark's companion Python toolkit (bop_toolkit) wraps the same files with its own helpers if you would rather not own the reader.

import json

# Illustrative names and values; field names are the format's.
scene_dir = 'datasets/myset/test/000006'   # str, one scene folder
image_id  = 17                              # int, frame to load

with open(scene_dir + '/scene_gt.json') as f:
    gt   = json.load(f)                     # dict[str, list[dict]]
with open(scene_dir + '/scene_gt_info.json') as f:
    info = json.load(f)                     # dict[str, list[dict]], row-aligned
with open(scene_dir + '/camera.json') as f:
    cam  = json.load(f)                     # dict, cam_K and depth_scale

K = cam['cam_K']                            # list[float], 9 values, row-major
fx, fy, cx, cy = K[0], K[4], K[2], K[5]     # floats, pixels

poses = gt[str(image_id)]                   # list[dict], one per instance
boxes = info[str(image_id)]                 # list[dict], same order as poses

for row, inst in enumerate(poses):
    R = [inst['cam_R_w2c'][0:3],            # list[list[float]], 3x3 row-major
         inst['cam_R_w2c'][3:6],
         inst['cam_R_w2c'][6:9]]
    t = inst['cam_t_w2c']                   # list[float], [X, Y, Z] in mm
    x, y, w, h = boxes[row]['bbox_visib']   # ints, pixels
    u = fx * t[0] / t[2] + cx               # float, px
    v = fy * t[1] / t[2] + cy               # float, px
    assert x <= u <= x + w and y <= v <= y + h, (
        'instance %d projects to (%.1f, %.1f) px, outside bbox_visib' % (row, u, v))

The assert is the whole test. Project each object origin through cam_K and require the pixel to land inside that instance's bbox_visib; a parse that swaps row- for column-major, or grabs the wrong row's box, fails this line at once.

Worked example: project the origin, check the depth

Round numbers chosen so the arithmetic is easy to check; a real dataset fills the same fields with its own values. The image set gives cam_K = [1000.0, 0.0, 320.0, 0.0, 1000.0, 240.0, 0.0, 0.0, 1.0] for a 640 × 480 image and depth_scale = 0.1. Frame '17' carries one instance: obj_id 2, cam_R_w2c the identity, cam_t_w2c = [-120.0, 60.0, 900.0] millimeters, bbox_visib = [160, 270, 120, 110] pixels, px_count_all = 29700, px_count_visible = 18414.

1. u = 1000.0 × (-120.0 / 900.0) + 320.0 = -133.33 + 320.0 = 186.67 px.
2. v = 1000.0 × (60.0 / 900.0) + 240.0 = 66.67 + 240.0 = 306.67 px.
3. Box check: 160 ≤ 186.67 ≤ 280 and 270 ≤ 306.67 ≤ 380, so the origin lands inside bbox_visib and the assert passes.
4. Visibility: 18414 / 29700 = 0.62, equal to the stored visib_fract.
5. Depth: the pixel at (187, 307) reads 9000, and 9000 × 0.1 = 900.0 mm, matching t_z = 900.0 mm.
6. Back-projection with the rounded pixel: X = 900.0 × (187 − 320) / 1000.0 = −119.7 mm and Y = 900.0 × (307 − 240) / 1000.0 = 60.3 mm — within 0.3 mm of the annotation, which is one pixel of rounding at a 0.9 m depth.

From rotation matrix to robot pose

Reshape the nine floats row by row, then validate before use: a rotation matrix satisfies RᵀR = I and det(R) = 1. Example: cam_R_w2c = [0.0, -1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0] is a 90° rotation about +Z, determinant 1, and its axis-angle form is [0.0, 0.0, 1.5708] radians. Most pose libraries take either form, and OpenCV's cv2.Rodrigues converts between matrix and vector; the rotation vector and tvec units explainer works the conversion both ways, and the Euler order guide matters the moment a controller wants angles instead of a matrix.

Units close the gap to the robot: divide t by 1000 exactly once, where the pose leaves the dataset and enters a meter-based frame.

The hand-eye calibration unit mismatch case is that failure seen in the field.

Four parse mistakes worth five minutes

1. Sorting image keys as strings. sorted(gt.keys()) puts '10' before '2'; sort with key=int or iterate a numeric range.
2. Reading cam_R_w2c column-major. The nine floats are row-major; a transpose mirrors the pose, and the projection assert above is what catches it.
3. Confusing the boxes. Occlusion-aware training wants mask_visib/ and bbox_visib; full-extent crops and amodal evaluation want mask/ and bbox_obj.
4. Hard-coding depth_scale = 1.0. A set with 0.1 then over-states every depth tenfold, and plane fits and point clouds stretch with it.

Capture scenes in this format yourself

Benchmarks rarely contain your part, and the format pays off because you can produce it: capture aligned color and depth with known intrinsics, record each set's depth_scale, write poses into the four JSON files, and every reader, augmenter and metric you just built keeps working. For self-captured scenes the head of the rig is a 3D camera: a projected structured-light model such as the MV-DPS200P-02 supplies the dense depth channel, and your station's color camera and intrinsics complete the scene files.

The rig and the training runs want compute that stays up all shift. A compact vision controller such as the MV-VC2140-128G40-1T carries 2 USB 3 ports, 8 GB of memory and a 1 TB HDD. At roughly 1 MB per annotated 640 × 480 frame with color, depth and masks, that disk holds on the order of a million frames — a season of captures before archiving. Pair a depth head with it through the 3D sensors or the camera finder.

Where to go next

Load one scene, run the projection assert, then compare depth at the projected origin with t_z. Three green checks mean the annotations are trustworthy in your pipeline; from there, train or evaluate, and calibrate the camera to the robot before the first guided pick. The 3D topic page collects the surrounding guides, and the system configurator pairs a depth camera with the controller and lighting for your station.