3D visionHow-to

Real-time point cloud streaming from a depth camera in a viewer loop

Real-time point cloud streaming from a depth camera: set the intrinsics once, replace the cloud each frame, and hold 30 fps in a viewer loop.

LumaSpec Vision10 min read Share on LinkedIn

On this page 11 sections
  1. 01What real-time streaming actually is
  2. 02What you need
  3. 03Set the intrinsics once, outside the loop
  4. 04The streaming loop, step by step
  5. 05Worked example: budgeting a 640 × 400 depth stream at 30 fps
  6. 06Prove the loop holds 30 fps
  7. 07Common mistakes that break the stream
  8. 08Sizing the host for one camera or six
  9. 09Checks before you go live
  10. 10Where to go next
  11. 11Questions

You have a depth camera delivering frames, and you want the vendor viewer's live point cloud inside your own window. That is not a special streaming mode. It is the ordinary depth-to-points deprojection, run once per frame inside your viewer loop, with intrinsics held fixed and the previous cloud replaced rather than kept. The one-shot maths live in depth map to point cloud: the deprojection calculation explained; this guide makes them run every frame. You will set the intrinsics once, wire up the loop with Open3D, budget it for a 640 × 400 stream at 30 fps, and prove the frame rate with a timing test.

What real-time streaming actually is

Streaming adds no new maths. Every frame arrives as a 16-bit depth image: one distance per pixel, measured along the camera's optical axis. The pinhole model turns each pixel (u, v) with depth Z into a metric point:

X = (u - cx) * Z / fx Y = (v - cy) * Z / fy Z = raw depth / depth_scale

fx and fy are focal lengths in pixels; cx and cy are the principal point; depth_scale converts raw counts to metres. All four come from the depth stream's intrinsics and hold constant while the stream runs, so the loop never recomputes them. What the loop owes you is throughput. A 640 × 400 frame carries 256,000 pixels, and at 30 fps all 256,000 points must be placed inside every 33 ms cycle. That is a fixed, countable workload, which is why the worked example below can budget it on paper before you write any code.

What you need

  • A depth camera that streams a calibrated depth image: ToF, stereo and structured light all work. The 3D sensors category lists them by type.
  • The stream's intrinsics (fx, fy, cx, cy), frame size and depth scale, read from the SDK at run time.
  • A host for the loop: a desktop is fine to develop on, and a vision controller is the shop-floor version.
  • Python with numpy and open3d installed.
  • One PointCloud object, created once and reused for the whole stream.

A compact ToF sensing camera such as the MV-DT01WMU produces exactly the depth stream this loop consumes.

3D sensorsHikrobot MV-DT01WMU ToF sensing cameraMV-DT01WMU
Type
Depth / ToF camera
Price on requestView and quote

Where you want a denser surface inside a fixed workspace, a projected structured-light camera such as the MV-DPS200P-02 feeds the same loop from a different depth principle.

3D sensorsHikrobot MV-DPS200P-02 Projected structured-light 3D cameraMV-DPS200P-02
Type
Stereo / structured light
$8,829.00 · quotedView and quote

Set the intrinsics once, outside the loop

Read fx, fy, cx, cy, the frame size and the depth scale from the device's depth stream profile when you open the camera, and never type them from memory. Two rules keep the geometry honest. First, intrinsics belong to a frame size: if you crop or resize the depth image, scale fx, fy, cx and cy by the same factors, the way point cloud after resizing RGB and depth images works through. Second, deproject the depth image with the depth camera's intrinsics, not the colour camera's; when RGB and depth arrive from separate sensors, aligned depth point cloud intrinsics settles which camera to deproject with. Change the stream's resolution and the device reports a new intrinsic set; pick up the new values and keep streaming.

The streaming loop, step by step

  1. Open the depth stream and read its width, height, fx, fy, cx, cy and depth scale.
  2. Build the intrinsics object once: PinholeCameraIntrinsic(width, height, fx, fy, cx, cy).
  3. Create one PointCloud, hand it to a Visualizer with add_geometry, and create the window once.
  4. Each cycle, grab the depth frame as a uint16 numpy array shaped (height, width).
  5. Deproject with create_from_depth_image, passing depth_scale in counts per metre and depth_trunc at your maximum working distance; leave project_valid_depth_only at its default so pixels with no return are dropped instead of pinned to the origin.
  6. Replace the cloud's points with the fresh cloud's points, never append, then call update_geometry, poll_events and update_renderer.
  7. Measure the loop; when it breaks the 25 ms working budget, pass stride=2 and halve each axis.

The loop, complete:

import numpy as np
import open3d as o3d
import time

# Read from the device's depth stream profile at run time.
# Values below are placeholders sized for a 640 x 400 depth mode.
image_width   = 640       # int,   depth frame width in pixels
image_height  = 400       # int,   depth frame height in pixels
fx = 460.0                # float, focal length in pixels, x axis
fy = 460.0                # float, focal length in pixels, y axis
cx = 319.5                # float, principal point, x axis
cy = 199.5                # float, principal point, y axis
depth_scale   = 1000.0    # float, raw counts per metre: 1000.0 for 16-bit depth in mm
depth_trunc_m = 4.0       # float, discard points beyond this distance, in metres

def grab_depth_frame() -> np.ndarray:
    # Replace with your SDK's capture call; it must return
    # uint16 with shape (image_height, image_width).
    return np.zeros((image_height, image_width), dtype=np.uint16)

intrinsic = o3d.camera.PinholeCameraIntrinsic(
    image_width, image_height, fx, fy, cx, cy)

cloud = o3d.geometry.PointCloud()        # one object, reused every frame

viewer = o3d.visualization.Visualizer()
viewer.create_window(window_name="depth point cloud stream", width=960, height=600)
viewer.add_geometry(cloud)

try:
    while True:
        depth_mm = grab_depth_frame()
        fresh = o3d.geometry.PointCloud.create_from_depth_image(
            o3d.geometry.Image(depth_mm),
            intrinsic,
            depth_scale=depth_scale,
            depth_trunc=depth_trunc_m)
        cloud.points = fresh.points      # replace, never append
        viewer.update_geometry(cloud)
        viewer.poll_events()
        viewer.update_renderer()
except KeyboardInterrupt:
    viewer.destroy_window()

The view stays where you leave it between frames, because update_geometry refreshes the registered geometry rather than adding a new one.

Worked example: budgeting a 640 × 400 depth stream at 30 fps

Inputs, each with its source:

  • Frame 640 × 400 pixels, 16-bit: the depth mode the loop above is sized for. The arithmetic follows from width, height and fps alone, so your own mode drops in the same way.
  • 30 fps: the display rate to hold for a live look; it fixes the cycle at 1000 ms ÷ 30 = 33.3 ms.
  • depth_scale 1000.0 counts per metre: the conversion for 16-bit depth in millimetres.
  • float64 points: Open3D stores each point as 3 values × 8 bytes = 24 bytes.

The rule in words: points per frame equals width times height; bytes per second equals bytes per point, times points, times fps.

Arithmetic:

  • Points: 640 × 400 = 256,000 per frame.
  • Raw depth on the link: 256,000 × 2 B = 512,000 B ≈ 0.51 MB per frame; × 30 fps = 15.4 MB/s ≈ 123 Mbit/s. A Gigabit link moves about 125 MB/s, so one camera takes 15.36 ÷ 125 ≈ 12% of it.
  • Cloud in host memory: 256,000 × 24 B = 6,144,000 B ≈ 6.1 MB per frame; × 30 fps = 184 MB/s ≈ 1.47 Gbit/s.
  • Deprojection cost: 6 multiply-adds per point × 256,000 ≈ 1.5 million per frame, ≈ 46 million per second at 30 fps. Vectorized, that is sub-millisecond arithmetic; capture and drawing set the budget, not the maths.

Decisions:

  • Five such cameras share one Gigabit link (5 × 15.4 ≈ 77 MB/s, about 61% utilization); the sixth camera gets its own port.
  • The derived cloud never crosses a network: 184 MB/s exceeds any Gigabit link, so deproject on the machine that receives the depth frames and keep the cloud in that host's memory and GPU.
  • Hold measured deprojection plus drawing under 25 ms of the 33.3 ms cycle. Above that, stride=2 gives 320 × 200 = 64,000 points and 64,000 × 24 B ≈ 1.5 MB per frame, a quarter of the load, at the cost of visible density.
  • Recording the cloud at full rate writes about 11 GB per minute; record the 16-bit depth at 15.4 MB/s instead and deproject on replay.

Prove the loop holds 30 fps

Assume nothing; measure. Time 150 frames end to end and stage by stage:

n_frames    = 150    # int,   one 5 s test at 30 fps
t_capture   = 0.0    # float, seconds spent capturing
t_deproject = 0.0    # float, seconds spent deprojecting
t_draw      = 0.0    # float, seconds spent in viewer updates

t_start = time.perf_counter()
for _ in range(n_frames):
    t0 = time.perf_counter()
    depth_mm = grab_depth_frame()
    t1 = time.perf_counter()
    fresh = o3d.geometry.PointCloud.create_from_depth_image(
        o3d.geometry.Image(depth_mm), intrinsic,
        depth_scale=depth_scale, depth_trunc=depth_trunc_m)
    t2 = time.perf_counter()
    cloud.points = fresh.points
    viewer.update_geometry(cloud)
    viewer.poll_events()
    viewer.update_renderer()
    t3 = time.perf_counter()
    t_capture += t1 - t0
    t_deproject += t2 - t1
    t_draw += t3 - t2

elapsed = time.perf_counter() - t_start   # float, seconds
measured_fps = n_frames / elapsed         # float
print(f"{measured_fps:.1f} fps | capture {1000 * t_capture / n_frames:.1f} ms | "
      f"deproject {1000 * t_deproject / n_frames:.1f} ms | "
      f"draw {1000 * t_draw / n_frames:.1f} ms")
assert measured_fps >= 28.0, "loop is slower than the camera: cut the workload"

A measured 28.0 fps or better passes: at 30 fps that tolerates losing about one frame in 17 while the display still reads as live. Read the three timings against the 33.3 ms cycle. Capture above 33 ms points at the camera or its link, not your loop; three depth cameras drop to low fps shows the isolation drill of testing one stream at a time. Draw above 25 ms points at the point count, and stride=2 is the lever.

Common mistakes that break the stream

  • A new cloud and a new add_geometry every frame. The scene's geometry list grows without bound and the fps decays inside a minute. One cloud, points replaced, frame time flat; the same discipline as the per-frame image leak fix.
  • A Python for-loop over pixels. An interpreted loop over 256,000 pixels costs seconds per frame; the vectorized create_from_depth_image call costs milliseconds. The per-pixel-loop article shows the pattern and the way out.
  • depth_scale mismatched to the raw unit. With 16-bit millimetre depth left at scale 1.0, a part at 800 mm lands 800 m away and the cloud becomes a dot near the origin.
  • depth_trunc tighter than the working distance. Everything past the cut vanishes mid-scene. The missing points article diagnoses the holes, and deprojection zero covers pixels that come back as zeros.
  • Colour intrinsics applied to the depth image. Separate sensors have separate optics, so depth always deprojects with the depth intrinsics, the second rule in the intrinsics section above.
  • Intrinsics carried across a crop or resize. Halve the frame and fx, fy, cx and cy halve with it; rescale before you deproject.

Sizing the host for one camera or six

The budget above sizes the host for you. One 640 × 400 camera at 30 fps asks for 15.4 MB/s on its link and a 6.1 MB cloud per frame in memory, so any current x86 host manages one or two cameras. Scale by camera count, not by frames: the link total is the per-camera rate times the camera count, and memory grows by one cloud per camera you keep resident.

For a single-camera bench, the MV-VC3301H-128G66 runs this loop on an Intel Core i3 with 8 GB of memory and includes the VM6200Pro license. For a cell with several depth cameras, the MV-VC5525-256G70-NN pairs an Intel Core i5 with 32 GB of memory, PoE camera ports and 5+ LAN ports, so each camera keeps its own link.

Both sit in the shop's vision controllers range:

Vision controllersHikrobot MV-VC5525-256G70-NN VC5000 Vision Controller, 12th-I5, 32GB+256G SSD, Supported PCIEMV-VC5525-256G70-NN
LAN ports
5+ LAN ports
USB 3 ports
3-4 USB 3 ports
$4,479.00 · quotedView and quote
Vision controllersHikrobot MV-VC3301H-128G66 I3-8100, 8GB+128G SSD, No PoE, PCIe expansion supported, Incl. VM6200Pro licenseMV-VC3301H-128G66
LAN ports
5+ LAN ports
USB 3 ports
3-4 USB 3 ports
$2,419.00View part

Checks before you go live

  • The intrinsics print once at start-up and match the stream's width and height.
  • depth_scale matches the SDK's raw unit: 1000.0 for 16-bit millimetre depth.
  • depth_trunc sits at your maximum working distance, not at the sensor's maximum range.
  • Measured fps stays at 28 or better across a ten-minute run, not a ten-second one.
  • Host memory is flat over that run, the proof that the cloud is replaced rather than accumulated.
  • A one-minute recording of raw depth, about 0.9 GB in this mode, replays and deprojects offline.

Where to go next

Pick the camera, then the host, then write the loop. The camera finder filters 3D sensors by type and range, and the system configurator starts the build at the camera step. For more on turning depth frames into geometry, the 3D topic page collects the guides on this site, and the contact page reaches the applications team for a layout review of your frame size and camera count.

Cameras3D sensors

Laser profilers and stereo 3D cameras

170 models · from $699.00

Questions

Can I colour the streaming point cloud from the RGB image?

Yes, if the RGB frame is registered to the depth frame's optical frame. Deproject with the depth intrinsics, then colour each point by reading the same u, v pixel from the registered RGB image. If colour and depth come from separate sensors, pair the frames by sequence number before merging; colour commonly lags depth by one frame, and mismatched pairs smear the colours across the surface.

How do I keep the viewer responsive while the camera delivers frames?

Run capture in a producer thread that pushes depth frames into a queue with a maximum size of one or two, and let the viewer loop take only the newest frame. Display never needs every frame, so dropping stale ones is correct behaviour. Keep the GUI calls and the camera calls out of each other's threads and the window stays responsive.

Why does my streamed cloud come out flat, mirrored or upside down?

The deprojection is correct but the axes are not. Depth cameras differ in which axis points forward, up and right, so apply one fixed permutation or sign flip to the cloud, chosen so a box set in front of the camera appears in front in the viewer. If the whole cloud collapses into a plane, a single-channel image is being read as three.

Should I save every frame of the stream to disk?

No. At 256,000 points per frame in double precision you write about 6.1 MB per frame, roughly 184 MB/s, or about 11 GB per minute at 30 fps. Record a preview frame every 30th capture, or record the raw 16-bit depth instead at about 15.4 MB/s and deproject again offline. Depth images are the compact source of truth.

Does the same loop work for a laser profile sensor?

Not directly. A laser profiler delivers an already calibrated profile cloud per scan instead of a depth image to deproject, so streaming means plotting each new profile against its encoder position and, for a volume, stitching profiles as the part or sensor moves. The display pattern is the same, one reusable cloud that gets replaced and redrawn, but the per-frame geometry arrives ready made.