You have a depth camera delivering frames, and you want the vendor viewer's live point cloud inside your own window. That is not a special streaming mode. It is the ordinary depth-to-points deprojection, run once per frame inside your viewer loop, with intrinsics held fixed and the previous cloud replaced rather than kept. The one-shot maths live in depth map to point cloud: the deprojection calculation explained; this guide makes them run every frame. You will set the intrinsics once, wire up the loop with Open3D, budget it for a 640 × 400 stream at 30 fps, and prove the frame rate with a timing test.
What real-time streaming actually is
Streaming adds no new maths. Every frame arrives as a 16-bit depth image: one distance per pixel, measured along the camera's optical axis. The pinhole model turns each pixel (u, v) with depth Z into a metric point:
fx and fy are focal lengths in pixels; cx and cy are the principal point; depth_scale converts raw counts to metres. All four come from the depth stream's intrinsics and hold constant while the stream runs, so the loop never recomputes them. What the loop owes you is throughput. A 640 × 400 frame carries 256,000 pixels, and at 30 fps all 256,000 points must be placed inside every 33 ms cycle. That is a fixed, countable workload, which is why the worked example below can budget it on paper before you write any code.
What you need
- A depth camera that streams a calibrated depth image: ToF, stereo and structured light all work. The 3D sensors category lists them by type.
- The stream's intrinsics (fx, fy, cx, cy), frame size and depth scale, read from the SDK at run time.
- A host for the loop: a desktop is fine to develop on, and a vision controller is the shop-floor version.
- Python with numpy and open3d installed.
- One PointCloud object, created once and reused for the whole stream.
A compact ToF sensing camera such as the MV-DT01WMU produces exactly the depth stream this loop consumes.
- Type
- Depth / ToF camera
Where you want a denser surface inside a fixed workspace, a projected structured-light camera such as the MV-DPS200P-02 feeds the same loop from a different depth principle.
- Type
- Stereo / structured light
Set the intrinsics once, outside the loop
Read fx, fy, cx, cy, the frame size and the depth scale from the device's depth stream profile when you open the camera, and never type them from memory. Two rules keep the geometry honest. First, intrinsics belong to a frame size: if you crop or resize the depth image, scale fx, fy, cx and cy by the same factors, the way point cloud after resizing RGB and depth images works through. Second, deproject the depth image with the depth camera's intrinsics, not the colour camera's; when RGB and depth arrive from separate sensors, aligned depth point cloud intrinsics settles which camera to deproject with. Change the stream's resolution and the device reports a new intrinsic set; pick up the new values and keep streaming.
The streaming loop, step by step
- Open the depth stream and read its width, height, fx, fy, cx, cy and depth scale.
- Build the intrinsics object once: PinholeCameraIntrinsic(width, height, fx, fy, cx, cy).
- Create one PointCloud, hand it to a Visualizer with add_geometry, and create the window once.
- Each cycle, grab the depth frame as a uint16 numpy array shaped (height, width).
- Deproject with create_from_depth_image, passing depth_scale in counts per metre and depth_trunc at your maximum working distance; leave project_valid_depth_only at its default so pixels with no return are dropped instead of pinned to the origin.
- Replace the cloud's points with the fresh cloud's points, never append, then call update_geometry, poll_events and update_renderer.
- Measure the loop; when it breaks the 25 ms working budget, pass stride=2 and halve each axis.
The loop, complete:
import numpy as np
import open3d as o3d
import time
# Read from the device's depth stream profile at run time.
# Values below are placeholders sized for a 640 x 400 depth mode.
image_width = 640 # int, depth frame width in pixels
image_height = 400 # int, depth frame height in pixels
fx = 460.0 # float, focal length in pixels, x axis
fy = 460.0 # float, focal length in pixels, y axis
cx = 319.5 # float, principal point, x axis
cy = 199.5 # float, principal point, y axis
depth_scale = 1000.0 # float, raw counts per metre: 1000.0 for 16-bit depth in mm
depth_trunc_m = 4.0 # float, discard points beyond this distance, in metres
def grab_depth_frame() -> np.ndarray:
# Replace with your SDK's capture call; it must return
# uint16 with shape (image_height, image_width).
return np.zeros((image_height, image_width), dtype=np.uint16)
intrinsic = o3d.camera.PinholeCameraIntrinsic(
image_width, image_height, fx, fy, cx, cy)
cloud = o3d.geometry.PointCloud() # one object, reused every frame
viewer = o3d.visualization.Visualizer()
viewer.create_window(window_name="depth point cloud stream", width=960, height=600)
viewer.add_geometry(cloud)
try:
while True:
depth_mm = grab_depth_frame()
fresh = o3d.geometry.PointCloud.create_from_depth_image(
o3d.geometry.Image(depth_mm),
intrinsic,
depth_scale=depth_scale,
depth_trunc=depth_trunc_m)
cloud.points = fresh.points # replace, never append
viewer.update_geometry(cloud)
viewer.poll_events()
viewer.update_renderer()
except KeyboardInterrupt:
viewer.destroy_window()
The view stays where you leave it between frames, because update_geometry refreshes the registered geometry rather than adding a new one.
Worked example: budgeting a 640 × 400 depth stream at 30 fps
Inputs, each with its source:
- Frame 640 × 400 pixels, 16-bit: the depth mode the loop above is sized for. The arithmetic follows from width, height and fps alone, so your own mode drops in the same way.
- 30 fps: the display rate to hold for a live look; it fixes the cycle at 1000 ms ÷ 30 = 33.3 ms.
- depth_scale 1000.0 counts per metre: the conversion for 16-bit depth in millimetres.
- float64 points: Open3D stores each point as 3 values × 8 bytes = 24 bytes.
The rule in words: points per frame equals width times height; bytes per second equals bytes per point, times points, times fps.
Arithmetic:
- Points: 640 × 400 = 256,000 per frame.
- Raw depth on the link: 256,000 × 2 B = 512,000 B ≈ 0.51 MB per frame; × 30 fps = 15.4 MB/s ≈ 123 Mbit/s. A Gigabit link moves about 125 MB/s, so one camera takes 15.36 ÷ 125 ≈ 12% of it.
- Cloud in host memory: 256,000 × 24 B = 6,144,000 B ≈ 6.1 MB per frame; × 30 fps = 184 MB/s ≈ 1.47 Gbit/s.
- Deprojection cost: 6 multiply-adds per point × 256,000 ≈ 1.5 million per frame, ≈ 46 million per second at 30 fps. Vectorized, that is sub-millisecond arithmetic; capture and drawing set the budget, not the maths.
Decisions:
- Five such cameras share one Gigabit link (5 × 15.4 ≈ 77 MB/s, about 61% utilization); the sixth camera gets its own port.
- The derived cloud never crosses a network: 184 MB/s exceeds any Gigabit link, so deproject on the machine that receives the depth frames and keep the cloud in that host's memory and GPU.
- Hold measured deprojection plus drawing under 25 ms of the 33.3 ms cycle. Above that, stride=2 gives 320 × 200 = 64,000 points and 64,000 × 24 B ≈ 1.5 MB per frame, a quarter of the load, at the cost of visible density.
- Recording the cloud at full rate writes about 11 GB per minute; record the 16-bit depth at 15.4 MB/s instead and deproject on replay.
Prove the loop holds 30 fps
Assume nothing; measure. Time 150 frames end to end and stage by stage:
n_frames = 150 # int, one 5 s test at 30 fps
t_capture = 0.0 # float, seconds spent capturing
t_deproject = 0.0 # float, seconds spent deprojecting
t_draw = 0.0 # float, seconds spent in viewer updates
t_start = time.perf_counter()
for _ in range(n_frames):
t0 = time.perf_counter()
depth_mm = grab_depth_frame()
t1 = time.perf_counter()
fresh = o3d.geometry.PointCloud.create_from_depth_image(
o3d.geometry.Image(depth_mm), intrinsic,
depth_scale=depth_scale, depth_trunc=depth_trunc_m)
t2 = time.perf_counter()
cloud.points = fresh.points
viewer.update_geometry(cloud)
viewer.poll_events()
viewer.update_renderer()
t3 = time.perf_counter()
t_capture += t1 - t0
t_deproject += t2 - t1
t_draw += t3 - t2
elapsed = time.perf_counter() - t_start # float, seconds
measured_fps = n_frames / elapsed # float
print(f"{measured_fps:.1f} fps | capture {1000 * t_capture / n_frames:.1f} ms | "
f"deproject {1000 * t_deproject / n_frames:.1f} ms | "
f"draw {1000 * t_draw / n_frames:.1f} ms")
assert measured_fps >= 28.0, "loop is slower than the camera: cut the workload"
A measured 28.0 fps or better passes: at 30 fps that tolerates losing about one frame in 17 while the display still reads as live. Read the three timings against the 33.3 ms cycle. Capture above 33 ms points at the camera or its link, not your loop; three depth cameras drop to low fps shows the isolation drill of testing one stream at a time. Draw above 25 ms points at the point count, and stride=2 is the lever.
Common mistakes that break the stream
- A new cloud and a new add_geometry every frame. The scene's geometry list grows without bound and the fps decays inside a minute. One cloud, points replaced, frame time flat; the same discipline as the per-frame image leak fix.
- A Python for-loop over pixels. An interpreted loop over 256,000 pixels costs seconds per frame; the vectorized create_from_depth_image call costs milliseconds. The per-pixel-loop article shows the pattern and the way out.
- depth_scale mismatched to the raw unit. With 16-bit millimetre depth left at scale 1.0, a part at 800 mm lands 800 m away and the cloud becomes a dot near the origin.
- depth_trunc tighter than the working distance. Everything past the cut vanishes mid-scene. The missing points article diagnoses the holes, and deprojection zero covers pixels that come back as zeros.
- Colour intrinsics applied to the depth image. Separate sensors have separate optics, so depth always deprojects with the depth intrinsics, the second rule in the intrinsics section above.
- Intrinsics carried across a crop or resize. Halve the frame and fx, fy, cx and cy halve with it; rescale before you deproject.
Sizing the host for one camera or six
The budget above sizes the host for you. One 640 × 400 camera at 30 fps asks for 15.4 MB/s on its link and a 6.1 MB cloud per frame in memory, so any current x86 host manages one or two cameras. Scale by camera count, not by frames: the link total is the per-camera rate times the camera count, and memory grows by one cloud per camera you keep resident.
For a single-camera bench, the MV-VC3301H-128G66 runs this loop on an Intel Core i3 with 8 GB of memory and includes the VM6200Pro license. For a cell with several depth cameras, the MV-VC5525-256G70-NN pairs an Intel Core i5 with 32 GB of memory, PoE camera ports and 5+ LAN ports, so each camera keeps its own link.
Both sit in the shop's vision controllers range:
- LAN ports
- 5+ LAN ports
- USB 3 ports
- 3-4 USB 3 ports
- LAN ports
- 5+ LAN ports
- USB 3 ports
- 3-4 USB 3 ports
Checks before you go live
- The intrinsics print once at start-up and match the stream's width and height.
- depth_scale matches the SDK's raw unit: 1000.0 for 16-bit millimetre depth.
- depth_trunc sits at your maximum working distance, not at the sensor's maximum range.
- Measured fps stays at 28 or better across a ten-minute run, not a ten-second one.
- Host memory is flat over that run, the proof that the cloud is replaced rather than accumulated.
- A one-minute recording of raw depth, about 0.9 GB in this mode, replays and deprojects offline.
Where to go next
Pick the camera, then the host, then write the loop. The camera finder filters 3D sensors by type and range, and the system configurator starts the build at the camera step. For more on turning depth frames into geometry, the 3D topic page collects the guides on this site, and the contact page reaches the applications team for a layout review of your frame size and camera count.
Laser profilers and stereo 3D cameras
170 models · from $699.00Questions
Can I colour the streaming point cloud from the RGB image?
Yes, if the RGB frame is registered to the depth frame's optical frame. Deproject with the depth intrinsics, then colour each point by reading the same u, v pixel from the registered RGB image. If colour and depth come from separate sensors, pair the frames by sequence number before merging; colour commonly lags depth by one frame, and mismatched pairs smear the colours across the surface.
How do I keep the viewer responsive while the camera delivers frames?
Run capture in a producer thread that pushes depth frames into a queue with a maximum size of one or two, and let the viewer loop take only the newest frame. Display never needs every frame, so dropping stale ones is correct behaviour. Keep the GUI calls and the camera calls out of each other's threads and the window stays responsive.
Why does my streamed cloud come out flat, mirrored or upside down?
The deprojection is correct but the axes are not. Depth cameras differ in which axis points forward, up and right, so apply one fixed permutation or sign flip to the cloud, chosen so a box set in front of the camera appears in front in the viewer. If the whole cloud collapses into a plane, a single-channel image is being read as three.
Should I save every frame of the stream to disk?
No. At 256,000 points per frame in double precision you write about 6.1 MB per frame, roughly 184 MB/s, or about 11 GB per minute at 30 fps. Record a preview frame every 30th capture, or record the raw 16-bit depth instead at about 15.4 MB/s and deproject again offline. Depth images are the compact source of truth.
Does the same loop work for a laser profile sensor?
Not directly. A laser profiler delivers an already calibrated profile cloud per scan instead of a depth image to deproject, so streaming means plotting each new profile against its encoder position and, for a volume, stitching profiles as the part or sensor moves. The display pattern is the same, one reusable cloud that gets replaced and redrawn, but the per-frame geometry arrives ready made.



