Model Camera: Real-Time Modeling from Dense Color and Sparse Depth

Summary
The ModelCamera is a fast, easy to use, and inexpensive 3D scene modeling system. The ModelCamera acquires dense color (720×480 video frames) augmented with sparse depth (7×7 to 11×11 depth samples). The frames are registered and merged into an evolving model at the rate of five frames per second. The model is displayed continually to provide immediate operator feedback.

Structured Scene Modeling
Structured scenes (e.g. large pieces of furniture, walls, ceiling, and floor) are acquired freehand. The operator holds the ModelCamera in his/her hand and sweeps the scene. The frames are registered using the depth and color data. Registered frames are merged into a set of depth images created on demand.
Freehand ModelCamera prototype consists of a video camera and a laser source whose beam is split in a 7×7 pattern with a diffraction grating. Total cost $2,000.	Real-time freehand modeling with operator feedback.	Depth image placement shown with green dots (left) and depth image modeling (right).

Unstructured Scene Modeling
Unstructured scenes are acquired using a parallax-free pan-tilt bracket. Frames are stitched together using color to form a cube map. The depth samples are projected onto the cube map and triangulated in 2D. The connectivity information is applied to the depth samples to form a depth enhanced panorama (DEP), which is a texture-mapped 3D triangle mesh. A DEP supports view point translation while it maintains the advantages of color panoramas such as fast acquisition, inexpensive acquisition device, and interactive photorealistic rendering of the acquired scene. Several DEPs are combined on the GPU for better scene coverage.
Parallax-free pan-tilt bracket.	Real-time parallax free modeling and operator feedback.	Model of plant and room acquired in 5 and 15 minutes, respectively.

Large-Scale Modeling
We have extended the ModelCamera system with shaft encoders, a more powerful laser and a custom 11×11 diffraction grating to model large indoor environments. A team of two operators and a single acquisition device acquired in 40 hours a model spanning 1,400 m2 of floor space. The model covers corridors and 20 individual rooms over six floors of a large building on our campus. The model is assembled from room and I, L, and T corridor sections. A section is modeled by fitting proxy geometry to one or two DEPs. Corridor sections have high geometric detail where needed. Relying on the immediate feedback, the operator avoids over-sampling simple geometry. The resulting model supports realistic walkthroughs of the large indoor environment.
Large-scale indoor model assembled from room (blue) and corridor sections (orange).	I, L, and T corridor sections.	Corridor sections have embedded geometric detail where needed (water fountain here).

Models
	Math 409 Office 220 519 points ~420 000 triangles Capture time: < 15 minutes VRML size: 9 MB For comparison: the same model without any geometry - the cubic color panorama only. Notice the effect of translation away from original point of capture.
	Math 409 after carpet installation VRML size: 15 MB
	Rose 10951 points ~21 700 triangles Capture time: < 4 minutes VRML size: 0.5 MB
	Flower on the table 30 568 points ~59 000 triangles Capture time: < 5 minutes VRML size: 1.3 MB
	CS 146 169 083 triangles Capture time: 20 minutes VRML size: 4.11 MB

Publications

People