Rapid 360 degree imaging and stitching of 3D objects ...

Rapid 360 degree imaging and stitching of 3D objects using multiple precision 3D cameras Thomas Lu1, Stuart Yin2, Jianzhong Zhang3, Jiangan Li4, and Frank Wu4 1

JPL/CalTech, Pasadena, CA USA, 2The Pennsylvania State University, Unversity Park, PA,3 Trellis Bioscience, 2B Corporate Drive, South San Francisco, CA 94080, 4The University of Northern California, Petaluma, CA,

ABSTRACT In this paper, we present the system architecture of a 360 degree view 3D imaging system. The system consists of multiple 3D sensors synchronized to take 3D images around the object. Each 3D camera employs a single high-resolution digital camera and a color-coded light projector. The cameras are synchronized to rapidly capture the 3D and color information of a static object or a live person. The colorencoded structure lighting ensures the precise reconstruction of the depth of the object. A 3D imaging system architecture is presented. The architecture employs the displacement of the camera and the projector to triangulate the depth information. The 3D camera system has achieved high depth resolution down to 0.1mm on a human head sized object and 360 degree imaging capability. Keywords: 3D imaging, digital camera, high resolution, 360 degree, precision 3D model.

1. INTRODUCTION 3D scanning technology has been developed using laser scanning1, stereoscopic imaging2-4, phase shifting5, 6 , Moire grating7 and grating projection8-10 techniques. We have developed a high precision model stitching algorithms to automatically merge multiple pieces of 3D models into a single 360 degree 3D model11. The relative position and orientation of the cameras is calibrated using a polygon calibration object. The same polygon object is imaged by all cameras and individual 3D models are generated. The calibration models are aligned to calculate the transformation matrix between cameras. Those calibration results are saved in multi-camera system setting file. When the 360 degree camera system is in operation, it roughly aligns the 3D models generated by individual camera using the calibration file. For the overlap region, one camera may generate better 3D view than other camera. The program removes bad data from the 3D models. The program then automatically aligns the models to reduce the gap between two models. The vertex in the overlap region is adjusted to produce smooth blending. The duplicate triangles are removed from the overlap region. Then extra triangles are added to connect the two models. The mapping bitmap is stitched separated from 3D spatial data. For the simple object, such as human face, the mapping bitmaps from different view are projected cylindrically. Then the bitmap from overlap region is blended. Then the color mapping of stitched model is changed into cylindrical color mapping. The critical region, such as human eye is marked to avoid any blend to get vivid stitched image. The system generates a complete precision 3D model with high fidelity color map with minimum manual operation.

Three-Dimensional Image Capture and Applications 2008, edited by Brian D. Corner, Masaaki Mochimaru, Robert Sitnik Proc. of SPIE-IS&T Electronic Imaging, SPIE Vol. 6805, 680506, © 2008 SPIE-IS&T · 0277-786X/08/$18

SPIE-IS&T/ Vol. 6805 680506-1 2008 SPIE Digital Library -- Subscriber Archive Copy

2. MULTI-CAMERA 3D SCANNER SYSTEM The system architecture of the 3D camera is shown in Figure 1. A high resolution digital camera is used as the imaging sensor. A projector consists of a light source, a collimating lens, a grating, and an imaging lens used to project a precision color grating onto the object. The light source is usually a photographic flash. The grating structure needs to be projected onto the object precisely and the digital camera can resolve the fine structure of the color grating structure. The digital camera takes a shot of the object with the grating structure superimposed on the object. The image is used to calculate the 3D structure of the object11. A regular photographic flash is paced in between the digital camera and the projector. It is used to illuminate the object as the digital camera takes a second shot in a burst mode for a regular color photograph of the object as a color map of the 3D model. The two shots need to be as close as possible so that the object would not have movement between shots. The 3D model is formed based on only one shot, i.e., the first image. The second shot is only for the color map. It does not affect the 3D model calculation.

Object

1 meter

(b)

Imaging lens Flash

Grating Collimation lens

Digital Camera

Light Source Control Electronic

(a) (c) Figure 1: (a) System architecture of a 3D camera; (b) a single shot produces a high precision 3D model; (c) the 2 consecutive shots give a high fidelity 3D model with color map. An electronic control board synchronizes the digital camera, the projector and the flash to take a single shot for 3D model without color mapping, or two consecutive shots for a complete 3D model with high fidelity color map. One shot from the 3D camera can generate a 3D model of an object in about 90 degree angles. If the camera can not see the grating projection reflected from the object, then it would not be able to generate the

SPIE-IS&T/ Vol. 6805 680506-2

3D mesh. In order to take full 360 degrees of the object, 5 3D cameras are synchronized to shoot the 3D images of the object in almost simultaneous timing. The systematic diagram of the 360 degree 3D imaging system is shown in Figure 2. The 5 3D models has to overlap in order to be stitched together later in the computer. In Figure 2, the 3D cameras 1, 2, and 3 are placed closer than 3D cameras 4 and 5 so that the face of a person (Face down) can be imaged with more details. Figure 3 shows 360 degree views of a 3D model of a live person.

3D Camera 5

3D Camera 4

Object

Control Electronics

3D Camera 1 3D Camera 3

3D Camera 2 Figure 2: Systematic diagram of a 360 degree 3D imaging system.

:I

-

(a)

(b)

I

(c)

Figure 3 (a), (b) and (c): Views of a 3D model of a live person taken by the 360 degree imaging system.

3. AUTOMATIC 3D MODEL STITCHING Manual stitching of 3D models is time consuming. In order to facilitate the automatic stitching and merging of the 3D models of different views of an object and its color maps, we need to calibrate the

SPIE-IS&T/ Vol. 6805 680506-3

multiple 3D cameras. A calibration object is made, as shown in Figure 4. It is a cylindrical shaped object with many flat facets. It is designed so that each camera can see at least 2 adjacent facets, and any two adjacent 3D cameras can see common facets. The same calibration object is imaged by all the cameras simultaneously. Multi-3D models are generated by individual cameras. For every two camera pair, there are at least 3 matching vertices in 2 3d models. User manually selects 3 matching vertices. The software uses those 3 matching vertices to roughly align the 2 3D models. The program then optimizes the relative position of two cameras to minimize the gap between two models.

Figure 4: Multi-cameras system calibration The relative position of the camera is represented by the same matrix that OpenGL used for display. The transformation between two cameras can be written as

(1) For each manufactured multi-camera system, the relative position of each camera is recorded in the system configuration file in the form of transformation matrix. The software will automatically load this configuration file when the multi-camera system is in use. Within a very small range, the software will further align the models to get better fitting result. The fitting result of two models is visually displayed by pseudo-color. The color shows the gaps of two models. After models are aligned, the boundary of one model which falls into overlap region is marked. For each vertex in the overlap region, the closest boundary points are found, the distances are denoted as LA and LB. To get best blend effect, the vertex at overlap region is changed. The new vertex value can be expressed as (2)

SPIE-IS&T/ Vol. 6805 680506-4

The VA and VB are the original vertex value of model A and model B at same point. Eq.(2) is only a general rule for model blending. For the multi-camera system, at certain overlap region, one camera may produce better model than the other camera. Eq.(2) need consider extra weighting factor. General speaking, the smaller the angle between object normal direction and camera direction, and the better model quality it will be. In this case, the individual triangle generated by the 3D camera decoding software is also smaller. Therefore, we can put the triangle area as a weighting factor into Eq. (2)

LB

model A

LA V

model B

Figure 5: Model blending After the vertices at overlap region are modified, the model is cut at middle of the overlap region. Half the overlap region from each model is removed. Then the two models are stitched together along gap.

model A

model B

model A

model B

Figure 6: Illustration of the models stitching.

4. COLOR MAPPING IMAGE STITCHING The color mapping image is stitched separately from the 3D mesh model stitching. For each 3D model, the color information is stored in two parts of data; the mapping bitmap image and mapping coordinate of each vertex. Two different color stitching approaches are developed. In first approach, the mapping bitmap image from each 3d model is still kept in separated bitmap. The final 3D model actually consists of many 3D parts. Each part comes from each camera. The 3D parts of data go through the same stitching process described in section 3, but the data is organized into separated model. Since the models are closely contacted at the boundary, the viewer will only saw a single model.

SPIE-IS&T/ Vol. 6805 680506-5

The second approach can be used for simple object, such as human head. In this approach, all mapping bitmaps are projected cylindrically or spherically into a single mapping image. The mapping coordinate of each vertex is also calculated from corresponding projection. Figure 7 (a) shows a stitched color mapping image. The image is distorted because it is stretched from the cylindrical coordinate to flat surface.

(a)

(b)

(c) (d) (e) Figure 7: (a) Stitched color mapping image; (b) to (e): 3D model of a head.

(a)

(b)

(c)

(d)

Figure 8 (a) to (d): A full 360 degree model of an actor. The facial expression is preserved due to fast imaging capability of the 3D camera system.

SPIE-IS&T/ Vol. 6805 680506-6

Figure 7 (b) to (e) show the many views of a 360 degree 3D model with color map. Figure 8 (a) to (d) illustrate a full 360 degree model of an actor. The facial expression is preserved due to fast imaging capability of the 3D camera system. For both approaches, the mapping bitmap has to be adjusted to get smooth transition from one part of model to another part model. The multi-camera system use umbrella reflectors to get uniform illumination. However, for the overlap region imaged by two different cameras, there are still small differences in terms of brightness and contrast. Without compensation, the model will show sharp change in the boundary of stitching model. Since the overlap region of the bitmap is imaged by two cameras at different angle, one bitmap has to be morphed to match other bitmap. The 3D stitching already aligned the two models. Each vertex also has it mapping coordinate in the bitmap. So, many pixels at one bitmap already have the corresponding pixel coordinate at other bitmap. One bitmap can be morphed to match other bitmap.

Q

X

Q’

v

X’ v

u

u

P

P’ Figure 9: 3D vertex.

Suppose, there is matching pixels P, Q at two bitmaps. The left graph shows the targeted image, the right graph is from the original image. X is the point to be morphed. v is the distance from X to line PQ. u is the projection length of X on line PQ, normalized to the length PQ. Then coordinate of X at original bitmap is (4) is the length from P’ to Q’,

is the unit vector perpendicular to P’Q’. For the mapping bitmap

morphing, multi-matching line exists between two bitmap. weighting factor depends the distance from point X to line PQ.

Eq. (4) becomes a average result. The

After one bitmap is morphed, we can calculate the brightness and contrast of small region at both bitmap. The color value in this bitmap can be changed using Eq. (2). This way, we can get smooth transition of the mapping bitmap. The critical region, such as human eye, is marked so there will be no blending in this area. This way, we can get sharp image.

SPIE-IS&T/ Vol. 6805 680506-7

5. APPLICATIONS OF 3D IMAGING SYSTEM The 3D imaging system has many potential applications. Due to the high resolution and rapid imaging of the 3D camera system, it is particularly suitable to image live human. Figure 10 (a) shows the close up macro imaging of the skin that can be potentially used to record the dermatological changes of skin lesions and quantitative measurement of tumor growth. Figure (b) shows the precision 3D modeling of a tooth model crown. The 3D camera is ideal in taking 3D models of feet for digital production of orthotics, as illustrated in Figures 10 (c) and (d). The 360 degree 3D imaging system can be used to image live human full body 3D model, as shown in Figure 10 (e). It can also be used to digitize static objects, as shown in Figure 10 (f), and archeological objects, see Figures 10 (g) and (h).

(e)

(b)

(b)

(c)

(d)

(f)

(g)

(h)

Figure 10: Applications of the 360 degree 3D imaging system: precision 3D imaging of (a) skin lesions, (b) tooth crown, (c) and (d): foot, (e): full body, (f) a Tiffany lamp, (g) and (h) a Venus statue.

SPIE-IS&T/ Vol. 6805 680506-8

The software also provides many measurement tools. It can measure the distance between two points, the curve length, the area of selected region, and the volume enclosed by selected boundary.

6. CONCLUSIONS We have demonstrated a fast and accurate 3D imaging system using a single digital camera. The system synchronizes multiple 3D cameras to achieve fast and robust 3D imaging of 360 views of the object. The depth accuracy of the 3D imaging system can be 0.1 mm. The system can be scaled to image large objects. Automatic stitching of 3D models and color maps are discussed.

7. ACKNOWLEDGEMENTS The research described in this paper was carried out by the scientists at 3DMetrics, Inc and The Pennsylvania State University. The authors wish to thank the University of Northern California and Silicon Valley Investment Corporation for invaluable support.

8. REFERENCES 1.

2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14.

M. Levoy, J. Ginsberg, J. Shade, D. Fulk, K. Pulli, B. Curless, S. Rusinkiewicz, D. Koller, L. Pereira, M. Ginzton, S Anderson, J. Davis, "The Digital Michelangelo Project: 3D Scanning of Large Statues," Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp.131-144, 2000. B. Horn, Robot Vision, MIT Press, 1986. O. Faugeras, Three-Dimensional Computer Vision: a Geometric Viewpoint, MIT press, 1993. Z. Zhang, R. Deriche, O. Faugeras and Q.-T. Luong, “A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry”, Artificial IntelligenceJournal, Vol.78, pp.87-119, October 1995. J. Taijima, M. Iwakawa, "3-D data acquisition by rainbow range finder," Proc. of the 10th International Conference on Pattern Recognition, pp. 309-313, 1990. Z. J. Geng, "Rainbow Three-Dimensional Camera: New Concept of High-Speed Threedimensional Vision Systems," Opt. Eng. 35, pp. 376-383, 1996. K.G. Harding, M. P. Coletta, and C.H. VanDommelen, "Color Encoded Moire Contouring," SPIE vol. 1005, pp. 169-178, 1988. C. Wust and D.W. Capson. "Surface Profile Measurement Using Color Fringe Projection," Machine Vision and Applications, 4, pp. 193-203, 1991. T. P. Monks and J.N. Carter, "Improved Stripe Matching for Colour Encoded Structured Light," Proceedings of International Conference on Computer Analysis of Images and Patterns, pp. 476485, 1993. Z. Yang and Y. Wang, "Error Analysis of 3D Shape Construction from Structured Lighting," Pattern Recognition, 29, pp. 189-206, 1996. T. Lu, J. Zhang, “3 Dimensional Imaging System,” U.S. patent No. 6,252,623, 2001. T. Lu, T-H. Chao, “A high-resolution and high-speed 3D imaging system and its application on ATR,” SPIE 6245, Optical Pattern Recognition XVII, 2006. K. L. Boyer and A.C. Kak, "Color-encoded structured light for rapid active ranging," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PAMI-9, pp. 14-28, 1987. T. Sato, "Multispectral Pattern Projection Range Finder," SPIE vol. 3640, pp. 28-37, 1999.

SPIE-IS&T/ Vol. 6805 680506-9

15. G. Zhang and H. Wang, "Method of Establishing General Mathematical Model of Structured Light 3D Vision," SPIE, 2899, pp. 662-666, 1996. 16. G. J. Agin and P. T. Highnam, "Movable Light-Stripe Sensor for Obtaining Three-Dimensional Coordinate Measurements," SPIE Proc. Vol. 360, p. 326, 1983. 17. O. Faugeras and G. Toscani, “Camera Calibration for 3D Computer Vision”, International Workshop on Machine Vision and Machine Intelligence, pp. 240-247, Tokyo, 1987. 18. R. Hartley, “Estimation of relative camera positions for uncalibrated cameras”, Computer Vision ECCV’92, Lecture Notes in Computer Science, Vol. 588, Springer-Verlag, pp. 579-587, 1992. 19. Z. Zhang, R. Deriche, O. Faugeras and Q.-T. Luong, “A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry”, Artificial IntelligenceJournal, Vol.78, pp.87-119, October 1995.

SPIE-IS&T/ Vol. 6805 680506-10