Image registration of captured high resolution frames is essential for ensuring an ... in the spherical manifold domain without any intermediate image warping ...
Panoramic Imaging System for Mobile Devices Yingen Xiong1
Xianglin Wang1 1
Marius Tico1
Nokia Research Center
2
Chia-Kai Liang1,2
Kari Pulli1
National Taiwan University
Figure 1: Panoramic image (7664 × 672) created in a camera phone based on 18 high-resolution input images. We present the design and implementation of a mobile imaging system for high resolution panoramic image creation. The system comprises the following components: automatic camera motion tracking and high resolution image capturing, image registration on spherical manifold, image warping, image labeling, and image blending. Camera motion is estimated by tracking consecutive low resolution viewfinder frames captured at 30 frames per second. The alignment algorithm proposed in our previous work [Adams et al. 2008] creates compact summaries of the frames that allow rapid tracking of the camera motion. The high resolution images used to build the final panorama are automatically captured when the camera motion with respect to the previous image exceeds a threshold. Image registration of captured high resolution frames is essential for ensuring an accurate representation of the scene. The existing approaches to image registration could be classified in two categories: feature based, and image based methods [Zitova and Flusser 2003]. Our registration method for panorama application uses a coarse-to-fine strategy and a combination between image and feature based registration methods. Image-based registration is adopted at the coarse levels of an image pyramid where the features are less reliable. Finer resolution levels are registered using a feature-based approach. We employed corner features, and both image and feature matching operations are carried out using similarity measures that are independent to illumination changes. Feature based matching is performed in conjunction with RANSAC to achieve robustness to outliers, such as moving objects. An unlimited viewing angle is enabled by mapping the final panorama onto a spherical manifold. In order to reduce the computational complexity, the proposed registration method acts directly in the final manifold coordinates. Registration parameter estimation in spherical coordinates has been proposed in [Szeliski and Shum 1997]. In addition, our approach also carries out the corresponding feature selection and matching in spherical coordinates. Noting that spherical manifold warping will change the relative coordinates between features but it has a small effect on local neighborhood around each feature, we perform feature matching directly in the spherical manifold domain without any intermediate image warping operations. The image warping and spherical mapping are then performed in a single step and only once for each input image, based on the estimated registration parameters. Besides computational complexity, another advantage of carrying the registration out directly in spherical coordinates, is that the parametric motion model between images is highly simplified, i.e., similarity transformation is sufficient. Several tests and comparisons have been carried out in order to validate the proposed image registration approach. The results reveal that the proposed approach is robust to moving objects in the scene and significant illumination differences between images. The mobile implementation of the algorithm is also highly efficient with about 3 seconds per image pair registration when running on a mobile phone. To avoid ghosting and blurring due to moving objects, parallax, and registration errors, we perform graph cut to find the optimal seams
in the overlapping areas of the source images. Like the approach of Agarwala et al. [2004], our cost function includes two terms: the data penalty for each pixel and the interaction penalty for each pair of the neighboring pixels. To efficiently perform graph cut on the mobile phone, we simplify the cost function. For the data penalty, we set it as a very large number if the pixel is in an invalid area; otherwise, we set it as zero. We use the color distance to the neighboring pixels as the interaction penalty. By minimizing the cost function with alpha expansion, we can find the best seams in the overlapping areas. Image blending is the final and often very important step in creating high quality panoramic images. In this system, we employ several approaches for image blending. We use a fast approach called mask-based blending to produce a quick result for user preview and use a gradient domain blending approach to produce a high-resolution and high-quality result. In the mask-based blending, we first create a mask for each source image. The mask has highest values at the image center and the lowest values on the borders. Then the masks are warped and with their corresponding source images using GPU acceleration. Finally, the masks provide weights for alpha blending. Since camera conditions and environment illuminations may be very different when capturing the source images for panorama, the seams found by graph cut may still be visible. Therefore, we apply Poisson blending [P´erez et al. 2003] to smooth the color differences and hide the seams. In the gradient domain blending, we create a gradient field by copying the gradients from the constituent images and setting the gradients across the seams to be the average of the overlapped gradients. The best-fit image to the gradient field can be recovered by solving the Poisson equation. We solve the Poisson equation in the coarse multi-spline domain and generate the final image by seam-adaptive interpolation as in [Szeliski et al. 2008], and effectively reduce the memory and runtime by orders of magnitudes. On Nokia N95 with a ARM 11 332 MHz processor and 64 MB RAM, our system can generate a 150◦ panorama from 5 input images in 25 seconds for mask-based blending and 50 seconds for the gradient domain blending, respectively. Figure 1 is an example of the panoramic images.
References A DAMS , A., G ELFAND , N., AND P ULLI , K. 2008. Viewfinder alignment. Computer Graphics Forum 27, 2, 597–606. AGARWALA , A., D ONTCHEVA , M., AGRAWALA , M., D RUCKER , S., C OLBURN , A., C URLESS , B., S ALESIN , D., AND C OHEN , M. 2004. Interactive digital photomontage. ACM Trans. Graph 23, 294–302. P E´ REZ , P., G ANGNET, M., AND B LAKE , A. 2003. Poisson image editing. ACM Trans. Graph. 22, 3, 313–318. S ZELISKI , R., AND S HUM , H.-Y. 1997. Creating full view panoramic image mosaics and environment maps. In Proceedings of SIGGRAPH 1997, 251–258. S ZELISKI , R., U YTTENDAELE , M., using multi-splines.
AND
S TEEDLY, D. 2008. Fast poisson blending
Z ITOVA , B., AND F LUSSER , J. 2003. Image registration methods: a survey. Image and Vision Computing 21, 977–1000.
