3-D Shape Recovery from Image Focus Using ...

3-D Shape Recovery from Image Focus Using Gabor Features Fahad Mahmood, Jawad Mahmood, Ayesha Zeb, Javaid Iqbal National University of Sciences and Technology, Islamabad, Pakistan [email protected], [email protected], [email protected] ,[email protected] ABSTRACT Recovering an accurate and precise depth map from a set of acquired 2-D image dataset of the target object each having different focus information is an ultimate goal of 3-D shape recovery. Focus measure algorithm plays an important role in this architecture as it converts the corresponding color value information into focus information which will be then utilized for recovering depth map. This article introduces Gabor features as focus measure approach for recovering depth map from a set of 2-D images. Frequency and orientation representation of Gabor filter features is similar to human visual system and normally applied for texture representation. Due to its little computational complexity, sharp focus measure curve, robust to random noise sources and accuracy, it is considered as superior alternative to most of recently proposed 3-D shape recovery approaches. This algorithm is deeply investigated on real image sequences and synthetic image dataset. The efficiency of the proposed scheme is also compared with the state of art 3-D shape recovery approaches. Finally, by means of two global statistical measures, root mean square error and correlation, we claim that this approach –in spite of simplicity- generates accurate results. Keywords: Shape from focus, Gabor features, depth map, 3-D shape recovery, Focus Measure.

1. INTRODUCTION Estimation of depth map from RGB sensors is the most fundamental purpose of research of computer vision and image processing society. Every year numerous intelligent and robust 3-D shape recovery algorithms are proposed by the researchers to improve the 3-D shape recovery approach in real time and make available online for industrial applications. 3-D shape recovery applications include: Optical Microscopy [1], Medical Image analysis [2], Vision based indoor navigation for robots [3], Under water robotics [4], Depth map extraction using mobile phone camera [5], Multiple image fusion [6] and many others. Traditionally, a set of 2-D images are gathered during image acquisition step each having different focus information. Image acquisition is performed by varying distance of lens of microscope, changing the stage height having 3-D object placed on it along the optical axis of camera or fluctuating the focus of RGB sensor while keeping the lens and stage static. Figure 1 depicts the 15th image of each experimented object. The experimented dataset contains both the synthetic and real image dataset. Intelligent, computationally cheap and robust focus measure approaches are applied on the image set to convert the corresponding color value to the focus value. Initial depth map is thus recovered by varying the focus measure curve along the optical axis of camera. Approximation, smoothing filters or regression techniques are applied as postprocessing step to remove the discrete steps in the recovered 3-D shape and thus improving the accuracy of the procedure. This document is structured as follows: Section 2 describes the proposed focus measure which is the major contribution of the paper. Section 3 briefly summarize the relevant existing literature in 3-D shape recovery. Experimental results and analysis is exhibited in section 4 and section 5 concludes the paper. References are provided at the end.

2. OUR CONTRIBUTION In this article we are concerned with improving focus measure algorithm for 3-D shape recovery based on shape from focus architecture. Gabor features is widely known in image analysis domain due to their biological resemblance with the visual cortex of mammalian brains and low computational properties. Gabor kernels are similar to 2-D receptive field of human visual system and exhibit suitable characteristics with respect to spatial locality and orientation selectivity which characterize this approach superior than the most state of art feature extraction techniques. The greater performance of this architecture lies in its robustness of random noise sources, computationally cheap and high accuracy in every image processing and computer vision application. This approach is much widely utilized in face recognition, object recognition

and texture analysis in image domain. This document is restricted in utilizing Gabor feature as novel focus measure for 3D shape recovery of both microscopic synthetic images and real imagery using shape from focus architecture .

3. BACKGROUND Focus measure computation is the most essential and fundamental step in 3-D shape recovery based on shape from focus architecture. There are numerous focus measure approaches proposed in past few years mainly in spatial domain, frequency domain and wavelet domain. Said [7] provided extensive analysis of up-to-date focus measures applied to shape from focus approach. Fahad [8] proposed rank transform as focus measure for 3-D shape recovery application. Traditional shape from focus approach [9] is based on Gaussian interpolation. Asif [10] provided the solution of 3-D shape recovery using feed forward neural network technique. Tariq [11-12] proposed kernel regression and principal component analysis for recovering accurate depth map from a set of 2-D images. 3-D shape recovery can also be improved by approximation of edges after the focus measure application like Lorentzian Cauchy function [13] and Bezier surface approach [14]. In [15] the authors proposed innovative focus profile architecture along the optical axis by just considering the intensity distribution of focused image surface. Bilal and Choi [16] proposed dynamic programming (SFF.DP) to approximate the optimal FIS. The authors in [17] proposed curvelet transform to achieve the objective of shape from focus in the presence of different noise level. These techniques deliver good approximations but are computationally affluent and their implementation relies on initial estimates. Moreover, their performance deteriorates under noisy conditions and is sensitive to noise because most of them are gradient based

Fig. 1: Top: 15th image frame of simulated image sequences Simulated cone, Planar surface, TFT-LCD and Real cone. Middle: 20th image frame of real world image as balcony image. Bottom: 20th image frame of real world image sequence as alley.

4. PROPOSED SCHEME The proposed scheme employs the Gabor filter as focus measure operator. Gabor filter is named after Dennis Gabor. Gabor features is widely known in image analysis domain due to their biological resemblance with the visual cortex of mammalian brains and low computational properties. Normally This approach is much widely utilized in face recognition [18-21], object recognition and texture analysis in image domain.

4.1 Gabor filter A 2-D gabor kernel in spatial domain is basically a function which is modulated by sinusoidal plane wave. The general function is represented by

𝑔𝜃𝛾𝜎 (𝑥, 𝑦) = 𝑒𝑥𝑝 {−

𝑥2 + 𝑦2 𝑗𝜋 } 𝑒𝑥𝑝 { (𝑥𝑐𝑜𝑠𝜃 + 𝑦𝑠𝑖𝑛𝜃)} 2 2𝜎 𝛾𝜎

(1)

The parameter 𝜃 represents the orientation, 𝛾 represents the aspect ratio, 𝜎 represents the scale at the orthogonal directions, 𝜆 represents the wavelength where as 𝜆 = 𝜎𝛾. The visualization of gabor filter can be varied by changing the gabor magnitude parameter 𝜃 as visualized in figure 2(a) and varying the gabor phase parameter 𝛾 as visualized in figure 2(b).

Figure 2: Conception of Gabor phase and scale (a) Gabor scale, (b) Gabor phase.

4.2 Gabor filter as a focus measure operator Focus measure computation has the most fundamental role in 3-D shape recovery specially related to the shape from focus architecture. This document proposes a novel focus measure approach based on gabor features which convert the color information into corresponding focus values which will be further processed for 3-D shape recovery. The focus measure is computed by 2-D convolution of input image obtained during image acquisition with the proposed Gabor filter. Mathematically it is represented as: (2)

𝑓𝑔𝑎𝑏 (𝑥, 𝑦) = I ∗ 𝑔𝜃𝛾𝜎 (𝑥𝑜 , 𝑦𝑜 ) 𝑁−1 𝑁−1

𝑓𝑔𝑎𝑏 (𝑥, 𝑦) = ∑ ∑ 𝐼(𝑥, 𝑦) 𝑔𝜃𝛾𝜎 (𝑥𝑜 − 𝑥, 𝑦𝑜 − 𝑦)

(3)

𝑥=0 𝑦=0

The parameters used in this paper are (aspect ratio) 𝛾

𝜋

= 0.5,(theta) 𝜃 = 4 and (wavelength)𝜆 =4. By applying the

focus measure for each pixel in the sequence we will get the focus volume.

𝐼 ′ 𝑧 (𝑥, 𝑦) = 𝑓𝑔𝑎𝑏 (𝐼𝑧 (𝑥, 𝑦))

𝑧 = 1,2,3 … 𝑍

(4)

The Gabor filter is convoluted with each pixel of image sequences eight times having different orientation each time. This will generate eight different focus volumes. The L-2 norm based superposition method is employed on eight different focus volumes of Gabor function. 8

‖𝐼𝑧 (𝑥, 𝑦)‖2 = √∑(𝐼 ′ 𝑧 (𝑥, 𝑦))2 𝑧=1

𝑧 = 1,2,3 … . 𝑍

(5)

The bottom row of Figure 3 shows the output of the Gabor function of LCD-TFT, simulated cone, planar surface and real cone. For each object point, the sharpest pixel determines the depth information. The depth map 𝐷(𝑥, 𝑦) is computed by maximizing the focus curve along the optical axis direction. 𝐷(𝑖, 𝑗) = argmax|𝐼𝑧 |

(6)

𝑘

Where 𝐷𝑖𝑗 is the depth information of each frame number (𝑖, 𝑗), 𝐼𝑧 is the focus value of the pixel (𝑖, 𝑗) in the 𝑘 𝑡ℎ frame where 𝑛 is the total number of images in the image stack.

5. RESULTS AND DISCUSSION The newly suggested 3-D shape recovery approach based on morphological gradient element and wavelet decomposition is tested experimentally on the dataset of synthetic and real image sequences. Synthetic image dataset consists of real cone, simulated cone, TFT-LCD and planar surface. Synthetic images are captured in controlled lighting conditions. The dataset of each synthetic image consists of 100 color images. Real cone object is made up of hardboard with dense white and black lines to ensure the strong texture on the top surface. The dimension of real cone object is 90cm length with base diameter 14cm. Simulated cone images are prepared using computer simulation software (AVS- Active vision simulator). Planar surface is a slanted surface having rough texture on the top. TFT-LCD is the dataset of microscopic view of TFT-LCD. Real world image sequence is the dataset of 200 images of alley and balcony captured using a high resolution DSLR camera. Real world image sequence has a lot of challenges including shadows, occlusions, image noise and many others. Comprehensive examination and assessment of the results performed is examined in the successive sub section.

5.1 Focus Measure Curve The efficiency of a focus measure algorithm is mainly evaluated in terms of unimodality and monotonicity. In this experiment proposed focus measure algorithm is initially applied on a set of image sequences to replace the color information at each pixel with its corresponding focus value. Then a patch of image having 8 neighborhoods is recorded from each image by taking the same pixel location of the origin of the 8 neighborhood patch. In this article the proposed focus measure based on no-reference sharpness metric based on inherent sharpness approach is tested on the image dataset of LCD-TFT filter. A pixel location of (250,250)is chosen as an origin of 8 neighborhood patch, then a matrix of size 90 × 8 will be formed. The color value and focus values of the pixels of this matrix is recorded and plot is drawn to analyze its behavior against different image frame numbers. Figure 3(a) displays the gray level values of this matrix along the depth of optical axis direction. Figure 3(b) presents the focus value of this matrix normally known as focus curve and compared against traditional shape from focus and dynamic programming based shape from focus. It can be observed that focus measure curve of no-reference sharpness metric based on inherent sharpness approach is slender, smooth along the frame numbers and has a sharp curve as compared to other approaches.

Figure 3(left): Original data pixel (250,250) from LCD-TFT filter. (Right): Focus measure curve of dynamic programming based shape from focus, traditional shape from focus and proposed scheme.

5.2 Complexity The newly suggested focus measure approach is robust and computationally economical and can be used in real world applications under certain conditions. For recovering an accurate 3-D shape of any synthetic image having image volume 𝐼(𝑋 × 𝑌 × 𝑍), the presented algorithm has to repeat its iterations 𝑋𝑌𝑍 times. If the convoluted window for focus measure computation is (𝑁 × 𝑁) then the algorithm must have to be processed (𝑁 × 𝑁)𝑋𝑌𝑍 times to complete its operation. Assuming 𝑁 = 3 which is normally used then using the above mathematical equation, 9𝑋𝑌𝑍 iterations must be completed to recover a full 3-D shape. This describes the importance of computationally economical algorithm. The contributed algorithm named shape from focus based on Gabor features approach is examined in Matlab 2015 and compared against state of art 3-D shape recovery approaches. The proposed scheme took 21 seconds while traditional shape from focus, dynamic programming based shape from focus took 160 and 40 seconds respectively. The above time duration is the mean of four simulated image sequences that is real cone, simulated cone, TFT-LCD and planar surface.

5.3 Robustness This article experimented the newly proposed 3-D shape recovery algorithm on both real world sequences and synthetic image sequences. Although real world image sequences contains a little bit of variant image noise but we further extended our experimented by manually corrupting the set of synthetic images with Salt & pepper noise and Gaussian noise and testing how well our proposed architecture behaves in presence of different degree of image noise. We initially manually corrupted the image set of LCD-TFT filter with Gaussian noise having zero mean and 0.005 variance. The performance of traditional shape from focus and shape from focus based on dynamic programming degraded significantly in presence of Gaussian noise but the shape from focus based on no reference sharpness metric based on inherent sharpness showed robust performance. Figure 4 displayed the 3-D shape recovered by dynamic programming approach and proposed scheme in presence of Gaussian noise having zero mean. Traditional shape from focus scheme also recovered degraded 3-D shape of TFT-LCD similar to the dynamic programming approach therefore it is not included in the figure 3. Line Graph displayed in the Figure 5 depicts the performance of above mentioned 3-D shape recovery algorithms in presence of Salt & pepper noise by varying the variance. Correlation is used as performance measurement tool in this experiment. Table 1 shows the accuracy computation in presence of Salt & pepper noise of traditional shape from focus, shape from focus based on dynamic programming and the proposed scheme. It is clear from the table that in presence of Salt & pepper noise the proposed scheme exhibited lowest root mean square error whereas the correlation value of the proposed scheme is the highest. This graph shows that proposed scheme outperformed the state of art approaches by a clear margin which describes its efficiency and suitability in various 3-D shape recovery applications. 1

0.8

SFF.DP

Correlation

SFF.TR

0.6

Proposed

0.4 (a)

SFF.DP

(b) Proposed 0.2

Figure 4: 3-D shape of LCD-TFT with noise 0 1

1.5

2

2.5 Variance 3 3.5

4

4.5

5

Figure 5: Performance of different shape from focus approaches in the presence of Salt & pepper noise.

Table 1: Accuracy computation of variant shape from focus schemes in the presence of Gaussian noise

SFF method SFF.TR

RMSE

Correlation

9.2437

0.2453

SFF.DP Proposed

9.7563 8.0009

0.3238 0.9327

5.4 Accuracy Computation The most important parameter in deciding the effectiveness, efficiency and suitability in any application of any algorithm is the degree of its accuracy in synthetic and real world image sequences. If any focus measure or 3-D shape recovery algorithm is not generating an accurate result despite giving little complexity and focus curve then it is not advisable to further utilize that algorithm or conduct further research in that system.

(a) SFF.DP

(b) Proposed

(e) SFF.DP

(f) Proposed

(g) SFF.TR

(h) Proposed

(c) SFF.TR

(c) SFF.TR

(d) Proposed

Fig. 6: 3-D shape recovered by synthetic image sequences that is simulated cone, real cone, TFT-LCD and planar surface.

The proposed 3-D shape recovery algorithm is analyzed in this section on the basis of accuracy computation in both synthetic image sequences and real world image sequences. Figure 5 displays the 3-D shape recovered by the proposed algorithm on synthetic image sequences and compared against the result of traditional approach and dynamic programming architecture. There is no major difference between the 3-D shapes recovered by the two mentioned state of art approaches so in order to shrink the results mentioned in figure 6 we have displayed the comparison of results with one state of art approach. The 3-D shape recovered by the proposed algorithm is smooth while the dynamic programming and traditional approach recovered rough results. Table 2: Accuracy computation of variant shape from focus schemes

SFF method SFF.TR SFF.DP Proposed

RMSE

Correlation

8.2007 8.2621 8.0376

0.8440 0.9013 0.9327

Table 2 shows the accuracy computation of traditional shape from focus, shape from focus based on dynamic programming and the proposed scheme. It is clear from the table that the proposed scheme exhibited lowest root mean square error whereas the correlation value of the proposed scheme is the highest. It means that the proposed 3-D shape recovery scheme performs better than the state of art 3-D shape recovery approaches.

Fig. 7: Depth map recovered by real world images Row (1): Alley image by SFF.DP, Row (2): Alley image by proposed scheme, Row (3): Balcony image by SFF.TR, Row (4): Balcony image by proposed scheme

Figure 7 demonstrates the depth map generated by the proposed algorithm on real world image sequences and the benchmark techniques. The synthetic image sequences are prepared either by computer simulation software or under controlled lighting conditions. Real world image sequences are prepared by a high definition camera in indoor or outdoor environments. Real image sequences has a lot of high challenges including unwanted lighting conditions, shadows, image noises, blur and others which an algorithm can face during its application in robotic or industrial environment. The depth maps recovered by the traditional shape from focus and dynamic programming architecture are uneven while the suggested 3-D shape recovery algorithm based Gabor features generated an accurate and smooth result.

6. CONCLUSION Recovering an accurate and precise depth map from a set of acquired 2-D image dataset of the target object each having different focus information is an eventual purpose of 3-D shape recovery. Focus measure algorithm has fundamental role in accurate approximation of depth map because it converts the corresponding color information to the corresponding focus values. This document computes the focus value using Gabor features. Gabor features is widely known in image analysis domain due to their biological resemblance with the visual cortex of mammalian brains and low computational properties. Gabor kernels are similar to 2-D receptive field of human visual system and exhibit suitable characteristics with respect to spatial locality and orientation selectivity which characterize this approach superior than the most state of art feature extraction techniques. The greater performance of this architecture lies in its robustness of random noise sources, computationally cheap and high accuracy in every image processing and computer vision application. This algorithm is deeply investigated in terms of robustness, focus measure curve, accuracy and computational complexity. The experimental results exhibit that the proposed algorithm is computationally cheap and superior in performance as compared to recently proposed 3-D shape recovery approaches. Future work involves in employing approximation technique or regression algorithm to remove the discreteness in the computed 3-D shape. This research could be further extended by combining the above mentioned focus measure approach with an efficient shape recovery approach which further refines the initial depth map and obviously indicates to the better-quality results.

REFERENCES [1] M. Boissenin, J. Wedekind, Arul N. Selvan, Bala P. Amavasai, Fabio Caparrelli, and J. R. Travis, 'Computer Vision Methods for Optical Microscopes', Image and Vision Computing, 25 (2007), 1107-16. [2] Jun-Hyeong Do, Eunsu Jang, Boncho Ku, Jun-Su Jang, Honggie Kim, and Jong Yeol Kim, 'Development of an Integrated Sasang Constitution Diagnosis Method Using Face, Body Shape, Voice, and Questionnaire Information', BMC complementary and alternative medicine, 12 (2012), 85. [3] Tiago Gaspar, and Paulo Oliveira, 'New Depth from Focus Filters in Active Monocular Vision Systems for Indoor 3D Tracking', IEEE Transactions on Control Systems Technology, 23 (2015), 1827-39. [4] S. Karthik, and A. N. Rajagopalan, 'Underwater Microscopic Shape from Focus', in Pattern Recognition (ICPR), 2014 22nd International Conference on (IEEE, 2014), pp. 2107-12. [5] Supasorn Suwajanakorn, Carlos Hernandez, and Steven M. Seitz, 'Depth from Focus with Your Mobile Phone', in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015), pp. 3497-506. [6] Hui Zhao, Qi Li, and Huajun Feng, 'Multi-Focus Color Image Fusion in the Hsi Space Using the Sum-ModifiedLaplacian and a Coarse Edge Map', Image and Vision Computing, 26 (2008), 1285-95. [7] Said Pertuz, Domenec Puig, and Miguel Angel Garcia, 'Analysis of Focus Measure Operators for Shape-from-Focus', Pattern Recognition, 46 (2013), 1415-32. [8] Fahad Mahmood, Jawad Mahmood, Waqar Shahid Qureshi, and Umar Shahbaz Khan, '3-D Shape Recovery from Image Focus Using Rank Transform', in International Symposium on Visual Computing (Springer, 2016), pp. 51423. [9] Shree K. Nayar, and Yasuo Nakagawa, 'Shape from Focus', IEEE Transactions on Pattern analysis and machine intelligence, 16 (1994), 824-31. [10] Muhammad Asif, and Tae-Sun Choi, 'Shape from Focus Using Multilayer Feedforward Neural Networks', IEEE Transactions on Image Processing, 10 (2001), 1670-75. [11] Muhammad Tariq Mahmood, and Tae-Sun Choi, 'Shape from Focus Using Kernel Regression', in Image Processing (ICIP), 2009 16th IEEE International Conference on (IEEE, 2009), pp. 4293-96. [12] Muhammad Tariq Mahmood, Wook‐Jin Choi, and Tae‐Sun Choi, 'Pca‐Based Method for 3d Shape Recovery of Microscopic Objects from Image Focus Using Discrete Cosine Transform', Microscopy Research and Technique, 71 (2008), 897-907. [13] Mannan Saeed Muhammad, and Tae-Sun Choi, '3d Shape Recovery by Image Focus Using Lorentzian-Cauchy Function', in Image Processing (ICIP), 2010 17th IEEE International Conference on (IEEE, 2010), pp. 4065-68. [14] Mannan Saeed Muhammad, and Tae‐Sun Choi, 'A Novel Method for Shape from Focus in Microscopy Using Bezier Surface Approximation', Microscopy research and technique, 73 (2010), 140-51. [15] Mannan Muhammad, and Tae-Sun Choi, 'Sampling for Shape from Focus in Optical Microscopy', IEEE transactions on pattern analysis and machine intelligence, 34 (2012), 564-73. [16] M. B. Ahmad and T. S. Choi, “A heuristic approach for finding best-focused shape,” IEEE TCSVT, vol. 15, no. 4, pp. 566–574, 2005. [17] Rashid Minhas, Abdul Adeel Mohammed, and Q. M. Jonathan Wu, 'Shape from Focus Using Fast Discrete Curvelet Transform', Pattern Recognition, 44 (2011), 839-53. [18] Chengjun Liu, and Harry Wechsler, 'Independent Component Analysis of Gabor Features for Face Recognition', IEEE transactions on Neural Networks, 14 (2003), 919-28. [19] Linlin Shen, and Li Bai, 'A Review on Gabor Wavelets for Face Recognition', Pattern analysis and applications, 9 (2006), 273-92. [20] Meng Yang, and Lei Zhang, 'Gabor Feature Based Sparse Representation for Face Recognition with Gabor Occlusion Dictionary', Computer Vision–ECCV 2010 (2010), 448-61. [21] Baochang Zhang, Shiguang Shan, Xilin Chen, and Wen Gao, 'Histogram of Gabor Phase Patterns (Hgpp): A Novel Object Representation Approach for Face Recognition', IEEE Transactions on Image Processing, 16 (2007), 57-68.