Depth Reconstruction using Geometric Correction ...

Proc. of Int. Conf. on Advances in Communication, Network, and Computing, CNC

Depth Reconstruction using Geometric Correction with Anaglyph Approach for Satellite Imagery Swarnalatha P1, Tripathy B.K2, S. Prabu3, Ramakrishnan R4 and Manthira Moorthi S5 1,2,3 School of Computing Sciences & Engineering, VIT University, Vellore, India {pswarnalatha, tripathybk,sprabu}@vit.ac.in 4,5

Space Applications Centre, Ahmedabad, India

{rama, smmoorthi}@sac.isro.gov.in Abstract— The Images obtained through remote sensing systems are not often sufficient for high precision applications due to various distortions. The distortions can be due to errors like geometric errors, etc. Also multi-date satellite images of the same area under different conditions are difficult to compare because of change in atmospheric propagation, sensor response and illuminations. Keeping these points in view, in this paper, we deal with the first phase of pre-processing and we make the satellite images free from such errors and use clustering techniques With Geometric Correction (WGC) and Without Geometric Correction (WOGC) applied to the satellite images using our proposed algorithm. Finally, the image is reconstructed with depth dimension/depth map generation for the anaglyph image for better interpretation of satellite imagery. We have made experimental analysis of our algorithm using suitable satellite images and found the results to be very encouraging. Index Terms— Geometric Correction, Rough Intuitionistic Fuzzy C-Means (RIFCM) clustering algorithm, Depth Reconstruction, Anaglyph Graph, Satellite Images

I. INTRODUCTION Remote sensing is the acquisition of information about an object or phenomenon without making physical contact with the object. In modern usage, the term generally refers to the use of aerial sensor technologies to detect and classify objects on Earth (both on the surface, and in the atmosphere and oceans) by means of propagated signals (e.g. electromagnetic radiation) [1]. It may be split into active remote sensing, when a signal is first emitted from aircraft or satellites or passive (e.g. sunlight) when information is merely recorded. Images obtained through remote sensing systems are not often sufficient for high precision applications due to various distortions. The distortions can be due to geometric errors, radiometric errors and atmospheric errors. And compared to an efficient bitplane which is used for medical images may not give accurate results for satellite imagery [2].In this paper we focus on geometric errors only. The geometric errors can be corrected to a certain extent by considering only orbital geometry model approach which incorporates global transformations and takes into account the aspect ratio, earth rotation, image orientation, etc, [1] also yield poor accuracy. On the other hand, with detailed information extracted from ancillary, one can apply the correction processing to each pixel and hence get much higher accuracy. Pixel projection approach will consider the instrument distortion (measured pre-flight or in-flight), the © Elsevier, 2014

spacecraft attitude, position and velocity at the instant the image is formed in each CCD pixel. The Earth rotation effect will be calculated by a high-precision earth rotation model which involves regular rotation, precession, and mutation [3]. Therefore the geo-location (in terms of geodetic longitude and latitude) of each image pixel can be calculated. The geometric errors can be corrected to a certain extent with detailed instrument and spacecraft information (ancillary data) that provides instrument calibration data, spacecraft attitude, position and velocity at sufficiently small time interval which is essential for the satellite imagery. The literature study has been carried out on pre-processing of satellite imagery with statistical techniques and of geometric correction using non-parametric models [4][5]. The co-registration of three bands has been dealt in paper [6] which is a unique challenge due to the influence of orbit and attitude in the time gap in the imaging sequence. The co-registration problem of satellite imagery may be used to have better interpretation of depth reconstruction at the pre-processing phase of the author’s paper. Multi-date satellite images under different conditions of the same area are difficult to compare because of change in atmospheric propagation, sensor response and illuminations. Filters are commonly used for such things as edge enhancement, noise removal, and the smoothing of high frequency data. As a consequence, in this paper we first remove these types of errors in the pre-processing stage by applying With Geometric Correction (WGC). But, we also proceed without geometric corrections also, in order to provide a comparative analysis and get results in favour of the WGC. A novel approach of fuzzy c – means has been proposed in the paper [7] where the Bit Plane Filter(FCMBP) divides an image into planes using various mathematical techniques. The results proved quality classified images compared to conventional edge techniques with increase in the efficiency by reduction in the epsilon values of bit planes. And in future, the author aim is to apply other clustering techniques for reconstruction of the image with the help of the clustered image of the paper to get better interpretation of third dimension. The proposal has been made in [8] using rough c-means for analysis of satellite images with/without Bit Plane Filter (RCMBP-Rough C-Means Bit Plane) which segments the satellite image into planes using various mathematical techniques. And also the depth computation for all possibilities using with and without bit plane techniques have been carried out. On comparison, clustering using rough c-means yields better depth computation with minimum time complexity. In future, the authors aim is to apply other clustering techniques for further processing of satellite images. The depth-map (disparity image or Z-image) is an image which contains information about the depth of the picture and serves to convert a two-dimensional image into a three-dimensional one resulting depth reconstruction. In the depth-map the gray color gradation shows each pixel’s distance from the viewer. The lighter area in the depth-map corresponds to the areas nearer to the viewer, the darker one corresponds to more distant areas [9]. Anaglyph 3D images contain two differently filtered colored images, one for each eye which can be obtained by applying parallax algorithm is used which is a displacement or difference in the apparent position of an object viewed along two different lines of sight, and is measured by the angle or semi-angle of inclination between those two lines [10][11]. Clustering techniques are used in image segmentation and reconstruction of images after that generates enhanced images for analysis. Since a lot of uncertainty is involved in the image analysis we find a lot of uncertainty based clustering techniques which are more suited for such type of image clustering. In [12] a very efficient uncertainty based hybrid algorithm has been established and tested, called RIFCM, which combines intuitionistic fuzzy set and rough set approaches. In [13], a Comparative Study has been carried out using RIFCM with Other Related Algorithms from the point of view of their suitability in analysis of satellite images with other supporting techniques which deals with proving the superiority of RIFCM with RBP in clustering with other clustering methods and other supporting metrics with and without refined which integrates judiciously RIFCM with RBP. The superiority of the RIFCM using RBP is demonstrated, along with a comparison with other related algorithms, on a satellite images with NASA.org images(Hills, Drought) and national geographic photographic images(Freshwater, Fresh water valley). For two view depth reconstruction was extended to multiple view depth reconstruction in a manner similar to that of other [14, 15]. The use of multiple images provides more accurate depth results by mitigating the effect of image noise. The motion blur estimation problem is now represented by the depth estimation problem. The blur-robustness of the proposed algorithm is verified by comparing the depth reconstruction results with the conventional variational depth reconstruction using multiple images which is implemented by removing the blur-handling parts of the proposed method and thus achieved improved results [16]. 507

A True Color Composite is applied for satellite imagery for proper clusters which deals with a true color image which is retrieved when a multispectral image consists of the three visual primary color bands (red, green, blue). And the three bands may be combined to produce a "true color" image resulting color composite image which is observed by the human eyes for better visualization of satellite imagery[17]. In the later part of this paper, the image is reconstructed with depth dimension/depth map generation for the anaglyph image for better interpretation of satellite imagery. We have made experimental analysis of our algorithm using suitable satellite images and found the results to be very encouraging. The Section I of the paper deals with the introduction and survey of remote sensing, geometric correction, clustering, depth map generation, anaglyph image. And section II discuss about the methodology of the paper. Results and the discussion of the paper has been explained in section III. The conclusion and future work of the paper is dealt in section IV followed with acknowledgement and references cited for the paper. II. METHODOLOGY The proposed methodology deals with three phases of satellite imagery for depth map generation for better interpretation of images which is discussed in figure 1 as given below: 1. Initially, the satellite imagery has been given as input for geometric correction which consists of extraction of Ground Control Points(GCPs) for 35, 100 and 150 in three different ranges to get minimum rate of root mean square error as output image stating with geometric correction(WGC) and compared without geometric correction (WOGC). 2. Secondly, the output of WGC and WOGC is considered as input for clustering algorithm(Rough Fuzzy Intuitionistic C-Means) resulting WOGC-C(clustered) and WGC-C(clustered) images. 3. Thirdly, clustered images are given as input for depth map generation as reconstructed image with anaglyph approach for better interpretation of satellite imagery. a. Geometric Correction The pre-processed data is then used as the initial condition for the correction processing using GCP. The three stages has to be processed for geometric correction as, i) GCP extraction (using GRASS GIS) GCPs have been extracted by GRASS GIS for three categories (35,100,150 GCPs). The RMS error is shown for each GCPs. The two types of RMS error are displayed over the GCP manager panel. They are named as forward error and backward error which has been illustrated with two panels as source image(left) and target image(right) for every GCPs category in figure 2.a, b, and c. The performance has been tested by applying peak signal noise ratio and root mean square error metrics (figure 3) resulting minimum RMS for 150 GCPs compared to 35 GCPs and 100 GCPs (table I).

Fig. 1Architecture of proposed methodology with anaglyph approach of WOGC – C-D and WGC-C-D

508

a.

RMS – 35 GCPs

b.

RMS – 100 GCPs

c.

RMS – 150 GCPs

Fig..2. a.b.c.Extraction of GCPs (4thJan-May) TABLE I. A.GCP EXRACTION OF 1-35,1-100,1-150 FOR 4THJAN-MAY

Month

GCP

RMS

a. 4th Jan-May

35

0.034272

b. 4th Jan-May

100

0.037071

c. 4th Jan-May

150

0.036559

Figure 3. Graph of RMS for GCPs

Fig. 3 Performance of PSRN & RMSE for GCPs

ii) Filter and Image to Image Matching Invert filter is applied with neighborhood analysis and Image to Image matching process is done using Cross Correlation Coefficient with Resampling Techniques (fourier transform, matrix/convolution and Neighborhood Analysis (Average)) in GRASS GIS Open Source Software. The results of resampling techniques has been given in figure 4.a fourier analysis, 4.b. matrix /convolution and 4.c. neighborhood analysis (average) for further process. iii) Densification by ANN Densification process is used to add vertices to a line at specified distances without altering the line’s shape. This operation densifies geometries by points between existing vertices. And Densification Total Error Network Graph is applied to compute for two different months data to calculate Mean Square Error of Relative Geometric Corrected Image with least value of MSE of 0.0158(figure 5). 509

Fig.4.a Fourier Analysis Fig.No.4.b. Matrix/Convolution Fig..4.c. Neighborhood Analysis(Average)

Fig..5 January-Total Mean Square Error-0.0158

b. RIFCM The Rough Intuitionistic Fuzzy C-Means clustered technique has been applied for WOGC and WGC with PSNR and RMSE metric of performance validation. i. Rough Intuitionistic Fuzzy C-Means The rough intuitionistic fuzzy c-means (RIFCM) 12], uses the concept of rough sets, fuzzy sets and as well as intuitionistic fuzzy sets, thereby making it a perfect combination of IFCM and RCM. It can also be considered to be RFCM with IFS, hence adding the concept of lower and upper approximation of rough set, fuzzy membership of fuzzy set, non-membership and hesitation value of intuitionistic fuzzy set. It provides a holistic approach to clustering of data as it deals with uncertainty, vagueness, incompleteness which, enables the efficient handling of overlapping partitions and improves accuracy. In RIFCM, each cluster can be identified by three properties, a centroid, a crisp lower approximation and an intuitionistic fuzzy boundary. If an object belongs in the lower approximation of a cluster then its corresponding membership value is 1 and hesitation value is 0. The objects in the lower region have same influence on the corresponding cluster. If an object belongs in the boundary of one cluster then it possibly belongs to that cluster and potentially belongs to another cluster. Hence the objects in the boundary region have different influence on the cluster. Thus we can say that in RIFCM the membership value of objects in lower region is unity (µ = 1) and for those in boundary region behave like IFCM. The objective of this algorithm is to reduce the cost function given in [18].The parameters w have the standard meanings. Also The steps that are to be followed in this algorithm are as given below 1. Assign initial means v for c clusters by choosing any random c objects as cluster. 2. Calculate d using Euclidean distance formula (1) 3. Compute U matrix 510

and

w

if d

= 0 or x

BU then µ =1

else compute µ

4.

=

Compute

(x) = 1 5.

Compute µ

6.

Let µ

be the maximum and next to maximum membership values of object

x to cluster centroids v and v . if µ µ < then x BU and x approximation. else x BU 7. 8.

BU and x cannot be a member of any lower

Calculate new cluster means by using (2) Repeat from step 2 until termination condition is met or until there are no more assignment of objects.

d(x, y) = (x y ) + (x x w

vi

X

µ = µ +

and normalize

and µ

1 µ (x) |x 1 + µ (x)

µ (x)

lo w

k BU i

| B Ui |

x

x

k wu p

y ) + m

k BU i BU i x

ik

k BU i BU i

mx x k k BU i BU i ik m , x k BU i BU i ik x BU i x k k | B Ui |

x

m

+ (x

y ) … … … . (1)

k if | B U i |

a n d | BU i

ik

BU i |

…..(2) if |B U i |

a n d | BU i

BU i |

ELSE

The figure 6.a.i. deals with the Original image of Ahmedabad a.ii.Clustered image of Ahmedabad 6.b.i. Original corrected image of Ahmedabad b.ii.Clustered corrected image of Ahmedabad where the performance of satellite imagery has been calculated which is given in table 2 and figure 11. a.i. Original image of

a.ii..Clustered image of

b.i. Original Corrected image

b.ii. Clustered corrected image

Ahmedabad

Ahmedabad

of Ahmedabad

of Ahmedabad

Fig. 6.a.i. Original image of Ahmedabad a. ii.. Clustered image of Ahmedabad

Fig.6.b.i. Original Corrected image of Ahmedabad b.ii..Clustered corrected image of Ahmedabad

511

C. DEPTH R ECONSTRUCTION

The third dimension of the affected portion[19][20] is very vital as the degree to which the component has been deteriorated which can be calculated approximately. The angle of incidence of X-rays and the thickness of the component are taken as an input from the user. The third dimension of the component can be calculated using the formula given below. The formula for calculating third dimension is:

………………….(3) L = Length of the third dimension affected region, x = 0; Assuming that the third dimension is exactly in the centre = Angle of incidence of the X-rays, W = Width of the dimension, he component If the third dimension of the affected portion is greater than the thickness of the component then the component is fully affected and it has to be replaced with another one. Based on the three parameters (length, width and depth), the papers aims to reconstruct an image which may be possible as given below:Depth map generation algorithm (Depth Reconstruction) and Anaglyph Technique The third dimension map generation (Z-dimension)which is a gray-scale image should possess the resolution that is same as to the original input image for consideration of Depth Map Generation. Algorithm: Generation of Depth Map ( Depth Reconstruction )with Anaglyph Image 1. 2.

3. 4. 5. 6. 7.

Start Initialize images 2.1 First image has been taken as a original img,2D image (Oi) and 2.2 Second image is considered as a processed image (clustered image), 2D image (Ci),then Two images are opened in the interface: left and right frames of a stereo pair. And 3.1 A depth map (depth reconstruction) will be created from the two images Generate frames with the parallax method which characterizes the distance of the object's projections on the plane for the left and right eyes (disparity). Iterate the frames till is anaglyph image is obtained from the third dimension map generated for better visualization of an image . Finally, Anaglyph Image from third dimension map has been resulted Stop

To support the third dimension map generation, the parallax algorithm may be used where the results has been made in figure 7.a.WOGC-C-D and figure 7.b.WGC-C-D, which characterizes the distance of the object's projections on the plane for the left and right eyes (disparity). And the method, Parallax [21] which is a dislocation or dissimilarity in the apparent position of an entity viewed along two dissimilar lines of sight, is calculated by the point of view or semi-angle of leaning between those two lines. A third dimension (Z) map generation serves to exchange the original image into a depth map one. Intensity of a pixel in a third dimension map shows the space from the similar pixel in the original image to the watcher. The lighter areas in depth map match to the regions nearer to the viewer, the darker ones match to more distant areas. A white pixel in a depth map deals with the the pixel of the original image that has the smallest distance to the viewer (foreground),figure 8.a&9.a. and a black pixel with the pixel of the original that has the biggest distance to the viewer (background),figure 8.b&9.b. And frames of images will be generated from the third dimension map ,figure 8.c.& 9.c.resulting anaglyph 3D image, from a set of generated frames. The Anaglyph 3D images can be viewed through the "colorcoded" "anaglyph glasses", each of the two images reaches one eye, revealing an integrated stereoscopic image. The visual cortex of the brain fuses this into perception of a three dimensional scene or composition as given in figure8.d (WOGC-C-D)and figure 9.d (WGC-C-D). III. RESULTS AND DISCUSSION The proposed paper resulted overall method of WOGC-C –D and WGC-C-D resulting depth map with geometric correction (WGC)and without geometric correction(WOGC) which involves three steps as first phase deals without using geometric correction (WOGC) and with using geometric correction (WGC) has 512

a.

Original Image and Depth Map Image of Satellite Imagery of Ahmadabad without geometric correction

b.

Original Image and Depth Map Image of Satellite Imagery of Ahmadabad with geometric correction

Fig..7.a. Original Image and Depth Map of Satellite Imagery without geometric correction (WOGC-C-D) b. with geometric correction (WGC-C-D) a.

Background

b.

Foreground

c.

Depth map

d.

Anaglyph image

Fig..8. Depth images of a .background b. foreground c. depth map d. anaglyph images of without geometric correction (WOGC-C-D) a.

Background

b. Foreground

c.

Depth map

d.

Anaglyph image

Fig. 9. Depth images of a. background b. foreground c. depth map and d. anaglyph images of with geometric correction(WGC-C-D)

been dealt in figure 2.a.b.c and tabular representation(table-1) of performance analysis of metric PSNR and RMSE as per table 2 and figure 10 has been carried on which yields improvised value (8.3992 for WOGC to 7.5597dB for WGC). The clustering of WGC and WOGC has resulted in proper cluster of satellite imagery which yielded again better results as per table 2 and figure 10(8.3992-11.0707dB for WOGC) and ( 7.5597-10.8370dB for WGC) and clustered images also been tested which gives minimum PSNR and RMSE using WOGC. The clustered imagery has been considered for depth map generation as reconstruction with the help of stereo pair images resulting in anaglyph 3d image for better visualization of satellite imagery with an efficient unsupervised true colour composite results using without geometric correction(WOGC) and with geometric correction (WGC) as per figure 11.a.& figure 11.b. The metric PSNR and RMSE also applied to check the performance of satellite imagery (8.3992-6.9112dB for WOGC) and (7.5597-6.5697dB for WGC)as per figure 10 with related table 2. TABLE.II. PERFORMANCE OF PSNR AND R MSE VALUES Input Images

PSNR ValueWOGC

RMSE WGC

WOGC

8.3992

WGC

7.5597

Value-

PSNR Valueclustered

RMSE Valueclustered

PSNR Valuedepth map

RMSE Valuedepth map

13.7734

11.0707

13.7734

6.9112

10.3854

12.9627

10.8370

12.9627

6.5697

8.9570

513

Fig. 10. Overall performance of PSNR and RMSE of WOGC,WGC for original, clustered images and Depth Maps

(a-WOGC-C)

(b-WGC-C)

Fig.11. Accuracy of True Color Composite of Clustered with Original Satellite Imagery of a. without geometric correction(WOGC) and b. with geometric correction (WGC)

IV. CONCLUSION AND FUTURE WORK The proposed work of the paper deals with the Depth Reconstruction using Geometric Correction with Anaglyph approach for Satellite imagery. It discuss about without geometric correction (WOGC) and with geometric correction (WGC) as two ways for clustering using Rough Intuitionistic Fuzzy C-Means algorithm as WOGC-C and WGC-C. The resampling techniques have been used at WOGC and WGC level with PSNR and RMSE [ 0.036559 rate ] at WOGC-C and WGC-C level with PSNR and RMSE[7.5597-10.8370dB]and true color composite at WOGC-C and WGC-C level with PSNR and RMSE [7.5597-6.5697dB] for reconstruction of depth map generation. At all levels, the performance of satellite imagery yielded improvised results of WGC-C-D in comparison with WOGC-C-D. ACKNOWLEDGMENT The author thankfully acknowledges the Space Applications Centre people and others for their encouragement and support for the completion of the manuscript. REFERENCES [1] T. M. Lillesand and Ralph W. Kiefer, “Remote sensing and image interpretation”, Fourth Edition, John Wiley and sons inc, Singapore, 2003. [2] Swarnalatha P and P. Venkata Krishna,” An efficient method of Bitplane filtering algorithm using convex hull of – 2013 ,pp.9-16. [3] F. Arif, M. Akbar and A. M. Wu, “Most favorable automatic georeferencing based on GCPs selection using least square method”, proceedings of 4th International Conference on Digital Earth, 25-26, Taipei, Taiwan, May 2006. [4] Ramakrishnan.R, Manthira Moorthi.S, Prabu.S and Swarnalatha.P , “Comparative Analysis of Various Methods for Preprocessing of Satellite Imagery", International Journal of Engineering and Technology (IJET),Vol.5 No.1, pp.431-437,2013. [5] Prabu Sevugan, Ramakrishnan.R, Manthira Moorthi.S and Swarnalatha.P , "Non Parametric Model Re-Sampling Techniques for Geometric Correction of Remote Sensing Satellite Images", International Journal of Applied Engineering Research (IJAER), Vol.8(14),pp.1611-1621,2013.

514

[6] S.Manthira Moorthi, Raja Kayal, R.Ramakrishnan and P.K.Srivastava, “RESOURCESAT-1 LISS-4 MX bands on ground co registration by in-flight calibration and attitude refinement”, International Journal of Applied Earth Observation and Geoinformation Vol. 10 (2), pp. 140–146, June 2008. [7] B.K.Tripathy and P.Swarnalatha ,”A Novel Fuzzy C-Means Approach with Bit Plane Algorithm for Classification of Medical Images”, IEEE International Conference on Emerging Trends in Computing, Communication and Nanotechnology (ICE-CCN 2013), March(2013), pp.360 – 365.Carc,2013. [8] P.Swarnalatha and B.K.Tripathy,” Depth Computation using bit plane with clustering techniques for satellite images” accepted for publication in International journal of earth sciences and engineering,”in press”, ISSN: 09745904,2013. [9] Triaxes® StereoTracer, version 7.0, Triaxes Lab LLC. Russia, Tomsk, pp.1-2, 2013. [10] Shorter Oxford English Dictionary. 1968. "Mutual inclination of two lines meeting in an angle" [11] Jump upJump up "Parallax". Oxford English Dictionary (Second Edition ed.). 1989. "Astron. Apparent displacement, or difference in the apparent position, of an object, caused by actual change (or difference) of position of the point of observation; spec. the angular amount of such displacement or difference of position, being the angle contained between the two straight lines drawn to the object from the two different points of view, and constituting a measure of the distance of the object",1989. [12] Tripathy.B.K., Rohan Bhargava, Anurag Tripathy, Rajkamal Dhull, Ekta Verma and P.Swarnalatha , ”Rough Intuitionistic Fuzzy C-Means Algorithm and a Comparative Analysis” in proceedings of ACM Compute-2013, ,International Conference, SITE, VIT University, 21-22 August. 1,2013. [13] B.K.Tripathy and P.Swarnalatha, ”A Comparative Study of RIFCM with Other Related Algorithms from Their Suitability in Analysis of Satellite Images using Other Supporting Techniques”, Kybernetes, Vol.43.no.1,2013. [14] R. A. Newcombe, S. J. Lovegrove and A. J. Davison. Dtam,”Dense tracking and mapping in real-time” in Proc. ICCV, 2011. [15] J. Stuhmer, S. Gumhold and D. Cremers, ”Real-time dense geometry from a handheld camera” proceedings of the 32nd DAGM conference on Pattern recognition, pp. 11—20,(2010). [16] Hee Seok Lee and Kyoung Mu Lee,”Dense 3D Reconstruction from Severely Blurred Images using a Single Moving Camera”, 2012 [17] http://www.crisp.nus.edu.sg/~research/tutorial/opt_int.htm [18] Maji, P. and Pal, S.K., “Rough Set Based Generalized Fuzzy C-Means Algorithm and Quantitative Indices”, IEEE Transaction on System, Man and Cybernetics, Part B: Cybernetics, Vol. 37, No. 6, pp. 1529- 1540.2007. [19] P. Swarnalatha, Madhuri Kota, Nagarjuna Reddy Resu and G. Srivasanth.”Automated Assessment Tool for the Depth of Pipe Deterioration”, SCOPUS-IEEE International Advance Computing Conference, IACC 2009, Article number 4809101, Pages 721-724 ,2009. [20] P.Swarnalatha and B.K. Tripathy, “A Centroid Model for the Depth Assessment of Images using Rough Fuzzy Set Techniques” at International Journal of Intelligent Systems and Applications, Issue 2, Volume 1, ISSN: 2249-9954, Page Nos. 20-26, March 2012. [21] http://www.en.wikipedia.org/wiki/Parallax#cite_.

515