Recent Researches ιn Applied Informatics
Flower Information Retrieval using Color Feature and Location-based System WICHIAN PREMCHAISWADI1, NUCHAREE PREMCHAISWADI2 1 Graduate School of Information Technology in Business Siam University, Bangkok, 10160, Thailand
[email protected] 2 Faculty of Information Technology Dhurakij Pundit University, Bangkok, Thailand
[email protected] Abstract: - Mobile phones have currently become an essential part of human’s social life because of their potential benefits in communication. There are many useful services that have already been provided by mobile phones, such as the personal digital assistant function that allows users to effectively manage their daily tasks, alarm clock, emails, instant messaging, camera and windows media, etc. That’s why most people carry their mobile phones everywhere with them. Besides the common features that currently are available on mobile phones, it would be nice if they could integrate more capabilities, especially the capability of image recognition. One of the benefits that ‘mobile image recognition’ can bring to us is that in our life sometimes we can’t remember or retrieve the name of some places, fruits, herbs, flowers and these sort of things. For instance, when we approach a flower, we observe it and we become curious and then we want to find some information about that flower. This can be accomplished by using an image recognition service on our mobile phones. Therefore, the challenge of this research is to develop a mobile application that utilizes the camera feature on the mobile phone which is integrated with SOA-Services Oriented Architecture through the internet network. Moreover, we also improve recognition accuracy and provide fast image searching using the GPS to categorize the data as indexed.
Key-Words: - Flower Information Retrieval, Auto Color Correlogram and Correlation (ACCC) algorithm, Global Positioning System (GPS), Services Oriented Architecture (SOA) . ways to search and categorize the data. For example, we have difficulty searching using some keywords that are relevant to specific objects such as people, places, fruits, herb or flowers. It can be confusing and painstaking to search for such information when we don’t have any clear idea about their details while we are searching them. We may observe a specific kind of flower but we don’t know what the name is or the specie of that flower, so how we can search for it? This study suggests a practical technique to search the data based on the photos (without having general information about them). The terms “Mobile Image Search” represents a relatively new trend in the field for people using mobile devices and uses the camera equipment (on mobile) for taking photos and sending them to the system for processing and collecting the corresponding data with the results. Mobile devices present many unique characteristics that can be used in electronic tourist guides, such as ubiquity and convenience; positioning: by employing technologies like GPS, users may receive and access information and services specific to their location
1 Introduction At present, people carry their mobile phone with themselves for many reasons such as 1) mobile phones perform like a personal computer (personal digital assistant functions: office mobile, calculator, instant messaging), 2) includes many applications to support a variety of different tastes like windows media, games and network communication as WiFi, Bluetooth, GPS11 and GPRS22 , etc. Therefore, from the many mobile functions that provides facilities to support mobile users we can use mobile phones for retrieving the information through the internet by using images rather than keywords. The current keywords search engine which works based on “text” –like Google- has some limitations on the 1
The Global Positioning System (GPS) is a space-based global navigation satellite system (GNSS) that provides reliable location and time information in all weather and at all times and anywhere on or near the Earth. 2 General packet radio service (GPRS) is a packet oriented mobile data service on the 2G and 3G cellular communication systems global system for mobile communications (GSM).
ISBN: 978-1-61804-034-3
146
Recent Researches ιn Applied Informatics
[1, 2]. The existing mobile florist solutions typically support text queries. Therefore, users have to convert their needed information into words. However, it is sometimes difficult to describe the exact needs in text and the text input is inconvenient on small devices [3]. The content-based image indexing algorithms have been rapidly developed and improved in term of the accuracy and speed so that the existing algorithms for image indexing and retrieval can be applied to some realistic image retrieval applications. For example, Google has developed and launched an image search application on mobile phones, namely Google Goggles [4]. Google Goggles allows users to search pictures from a camera phone. This application uses a query image and several image recognition backend processes (object recognition, place matching, OCR, etc) to search different kinds of objects and places such as text, landmarks, books, contact info, places, wine, and logos that are similar to the intended target. In addition, a Google Similar Images lab [5] allows mobile’s user to search for images using pictures rather than words. But Goggles and Google Similar Images have limitation for supporting some smart phones and specifically the Android market, and requires an engine to be installed i.e. the Mobile Visual Search Engine on the Apple IPhone [6] from the independent vender Evolution Robotics [7]. It returns the image query results to the users via email but the IPhone cannot return the image query results in real-time. In this paper, we present a “mobile florist image search” for retrieving a flower’s information such as common name, genus, species, cultivar, skill level, hardiness, height, flowering period, description and location where it can be found. Our application can run on both mobile phones and personal computers that connect through the internet without the need for any software installation. Moreover, it can be run on any mobile phone operator/network that supports a standard web browser. In addition to the application, we propose a fast search mechanism by using a color image indexing method and categorizing the data with a GPS index, which is suitable for real-time processing for querying images from a large database. In Section 2, we review and describe related work on image recognition methods and image processing applications. Section 3 presents the problem statement and describes the research design. In Section 4, we describe and discuss the results from the experiments. The final section provides a conclusion and describes the future work.
ISBN: 978-1-61804-034-3
2 RELATED WORK There are many papers that propose image search algorithms and applications related to the performance of search techniques. Many papers discuss the area of flower and plant information retrieval from a server or across server providers. We have categorized them into 2 parts as research papers and commercial web sites as the following:
2.1 Research Papers The Photo-to-Search system proposed by Xin Fan et, al.[3] allows users to input multimodal queries, including duplicate image detection, content-based image retrieval, text-based Web image search, and key phrase extraction in an ensemble system to provide a feasible solution to support multimodal queries from mobile devices. However, it is implemented as a prototype server-side program to receive the query from an email address using the POP3 protocol and to process the query. Photobased question answering proposed by Tom Yeh et al. [8] for finding information about physical objects allows direct use of a photo to refer to the object. They present three prototypes of a photo-based QA system including an online album, a text-based QA, and a mobile application. For the image matching process, the algorithm depends on types of input images queries, such as images of posters, magazine covers, video frames, CD covers, grocery items, and buildings. Premchaiswadi et al. [9, 10], E. MengYoke Ta et al.,[11], M.V. Setten [12] and Z. Wa et al., [13] proposed an image search for tourist information using a mobile phone. The system was implemented and tested with real mobile phone queries. The experimental results showed that the proposed system could be applied for practical applications. The Auto-correlogram and Color Different Correlogram (AC/CDC) algorithm [14] are utilized in the image retrieval process. However, the AC/CDC algorithm also needs to be improved for querying a large images database. Anucha et al. [15, 16] proposed a technique named the Auto Color Correlogram and Correlation (ACCC) that integrated the autocorrelogram and auto color correlation techniques based on computation time as O(m2d). Our application is based on the ACCC techniques and includes GPS indexing techniques.
2.2 Commercial Web Sites So far, many popular web sites and e-commerce web sites have been applied to flower and plants image and content-item based searches. For example, Google Similar Image Search [5] will find
147
Recent Researches ιn Applied Informatics
correlogram and correlation (ACCC) algorithms. (d) Use the positioning field for clustering data. (e) After finishing the index comparison, the top ten similar flower photos will be returned and displayed using the mobile phone application through internet network. (f) The florist information such as common name, genus, species, cultivar, skill level, hardiness, height, flowering period, description and location found are shown for the photo the user selected.
images based on their similarity. Holland Wildflower Farm [17] presents online florist information based on ‘name search’ and ‘detailed search’. OnlineflowerSearch [18] using color, categories and month for searching features of America’s most complete flowers catalog. Netvibes [19], supports Androids mobile phones that analyze an image from several images database and presents image search results in a nice AJAX interface. Google image search on Android and IPhone have revamped its mobile image search, doing away with the one-image-at-a-time, no-pinch-to-zoom nonsense. The new image results page uses square thumbnails to maximize the number of images on screen.
4 Implementation and Testing Practical applications of image retrieval typically require real-time processing for large image database systems. The development and implementation of an image search application is a complex process, requiring a comprehensive understanding of both the theory and practice of image processing. The procedure starts when the mobile phone user takes a picture of the flower they are interested in and want to find information about it. Once the picture is taken, the image file is sent to the web service in order to compare the color of the requested image against the indexed images in the database. The color comparison is performed until the ten most similar color images are found. The resulting images are ranked by similarity as shown in the following pseudocode 1.
3 Architectural Overview In this section, we present the mobile phone image search application used for searching for flowers based on their ‘color feature’ to increase the performance, efficiency and effectiveness (saving in time) integrated with a GPS positioning system. To implement the color feature, we used the Auto Color Correlogram and correlation (ACCC) algorithm for searching images from a large image database.
Pseudocode 1 1. Take a photo and rename the image to imageOr 2. Resize imageOr to 100*140 pixel 3. Convert RGB color of imageOr to 64 color 4. Check every K distance 5. Check every X, Y position 6. Assign Current pixel to Ci 7. Assign Get neighbor pixel of Ci at distance K to Cj 8. Check Color Cm 9. Assign colorCount[k][Ci]++ 10. Assign colorR[k][Ci] = colorR[k][Ci]+colorRCj 11. Assign colorG[k][Ci] = colorG[k][Ci]+colorGCj 12. Assign colorBR[k][Ci] = colorB[k][Ci]+colorBCj 13. ACCC[k][Ci]= colorR[k][Ci]/(colorCount[k][Ci]*K*8) ACCC[k][Ci]= colorG[k][Ci]/(colorCount[k][Ci]*K*8) ACCC[k][Ci]= colorB[k][Ci]/(colorCount[k][Ci]*K*8) 14. Query ACCC[k][Ci] from index’s image database 15. Get 10 results from querying 16. Store imageOr and GPS data into ten storage and imageOr log file 17. Selected image result query 18. Update image index number to imageOr log file
Fig. 1 The architectural overview of our proposed application. As Fig. 1 illustrates, our method uses the following major steps: (a) the captured photo and currently location (latitude and longitude) will be sent over the internet to the web server through internet network. (b) Web-services on the web server side are provided to handle incoming photos from the mobile phone. The mobile phone application interacts with the web service in a manner prescribed by its description using SOAP messages, typically conveyed using HTTP with an XML serialization in conjunction with other webrelated standards. (c) The mobile phone file is indexed and compared with the indexes of all pictures in the database by using the auto color
ISBN: 978-1-61804-034-3
148
Recent Researches ιn Applied Informatics
As Pseudocode 1 illustrates, after taking a photo of an unknown object, the image file is sent to the system and subsequently renamed to “imageOR”. After that, the image processing procedure performs as follows. First, the image is resized to 100x140 pixels and adjusted to 64-bit color. Then the image index calculation is performed using the ACCC algorithm. The returned index is stored in the variable ACCC[k][Ci] and used for index comparison in the database query process. During the database query, the value of ACCC[k][Ci] is compared to the index of images in the database. After the index comparison, ten images with the smallest difference values are retrieved from the database and sent to the user. Meanwhile, the system saves the retrieved images and the user’s GPS position in a temporary directory so that the system administrator can use the data later for analysis to improve the efficiency of image retrieval from the database. Finally, after getting the resulting images, if the user selects to view information of a particular image, the system will update the temporary directory with information of the selected image which the users thinks is most similar to the photo they just took. The values of each preference image are recorded into the image indexing database. In addition, the characteristics of images can be explored by using the CBIR engine to calculate the similarity measure among images. At the server side, we created the database to store the florist information. We used Microsoft SQL Server 2008 for database creation and index management. Fields Common Name Genus Cultivar Exposure Hardiness Height Flowering period Location Photo
Table 1 shows an example image from our database which contains 1,200 photos. There are 9 fields in the florist table: common name, genus, cultivar, exposure, hardiness, height, flowering period, location and photo. The following is the scenario of using a mobile phone to find the flower information via our application. After a mobile phone user takes a picture of the flower, he/she uses the mobile device to connect to the internet, and then browse our URL. Apparently, the application does not require a specific mobile platform or mobile OS to be installed on it, but only requires an internet connection capability and built in GPS capability. The system automatically starts the GPS service for retrieving the current location of the mobile user. The flower image is sent through the internet network. We applied the SOA concept when we developed the web services for supporting many functions such as photo comparison, database connection and returning the flower information. The system returns the top ten similar flower photos obtained by using the color feature based on ACCC[k][Ci] algorithm and then performs the photo re-ranking process. Finally, the mobile user selects the flower photo they are most interested in and gets the flower information as shown in Fig. 2. As Fig. 2 illustrates, A) A mobile user uses his/her mobile device to take a picture of an interesting location that he/she would like to get more information about, or other interesting places around that location and then saves the picture into the phone’s memory and browses our system URL. The user selects a flower picture for querying and search for similar images. The application gets the current position from mobile user such as latitude = 13.721650 and longitude = 100.453104. B) The system calculates and returns the flower pictures for querying and sorting the 10 resulting similar images. C) The mobile user selects the flower photo that they want to get information about. For example if the first photo is selected, the flower information of Mrs. Edwards Water lily is displayed. The information about the flower shown includes common name, genus, cultivar, width, hardness and description. The latitude and longitude of the location is also converted to “Siam University”. This means that Mrs. Edwards Water lily has been found at Siam University.
Description Tulip Tulipa Christmas Marvel Full Sun Hardy 35 cm. April to May -33.829742,151.244286 Photo Name: PIC022.jpg File Size: 85.8K Created Date:01/03/2011
Table 1 Example Data and Fields in Florist Table
ISBN: 978-1-61804-034-3
149
Recent Researches ιn Applied Informatics
A
Fig. 4 The example of resulting images from query image based on ACCC[k][Ci] algorithm, first row is original image.
B
As Fig. 4 illustrates, the original image was taken and after the system processed the image, it will return the top ten similar photos and flower information which is displayed using the mobile phone application through the internet network. The main functions of the CBIR engine are to explore the image’s color characteristic values of each selected image and to record them into an image indexing database. In addition to exploring the characteristics of the images, the CBIR engine also calculates the similarity measure among images. C Fig.2 The screenshot of resulting images from query image based on ACCC[k][Ci] algorithm.
Fig. 5 The example of resulting 10 images As Fig. 5 illustrates, the ranking method uses the distance metric for comparing feature vectors. The system returned the similar color results based on sixty-four color depth for computation of all photos in the florist table and sorts all the images, and returns them to the users in real time.
Fig. 3 The top 10 of resulting images from query image based on ACCC[k][Ci] algorithm, first row is original image.
5 Conclusion
In Fig. 3, 4 and 5, we show the original flower on the first row and the top ten results on row 2 and row 3 starting from left to right such as 1, 2, 3... 10.
ISBN: 978-1-61804-034-3
In this paper, we applied the Auto Color Correlogram and Correlation (ACCC) algorithm for improving the performance of searching based on the color ranking feature. In fact, our proposed application is a part of indexing in that the returned index is stored in the variable ACCC[k][Ci] and
150
Recent Researches ιn Applied Informatics
[14] J. Huang et al, “Spatial Color Indexing and Applications”, in proceeding of Sixth International Conference on Computer Vision, pp. 606 – 607, 1998. [15] T. Anucha et al., “Spatial Color Indexing using ACC Algorithms”, Proceeding of the ICT&KE, pp. 113-117, 2009. [16] Anucha Tungkasthan, Sarayut Intarasema and Wichain Premchaiswadi, Spatial Color Indexing using ACCC Algorithm, IEEE ICT and Knowledge Engineering, pp. 113-117, 2009. [17] http://www.hwildflower.com/ [18] http://www.onlineflowersearch.com [19] http://www.netvibes.com/ [20] T. Yeh, J. J. Lee, T. Darrell, “Photo-based Question Answering”, ACM, MM’08, October , pp. 26–31, 2008. [21] X. Anguera, J. J. Xu, N. Oliver, “Multimodal Photo Annotation and Retrieval on a Mobile Phone”, ACM, MIR’08, October, pp. 30–31, 2008. [22] S. Brewster and M. Dunlop, “ Mobile human– computer interaction”, Mobile HCI 2004. Springer LNCS 3160, ISBN: 3-540-23086-6, 2004. [23] W3C Mobile Web Best Practices 1.0, Basic Guidelines, W3C Candidate Recommendation, http://www.w3.org/TR/
used for index comparison purposes in the database query process. The main function of the database query is to compare the value of ACCC[k][Ci] to the index of images in the database and retrieve ten images with least difference values from the database and send them to the user. In the feature selection process, we use our proposed algorithm to find the major and dominant portion of the picture by eliminating irrelevant parts to increase the efficiency and effectiveness of the photo search performance.
References: [1] M. Kenteris, D. Gavalas, and D. Economou, “An innovative mobile electronic tourist guide application”, London: Springer-Verlag, ch. 13, 2007. [2] Varshney U., “Issues, requirements and support for location intensive mobile commerce applications”, Int J Mob Commun, 1(3) pp. 247–263, 2003. [3] Xin Fan et al., “Photo-to-Search: Using Multimodal Queries to Search the Web from Mobile Devices”, ACM, MIR’05, November, pp. 10-11, 2005. [4] http://www.google.com/mobile/goggles/#land mark [5] http://similar-images.googlelabs.com/ [6] http://www.applevideos.info/?p=145 [7] http://www.evolution.com/core/ViPR/ [8] T. Yeh, J. J. Lee, T. Darrell, “Photo-based Question Answering”, ACM, MM’08, October , pp. 26–31, 2008. [9] W. Premchaiswadi, “An Image Search for Tourist Information using Mobile Phone”, WSEAS TRANSACTIONS on INFORMATION SCIENCE and APPLICATIONS, Issue 4, Volume 7, April 2010, pp.532-541. [10] W.Premchaiswadi, N.Premchaiswadi, S. Chimlek and S.Narita, “Image Indexing and Retrieval using Autocorelogram and Color Difference Corelograms (AC/CDC)”, ICFS’2002, 2002. [11] E. Meng-Yoke Ta et al., “An Analysis of Services for the Mobile Tourist”, ACM, MC'07 (Mobility'07), September, pp. 10-12, 2007. [12] M. v. Setten, S. Pokraev and J. Koolwaaij, “Context-Aware Recommendations in the Mobile Tourist Application” , COMPASS. Spring LNCS 3137, pp. 235-244, 2004. [13] Z. Wa et al, “Personalized Tourism Information System in Mobile Commerce”, IEEE, ICMECG’09, October, pp. 387 – 391, 2009.
ISBN: 978-1-61804-034-3
151
