NovaMedSearch: A multimodal search engine for medical case-based retrieval André Mourão
[email protected]
Flávio Martins
[email protected]
Departamento de Informática Faculdade de Ciências e Tecnologia Universidade Nova de Lisboa Caparica, Portugal
ABSTRACT Medical information retrieval systems help support health care experts in diagnostic and treatment decisions through the management of large amounts of clinical data. However, the heterogeneity and the ever growing amount of data produced in medical environments poses several challenges. In this paper, we propose a multimodal search interface for medical articles to provide better analysis tools for diagnosis and medical case retrieval. The underlying framework entered the Medical ImageCLEF 2013 challenge and is based on state of the art information retrieval, image retrieval and data fusion techniques.
Keywords search, image retrieval, search interfaces, medical retrieval
1.
INTRODUCTION
Search engines are often the main aids for case-based retrieval for medical personnel activities including diagnostic. Textual search is the norm, but images can often provide additional information that is difficult to convey in textual queries. In medical information retrieval, additional information regarding the modality classification of the images can be conveyed from the query images (e.g. charts vs. Xrays) and information that is hard to writeup (e.g. the position of a mass on a MRI). In this demo, we present Nova MedSearch, a medical search engine that integrates the two search modalities: text and image. Our goal is to provide an intuitive and simplified way of supporting multimodal queries in medical search.
2.
SEARCH ENGINE
Nova MedSearch is a multimodal (text and image) medical search engine that can retrieve either similar images or related medical cases. The user can upload his own images to build their query or use existing sample images in their
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. OAIR ’13 May 22-24, 2013, Lisbon, Portugal Copyright 2013 CID 978-2-905450-09-8
223
Figure 1: Interface mockup.
queries. The results are displayed in an ranked list with basic information (e.g. title, keywords, images (if available)) and a link to the corresponding article details. The interface takes into account both the relevancy of the images and text similarity. Previous work exists on multimodal search engines such as MedGIFT [3] (text or image based search for medical articles) and img(Anaktisi) [6] (image based retrieval on multiple (i.e. medical) datasets)) two previously existing systems. Our interface aims at simplifying use of images and text simultaneous, both in queries and in the presentation of the results. For instance, for custom queries we use native drag and drop support in modern browsers for image uploading. We have also implemented a guided query expansion sys-
Query results
Example Query
the same features and an SVM. The text is indexed using Lucene and Lucene-SKOS is used to add the query expansion feature. The combination of the image and text results is based on a late fusion approach. The text and images are analyzed separately and the results are combined in a rank-based CombSUM variant. We found that the late fusion of the results was useful for heterogeneous queries (e.g only text, only 1 image, text and 3 images), as the combination of the image and text search can be ignored if the query does not contain images.
Articles: doi:XXXX: Title: ... doi:YYYY: Title: ... Images:
osteoporosis x-ray
2.2
Query parser osteoporosis x-ray
Image Analysis
Text processing
Image features
Textual features
2.3 Image index
Article index
Ordered results
Ordered results
Demo
In the demo, we display a fully functioning web based interface for our medical search engine. The users are able to make textual queries, image queries and combined queries (text or images). The users can upload custom images and we provide a set of sample combined queries. The results are either images or cases (medical articles). The type of queries follow the same structure of the Medical ImageCLEF.
Result combination
3.
Combined results
Figure 2: Framework overview. tem that interactively provides auto-complete suggestions and expansion feedback sourced from a SKOS version of the MeSH indexing terms. A mockup of the diferent components of the interface is in Figure 1. The query box (a) mockup contains a textbox to enter the text queries, emphasising the guided expansion interface. In the example, we see the terms that will also match when searching for the word ¨ osteoporosis¨. There is also an area to drop query images. The article search result mockup (b) contains a sample result for the query example above. Besides general article information (title with link to full article, date, venue and abstract), we display the images in the article and their modalities (e.g. x-ray, MIR). The image search result mockup (c) contains the best match for the query and information regarding the related article (including other images).
2.1
Technologies
The Nova MedSearch web application is developed in Java and C++ and modern HTML5 browser technologies. On the server side, the application is based on Apache Solr, Dropwizard REST framework to support the retrieval webservices for images and text. The underlying operating system is Ubuntu 12.04 LTS.
Framework
The framework (see Figure 2) behind the search engine is based on three major components: image processing, natural language processing and multimodal retrieval. The system is described in detail on our working notes of the medical ImageCLEF 2013 [4]. The image processing component is based on the analysis and retrieval of images using CEDD, FCTH, [2], Local Binary Pattern histograms [1] and color histograms of segmented image. We extract the features for the images and the dataset and store them on a FLANN index [5]. The retrieval is based on the L2 distance between the features of the query images and the features present on the dataset. Modality classification is also performed using
224
CONCLUSIONS
Our system combines a powerful framework based on state of the art image and text processing algorithms with a simple yet powerful multimodal search interface to provide a valuable tool to the retrieval of medical data.
4.
REFERENCES
[1] T. Ahonen, A. Hadid, and M. Pietik¨ ainen. Face description with local binary patterns: application to face recognition. IEEE transactions on pattern analysis and machine intelligence, 28(12):2037–41, Dec. 2006. [2] S. A. Chatzichristofis, K. Zagoris, Y. S. Boutalis, and N. Papamarkos. Accurate Image Retrieval Based on Compact Composite Descriptors and Relevance Feedback Information. International Journal of Pattern Recognition and Artificial Intelligence (IJPRAI), 24(2):207 – 244, 2010. [3] MedGIFT Group. medSearch - Medical search engine by HES-SO Valais, 2009. [4] A. Mour˜ ao, F. Martins, and J. Magalh˜ aes. NovaSearch on medical ImageCLEF 2013. In Working Notes of CLEF 2013, pages 1–10, 2013. [5] M. Muja and D. Lowe. Fast approximate nearest neighbors with automatic algorithm configuration. In International Conference on Computer Vision Theory and Application VISSAPP’09), volume 340, pages 331–340. INSTICC Press, 2009. [6] K. Zagoris, S. A. Chatzichristos, N. Papamarkos, and Y. S. Boutalis. img(Anaktisi): A Web Content Based Image Retrieval System. In 2009 Second International Workshop on Similarity Search and Applications, pages 154–155. IEEE, Aug. 2009.
