Combining Textual and Visual Features for Image Retrieval

Combining Textual and Visual Features for Image Retrieval J.L. Martínez-Fernández1, Julio Villena Román1 Ana M. García-Serrano2, and José Carlos González-Cristóbal3 1 Universidad Carlos III de Madrid [email protected], [email protected] 2 Facultad de Informática, Universidad Politécnica de Madrid [email protected] 3 DAEDALUS - Data, Decisions and Language, S.A. [email protected]

Abstract. This paper presents the approaches used by the MIRACLE team to image retrieval at ImageCLEF 2005. Text-based and content-based techniques have been tested, along with combination of both types of methods to improve image retrieval. The text-based experiments defined this year try to use semantic information sources, like thesaurus with semantic data or text structure. On the other hand, content-based techniques are not part of the main expertise of the MIRACLE team, but multidisciplinary participation in all aspects of information retrieval has been pursued. We rely on a publicly available image retrieval system (GIFT 4) when needed.

1 Introduction ImageCLEF is the cross-language image retrieval track which was established in 2003 as part of the Cross Language Evaluation Forum (CLEF), a benchmarking event for multilingual information retrieval held annually since 2000. The scope of ImageCLEF is to collect and provide resources and encourage de exchange of ideas about image retrieval. Images are language independent by nature, but often they are accompanied by texts semantically related to the image (e.g. textual captions or metadata). Images can then be retrieved using primitive features based on their contents (e.g. visual exemplar) or abstract features expressed through text or a combination of both. Originally, ImageCLEF focused specifically on evaluating the retrieval of images described by text captions using queries written in a different language, therefore having to deal with monolingual and bilingual image retrieval (multilingual retrieval was not possible as the document collection is only in one language) 17. Later, the scope of ImageCLEF widened and goals evolved to investigate the effectiveness of combining text and visual information for retrieval 9. The MIRACLE team is made up of three university research groups located in Madrid (UPM, UC3M and UAM) along with DAEDALUS, a company founded in 1998 as a spin-off of two of these groups. DAEDALUS is a leading company in linguistic technologies in Spain and is the coordinator of the MIRACLE team. This is the third participation in CLEF 5, 10. As well as bilingual, monolingual and cross C. Peters et al. (Eds.): CLEF 2005, LNCS 4022, pp. 680 – 691, 2006. © Springer-Verlag Berlin Heidelberg 2006

Combining Textual and Visual Features for Image Retrieval

681

lingual tasks, the team has participated in the adhoc multilingual, Q&A, WebCLEF and GeoCLEF tracks. This paper describes experiments performed in the bilingual adhoc and medical image retrieval tasks defined in ImageCLEF 2005. For the first task, a semantic driven approach has been tried. Semantic tools used have been: EuroWordnet 2 and structure of the textual image descriptions. A new method for semantic query expansion has been developed, centered on the computation of closeness among the nodes of the EuroWordnet tree, where each node corresponds to a word appearing in the query. An expansion method based on the same idea was previously described in 11. For the second task, different combinations of content-based and text-based subsystems have been tested, trying to improve the results of the content-based system with the addition of text retrieval. The remainder of the paper is structured as follows: Section 2 describes the techniques and results obtained in the bilingual adhoc task; Section 3 explains the experiments defined for the medical image retrieval task; finally, Section 4 provides some conclusions and future directions to follow in image retrieval.

2 Bilingual Adhoc Task This year, a special focus on semantics has been made for the bilingual adhoc task. Two main techniques have been developed: the first one, a semantic expansion algorithm, based on EuroWordnet, where the focus is to obtain a common path among the concept tree represented in EuroWordNet. The idea is to make a more finegrained query expansion than including every synonym for each word in the query. Along with this expansion algorithm morphological information is also used to apply a set of simple rules to identify words introduced by negative particles (such as 'not', 'excluding', etc.) that must not appear in captions of images to be retrieved. The second technique was devoted to the construction of different indexes according to the fields used in image captions. Then several linguistic patterns were automatically built to recognize data included in one of these specialized indexes. These patterns were matched against the queries to focus the search process in some specific indexes. The following subsections describe these techniques. 2.1 Semantic Expansion Using EuroWordnet EuroWordnet is a lexical database with semantic information in several languages. This resource includes, for a given language, different semantic relations among dictionary entries, also called concepts. These relations include: hyperonym, where links with more general concepts are defined, hyponym, where relations with more specific terms are included and synonym, where constructions grouping entries with the same meaning (named synsets) are built. All possible meanings for a given concept are part of the EuroWordnet data structure. It is worth mentioning that not all languages are equally covered by EuroWordNet. As can be seen, a tree graph can be built using these semantic relations, and the distance among concepts, i.e., the semantic similarity among concepts, in this tree can be used as a disambiguation method for the terms included in the query 11.

682

J.L. Martínez-Fernández et al.

For example, bank is defined in EuroWordnet as "a financial institution that accepts deposits and channels the money into lending activities" and also as "sloping land (especially the slope beside a body of water)" along with eight more different senses. The question arising is: how can be the word bank disambiguated when used as part of a query? The answer considered in this work is: by means of the rest of the words appearing with bank in the query. That is, some of the synonyms for the words appearing with the word bank will overlap with the synonyms of bank. If it does not happen hyponyms and hyperonyms of the given words are considered, until some relations among the initial words are found. The senses which are not linked with the senses of other words appearing in the query expression can be discarded. Somehow, the main goal is to find one unique path through the EuroWordnet tree that joins all query words. By applying this approach, a fewer number of synonyms are included in the expanded query if compared with a rough expansion, where every synonym of a word is included in the expanded query.

C1

Sn1

S11

Hyperonym Hyperonym

C2

C3

Sn2

S12

S13

Sn3

Hyperonym

Cm S1m

Snm

Fig. 1. Hyperonym relations in EuroWordnet

The described situation is depicted in Figure 1. The marked area corresponds to semantically related concepts, where the sense Sn2 for the concept C2 (appearing in the query) is related, by a hyperonym relation, with the sense S11 for the concept C1 appearing in the query. In this way, concepts C1 and C2 can be expanded including words in S11 and Sn2 sets, discarding the rest of senses, Sn1 and S12.


683

The described algorithm has been implemented using Ciao Prolog and an adaptation of the Dijkstra algorithm has been developed to compute the shortest way between two nodes. An efficient implementation of the expansion method has been pursued and, for this reason, not all possible paths among nodes are computed, a maximum of three jumps are allowed to limit execution times to an affordable value. 2.2 Exploiting the Structure of Image Captions Captions supplied for the St. Andrews image collection are divided in fields, each of them containing specific information such as short title, location, etc. Image textual descriptions are composed of a total of 9 fields. Taking into account this structure, several indexes have been defined, one containing only image descriptions, another one with short title, one more with the photographer, another one with the places shown in the images, one with the dates when the pictures were taken and the last one with the proper nouns that have been identified in the image caption. From data available of previous campaigns, linguistic patterns have been automatically identified which allow the identification of information contained in specific caption fields. These patterns are matched against the query captions trying to determine which of the indexes should be searched or, in other way, which indexes can be discarded during the search process. Some previous work in this line is described in 12, but using named entities to decide which index should be searched. The Xapian1 search engine has been used to index text representations for the image captions and the ability for this search engine to perform search processes combining independent indexes has been used. This information distribution allows for the assignment of semantic interpretation for each field and, with a minimum processing for the query, it is possible to search a specific entity over the right index. For example, several queries ask for images taken by a predefined photographer; a simple processing of the query allows for the identification of structures like "... taken by ..." where the name to be searched can be extracted and located over the picture author index. This strategy allows for a finegrained search process that is supposed to provide better precision figures. 2.3 Evaluation Results Results produced by the different experiments are grouped according to the languages involved. Table 1 shows the Mean Average Precision (MAP) for the monolingual experiments presented this year by the Miracle group. All the submission IDs shown in the table begin with the prefix 'imir', and the rest of the identifier is built as follows: • The first two letters denote the field of the topic used in the experiment: 't0', when only the query title is used, 'd0', when only the narrative field is used, and 'dt', when both title and narrative are used to build the query for the search engine. • The next part of the identifier denotes the linguistic processing applied to the query: 'base', when the processes for the baseline are applied (i.e.: parsing, stopword filtering, special characters substitution and lowercasing and stemming); 1

Xapian. On line http://www.xapian.org/

684


's', when a morphosyntactical analysis for the query is performed; 'e', when the semantic expansion based on EuroWordnet is applied; 'o', when the operator to combine the expanded words is OR; 'a' when the operator to join expanded query words is a combination of OR operators with AND operators; 'pn', when proper nouns are identified in the text. • The following part identifies which index (or indexes) is (are) used to retrieve images. Possible values are: 't0', if only the titles of the captions are indexed, 'd0', when only the descriptions for the captions are searched, 'dt', when both titles and descriptions constitute a unique index, 'attr', if indexes for the different caption fields are used (the identified fields are: text, author, date, place), and finally 'allf', when a unique index with the content of all fields is used. • The next two letters identify the language in which the query is supplied. In monolingual experiments it is English, but for bilingual experiments it can it can identify one from 22 different languages (Bulgarian, Croatian, Czech, Dutch, English, Finnish, Filipino, French, German, Greek, Hungarian, Italian, Japanese, Norwegian, Polish, Portuguese, Romanian, Russian, Spanish - Latin America, Spanish - Spain, Swedish, Turkish and Simplified Chinese. • Finally, the last two letters identify the language in which the image captions collection is written. At this moment, the target language is always English. The best result is obtained when the query string, built taken only the topic title, is searched against the combination of attribute indexes (text, place, author, date). As a Table 1. Mean Average Precision for monolingual runs

Run imirt0attren imirt0allfen imirt0based0enen imirt0basedtenen imirt0eod0enen imirt0eotdenen imirtdseod0enen imirtdseotdenen imirt0baset0enen imirt0eot0enen imirtdseot0enen imirtdbased0enen imirtdbasedtenen imird0based0enen imird0basedtenen imirtdbaset0enen imird0baset0enen

MAP 0.3725 0.3506 0.3456 0.3286 0.2262 0.2183 0.1914 0.1851 0.1798 0.1405 0.1254 0.1065 0.1039 0.1010 0.0972 0.0555 0.0545


685

previous step, the query was processed with a basic procedure (parsing, normalizing words, stopword removal and stemming). This experiment receives the identifier 'imirt0attren'. It should be mentioned that, due to a programming error, duplicate elements were included in the results list, which could blur precision figures. These duplicate entries were deleted (but not substituted), lowering precision figures for the experiments. Besides, there is no a great difference between the MAP of the best experiment, 'imirt0attren', 37%, and the MAP of the next one 'imirt0allfen', 35%, where a unique index is built with the information contained in all the fields included in image captions. Results for bilingual experiments are also very interesting. In Table 2, the differences among experiments for each language can be noticed. The MAP precision values for the best result for each language are compared. The best bilingual MAP result is 74% of English monolingual, and it is reached for the Portuguese language. Comparing with the best monolingual result, a difference of around 7% in MAP value can be seen. Table 2. Mean Average Precision for bilingual runs

Run imirt0attren imirt0attrpt imirt0attrdu imirt0allfsl imirt0attrfr imirt0attrja imirt0attrge imirt0allfru imirt0attrit imirt0allfgr imirt0attrsp imirt0allftk imirt0attrsw imirt0allfzh imirt0attrno imirt0attrpo imirt0allffl imirt0attrro imirt0allfbu imirt0allfcz imirt0attrcr imirt0attrfi imirt0allfhu

Query Language English Portuguese Dutch Spanish (Latin America) French Japanese German Russian Italian Greek Spanish (European) Turkish Swedish Chinese (simplified) Norwegian Polish Filipino Romanian Bulgarian Czech Croatian Finnish Hungarian

MAP 0.3725 0.3073 0.3029 0.2969 0.2797 0.2717 0.2559 0.2514 0.2468 0.2436 0.2304 0.2225 0.1965 0.1936 0.1610 0.1558 0.1486 0.1429 0.1293 0.1219 0.1187 0.1114 0.0968

% 100.0% 74.3% 73.3% 71.8% 67.6% 65.7% 61.9% 60.8% 59.7% 58.9% 55.7% 53.8% 47.5% 46.8% 38.9% 37.7% 35.9% 34.6% 31.3% 29.5% 28.7% 26.9% 23.4%

686


As already tested in previous campaigns, the translation process between languages introduces a lot of noise, decreasing the precision of the retrieval process. The process followed in the 'imirt0attrpt' experiment is equivalent to the one applied in the best monolingual run, but including a previous translation step using online translators 31415. That is, the topic title is translated from Portuguese into English and then parsed, normalized, stopwords are removed and the rest of words are stemmed. The words forming the query are ORed and searched against the combination of attribute indexes (text, place, author, date). Of course, the previously explained problem with duplicate results in the final list also applies to the bilingual runs submitted. The MIRACLE team was the only participant for some target languages such as Bulgarian, Croatian, Czech, Filipino, Finnish, Hungarian, Norwegian, Polish, Romanian and Turkish.

3 Medical Image Retrieval Task In this task (referred as ImageCLEFmed), example images are used to perform a search against a medical image database consisting of images such as scans and xrays 6 to find similar images. Each medical image or a group of images represents an illness, and case notes in English or French are associated with each illness. For this purpose, we focused our experiments on fully automatic retrieval, avoiding any manual feedback, and submitted runs both using only visual features for retrieval (content-based retrieval) and also runs using visual features and text (combination of content-based and text-based retrieval). To isolate from the content-based retrieval part of the process, we relied on GIFT (GNU Image Finding Tool) 4, a publicly available content-based image retrieval system which was developed under the GNU license and allows to perform query by example on images, using an image as the starting point for the search process. GIFT relies entirely on visual information such as colour, shape and texture and thus it doesn’t require the collection to be annotated. It also provides a mechanism to improve query results by relevance feedback. Our approach is based on the multidisciplinary combination of GIFT content-based searches with text-based retrieval techniques. Our system consists of three parts: the content-based retrieval component (mainly GIFT), the text-based search engine and the merging component, which combines the results from the others to provide the final results. We submitted 13 different runs to be evaluated by the task coordinators, which can be divided in two groups: • Content-based retrieval, which includes experiments using GIFT with two different configurations: with and without feedback. When feedback is used, each visual query is introduced into the system to obtain the list of images which are more similar to the visual query. Then the top N results are added to the original visual query to build a new visual query which is again introduced into the system to obtain the final list of results. • Content-based and text-based mixed retrieval, including experiments focused on testing whether the text-based image retrieval could improve the analysis of the content of the image, or vice versa. We were interested in determining how text and image attributes can be combined to enhance image retrieval, in this case, in


687

the medical domain. As a first step, all the case annotations are indexed using a text-based retrieval engine. Natural language processing techniques are applied before indexing. An adhoc language-specific (for English, German and French) parser 16 is used to identify different classes of alphanumerical tokens such as dates, proper nouns, acronyms, etc., as well as recognising common compound words. Text is tokenized, stemmed 1316 and stop word filtered (for the three languages). Only one index is created, combining keywords in the three different languages. Two different text-based retrieval engines were used. One was Lucene 8, with the results provided by the task organizers. The other engine was KSite 7, fully developed by DAEDALUS, which offers the possibility to use a probabilistic (BM25) model or a vector space model for the indexing strategy. Only the probabilistic model was used in our experiments. The combination strategy consists on reordering the results from the content-based retrieval using a textbased retrieval. For each ImageCLEFmed query, a multilingual textual query is built with the English, German and French queries. The list of relevant images from the content-based retrieval is reordered, moving to the beginning of the list those images which belong to a case that is in the list of top-1000 cases. The rest of the images remain in the end of the list. 3.1 Evaluation Results Relevance assessments have been performed by experienced medical students and medical doctors as described in 1. The experiments included in Table 3 have been performed as follows: • miraqbase.qtop: this experiment consists on a content-based-only retrieval using GIFT. Initially the complete image database was indexed in a single collection using GIFT, down-scaling each image to 32x32 pixels. For each ImageCLEFmed query, a visual query is made up of all the images contained in the ImageCLEFmed query. Then, this visual query is introduced into the system to obtain the list of more relevant images (i.e., images which are more similar to those included in the visual query), along with the corresponding relevance values. Although different search algorithms can be integrated as plug-ins in GIFT, finally only the provided separate normalization algorithm has been used in our experiments. • mirarf5.qtop: this run takes the 5 most relevant images for feedback, each one with a value of 1 for its relevance in the visual query. The relevance in the visual query for the original images remains 1. • mirarf5.1.qtop: the same as mirarf5.qtop but using a value of 0.5 for the relevance in query of feedback images. The relevance in query for the original images remains 1. • mirarf5.2.qtop: the same as mirarf5.qtop but using a value of 0.5 for the relevance in query of the original images. As shown in Table 3, the best result for the content-based runs was obtained with the base experiment, which means that relevance feedback has failed to improve the

688

J.L. Martínez-Fernández et al. Table 3. Evaluation of content-based runs

Run mirabase.qtop mirarf5.1.qtop mirarf5.qtop mirarf5.2.qtop

MAP 0.0942 0.0942 0.0941 0.0934

% 100.0% 100.0% 99.8% 99.1%

results (neither to worsen them). This may be due to an incorrect choice of the parameters, but this has to be further studied. Apart from MIRACLE, other 8 groups participated in this year's evaluation in the content-based-only runs. Only one group is above us in the group ranking, although their average precision is much better than ours. Our pragmatic approach using a “standard” publicly available content-based retrieval engine such as GIFT has proved to be a better approach than other presumably more complex techniques. We still have to test if another selection of indexing parameters (different from image downscaling to 32x32 pixels and separate normalization algorithm) may provide better results. For the mixed retrieval experiments, the 10 different runs included in Table 4 were obtained as follows: • mirabasefil.qtop, mirarf5fil.qtop, mirarf5.1fil.qtop, mirarf5.2fil.qtop: these runs consisted on the combination of content-based-only runs with the text-based retrieval obtained with KSite. • mirabasefil2.qtop, mirarf5fil2.qtop, mirarf5.1fil2.qtop, mirarf5.2fil2.qtop: the same experiment, but using Lucene. • Two other experiments were developed to test if there was any difference in results when using our own content-based GIFT index or using the medGIFT results provided by the task organizers. So, medGIFT was used as the starting point and then the same combination method as described before was applied. - mirabase2fil.qtop: medGIFT results filtered with text-based KSite results - mirabase2fil2.qtop: medGIFT results filtered with Lucene results Results of the content-based and text-based mixed retrieval runs are included in Table 4. The use of relevance feedback provides slightly better precision values. Considering the best runs, the optimum choice seems to be to assume 1.0 for the relevance of the top 5 results and reduce the relevance of the original query images. Table 4 also shows that the results are better with our own text-based search engine than using Lucene (all runs offer better precision values), at least with the adopted combination strategy. This difference could be attributed to better language dependent pre-processing and removal of stop words. It is interesting to observe that the worst combination is to take both results provided by the task organizers (content-based medGIFT results and text-based Lucene results), with a performance decrease of 15%.


689

Table 4. Comparison of mixed retrieval runs

Run mirarf5.2fil.qtop mirarf5fil.qtop mirabasefil.qtop mirabase2fil.qtop mirarf5.1fil.qtop mirarf5fil2.qtop mirarf5.2fil2.qtop mirarf5.1fil2.qtop mirabasefil2.qtop mirabase2fil2.qtop

MAP 0.1173 0.1171 0.1164 0.1162 0.1159 0.1028 0.1027 0.1019 0.0998 0.0998

% 100.0% 99.8% 99.2% 99.0% 98.8% 87.6% 87.6% 86.9% 85.1% 85.1%

Text Retrieval Engine KSite KSite KSite KSite KSite Lucene Lucene Lucene Lucene Lucene

Comparing content-based runs with the mixed runs, Table 5 shows2 that the combination of both types of retrieval offers better performance and even the worst mixed run is better than the best content-based only run. This actually proves that text-based image retrieval can be used to improve the content-based only retrieval, with much superior performance. Apart from MIRACLE, other 6 groups participated in this year's evaluation in the content-based and text-based runs. In this case, our position in the table shows that the submissions from other groups clearly surpassed our results. Anyway, these results are not bad for us, considering that our main research field is not image analysis. Table 5. Comparison of content-based and mixed retrieval strategies

Run mirarf5.2fil.qtop mirabase2fil2.qtop mirabase.qtop

MAP 0.1173 0.0998 0.0942

% 100.0% 85.1% 80.0%

It is also interesting to note that most groups managed to improve their results with mixed approaches over the content-based only runs. This is especially visible for the NCTU group, with an improvement from 0.06 to 0.23 (+355%) in MAP.

4 Conclusions The experiments performed in ImageCLEF 2005 point out some conclusions: regarding to bilingual retrieval, the application of semantic centered techniques must be further tested to assess their usefulness. Obtained results are not conclusive, our best monolingual result is 5% under the best mean average precision obtained by the 2

The last column in this table shows the difference, in percentage, from the best result.

690


Chinese University of Hong Kong group, but an interesting research line has been opened. On the other hand, the quality of the translation is decisive for the quality of the retrieval process, as can be seen according to the average precision values for different languages. The degree of development of the translation tools applied 31415 is not the same for all languages and, for those with lower coverage, such as Finnish or Hungarian, MAP figures fall down. Regarding techniques combining text-based with content-based image retrieval, average precision figures can be dramatically improved if textual features are used to support content-based retrieval. Future works will be focused on improving the semantic expansion algorithm, combined with the use of semantic representations of sentences directed by shallow syntactic information. Regarding content-based retrieval techniques, different indexing features will be tested, along with the application of better quality text-based retrieval techniques.

Acknowledgements This work has been partially supported by the Spanish R+D National Plan, by means of the project RIMMEL (Multilingual and Multimedia Information Retrieval, and its Evaluation), TIN2004-07588-C03-01. Special mention to our colleagues of the MIRACLE team should be done (in alphabetical order): Francesco Carbone, Michelangelo Castagnone, Juan José Fernández, Ana González-Ledesma, José Mª Guirao-Miras, Sara Lana-Serrano, Paloma Martínez-Fernández, Ángel MartínezGonzález, Antonio Moreno-Sandoval and César de Pablo-Sánchez.

References 1. Clough, Paul; Müller, Henning; Deselaers, Thomas; Grubinger, Michael; Lehmann, Thomas M.; Jensen, Jeffery; Hersh, William: The CLEF 2005 Cross-Language Image Retrieval Track, Proceedings of the Cross Language Evaluation Forum 2005, Springer Lecture Notes in Computer science, 2006 - to appear 2. “Eurowordnet: Building a Multilingual Database with Wordnets for several European Languages”. On line http://www.let.uva.nl/ewn. March (1996) 3. FreeTranslation. Free text translator. On line http://www.freetranslation.com [Visited 20/07/2005]. 4. GIFT: The GNU Image-Finding Tool. On line http://www.gnu.org/software/ gift/ [Visited 15/11/2005] 5. Goñi-Menoyo, José M; González, José C.; Martínez-Fernández, José L.; and Villena, J.: MIRACLE’s Hybrid Approach to Bilingual and Monolingual Information Retrieval. CLEF 2004 proceedings (Peters, C. et al., Eds.). Lecture Notes in Computer Science, vol. 3491, pp. 188-199. Springer, 2005 6. Goodrum, A.A.: Image Information Retrieval: An Overview of Current Research. Informing Science, Vol 3 (2) 63-66 (2000) 7. KSite [Agente Corporativo]. On line http://www.daedalus.es/ProdKSiteACE.php [Visited 15/11/2005] 8. Lucene. On line http://lucene.apache.org [Visited 15/11/2005]


691

9. Martínez-Fernández, José L.; García-Serrano, Ana; Villena, J. and Méndez-Sáez, V.: MIRACLE approach to ImageCLEF 2004: merging textual and content-based Image Retrieval. CLEF 2004 proceedings (Peters, C. et al., Eds.). Lecture Notes in Computer Science, vol. 3491. Springer, 2005 10. Martínez, José L.; Villena, Julio; Fombella, Jorge; G. Serrano, Ana; Martínez, Paloma; Goñi, José M.; and González, José C.: MIRACLE Approaches to Multilingual Information Retrieval: A Baseline for Future Research. Comparative Evaluation of Multilingual Information Access Systems (Peters, C; Gonzalo, J.; Brascher, M.; and Kluck, M., Eds.). Lecture Notes in Computer Science, vol. 3237, pp. 210-219. Springer, 2004 11. Montoyo, A.: Método basado en marcas de especificidad para WSD, In Proceedings of SEPLN, nº 24, September 2000 12. Peinado, V., Artiles, J., López-Ostenero, F., Gonzalo, J., Verdejo, F.: UNED at ImageCLEF 2004: Detecting Named Entities and Noun Phrases for Automatic Query Expansion and Structuring, Lecture Notes in Computer Science, Volume 3491, Aug 2005, Pages 643 - 652 13. Porter, Martin: Snowball stemmers. On line http:// www.snowball.tartarus.org. [Visited 15/11/2005] 14. SYSTRAN Software Inc., USA. SYSTRAN 5.0 translation resources. On line http:// www.systransoft.com [Visited 13/07/2005]. 15. Translation Experts Ltd. InterTrans translation resources. On line http:// www.tranexp.com [Visited 28/07/2005] 16. University of Neuchatel. Page of resources for CLEF. On line http:// www.unine.ch/info/clef/ [Visited 15/11/2005] 17. Villena, Julio; Martínez, José L.; Fombella, Jorge; G. Serrano, Ana; Ruiz, Alberto; Martínez, Paloma; Goñi, José M.; and González, José C.: Image Retrieval: The MIRACLE Approach. Comparative Evaluation of Multilingual Information Access Systems (Peters, C; Gonzalo, J.; Brascher, M.; and Kluck, M., Eds.). Lecture Notes in Computer Science, vol. 3237, pp. 621-630. Springer, 2004