Jun 23, 2016 - Proposal: ⢠A multimodal search engine for integration on a PACS archive. ... The query is obtained from a web service (or user interface).
A PACS-oriented Multimodal Search Engine Eduardo Pinho, Frederico Valente, Carlos Costa DETI/IEETA - University of Aveiro
Scenario • The use of digital medical imaging systems has increased significantly.
• Medical diagnosis, decision support; • Teaching and researching. • The Picture Archiving and Communication System (PACS) concept is widely accepted. •
DICOM: a relevant standard
• The number of medical imaging studies is constantly growing.
• New and better ways to retrieve medical information are essential.
2016-06-23
A PACS-oriented Multimodal Search Engine
2
Motivation Multimodal Information Retrieval The theories, algorithms, systems and challenges of indexing and
retrieving multiple modes (kinds) of data, which may include meta-data, free text, images, or other multimedia sources.
In medical imaging: • DICOM meta-data, pixel data, annotations, reports, ... • Leverage heterogeneous concepts to a practitioner's workflow.
2016-06-23
A PACS-oriented Multimodal Search Engine
3
Motivation However: • The application of multimodal information retrieval in the context of PACS is a challenging issue to be addressed • Multimodal retrieval tools are still rare in clinical practice
Proposal: • A multimodal search engine for integration on a PACS archive. • Built over Dicoogle • Text + Image Searches
2016-06-23
A PACS-oriented Multimodal Search Engine
4
Dicoogle • Open-Source PACS archive • Extensible query and indexation mechanisms – Apache Lucene index/query plugin
• Over 22 million images were indexed in Aveiro, Portugal.
github.com/bioinformatics-ua/dicoogle 2016-06-23
A PACS-oriented Multimodal Search Engine
5
Content-based Image Retrieval in Dicoogle • Implements feature extractors, midlevel feature descriptors and distance metrics. • Query by Example • Automatic indexation of features. • Profile-based approach
2016-06-23
A PACS-oriented Multimodal Search Engine
6
Multimodal Search Engine • A Dicoogle plugin • Uses existing query providers • Text • Image • Describes multimodal queries as trees of unimodal queries • Support for uploaded media content
• Implements multi-source retrieval algorithms • Score normalization • Late query fusion: CombSUM, CombMNZ, RRF, …
2016-06-23
A PACS-oriented Multimodal Search Engine
7
Multimodal Search Pipeline
The query is obtained from a web service (or user interface) 2016-06-23
A PACS-oriented Multimodal Search Engine
8
Multimodal Search Pipeline
Transform the input: query refining, expansion, relevance feedback, ... 2016-06-23
A PACS-oriented Multimodal Search Engine
9
Multimodal Search Pipeline
Trigger the unimodal query procedures 2016-06-23
A PACS-oriented Multimodal Search Engine
10
Multimodal Search Pipeline
Normalize scores and use query fusion algorithms to combine the lists of results 2016-06-23
A PACS-oriented Multimodal Search Engine
11
Multimodal Search Pipeline
Inject additional attributes to the results (i.e. DICOM identifiers) 2016-06-23
A PACS-oriented Multimodal Search Engine
12
Multimodal Search Pipeline
Optional DIM-level aggregation (study, series, …) 2016-06-23
A PACS-oriented Multimodal Search Engine
13
Machine-to-Machine Interface • A web service for interaction with the engine; • Existing multimedia retrieval standards? – MRML was considered, but has too many issues
RESTful service with JSON data schema
2016-06-23
A PACS-oriented Multimodal Search Engine
14
User Interface • Modern, pure web based user interface • Unimodal query boxes • Drag-and-Drop search results and local files • Manual or automatic fusion strategy selection
2016-06-23
A PACS-oriented Multimodal Search Engine
15
Final Remarks • We proposed an architecture for multimodal information retrieval, supporting complex queries and new algorithms without major changes to the software. • Provides a multimodal layer of abstraction over existing retrieval techniques. • The architecture has been evaluated for scalability, for use in real-world scenarios. • Further research over this foundation has already started, aiming to improve and validate the quality of retrieval.
Thank you. 2016-06-23
A PACS-oriented Multimodal Search Engine
16