A PACS-oriented Multimodal Search Engine

4 downloads 272 Views 902KB Size Report
Jun 23, 2016 - Proposal: • A multimodal search engine for integration on a PACS archive. ... The query is obtained from a web service (or user interface).
A PACS-oriented Multimodal Search Engine Eduardo Pinho, Frederico Valente, Carlos Costa DETI/IEETA - University of Aveiro

Scenario • The use of digital medical imaging systems has increased significantly.

• Medical diagnosis, decision support; • Teaching and researching. • The Picture Archiving and Communication System (PACS) concept is widely accepted. •

DICOM: a relevant standard

• The number of medical imaging studies is constantly growing.

• New and better ways to retrieve medical information are essential.

2016-06-23

A PACS-oriented Multimodal Search Engine

2

Motivation Multimodal Information Retrieval The theories, algorithms, systems and challenges of indexing and

retrieving multiple modes (kinds) of data, which may include meta-data, free text, images, or other multimedia sources.

In medical imaging: • DICOM meta-data, pixel data, annotations, reports, ... • Leverage heterogeneous concepts to a practitioner's workflow.

2016-06-23

A PACS-oriented Multimodal Search Engine

3

Motivation However: • The application of multimodal information retrieval in the context of PACS is a challenging issue to be addressed • Multimodal retrieval tools are still rare in clinical practice

Proposal: • A multimodal search engine for integration on a PACS archive. • Built over Dicoogle • Text + Image Searches

2016-06-23

A PACS-oriented Multimodal Search Engine

4

Dicoogle • Open-Source PACS archive • Extensible query and indexation mechanisms – Apache Lucene index/query plugin

• Over 22 million images were indexed in Aveiro, Portugal.

github.com/bioinformatics-ua/dicoogle 2016-06-23

A PACS-oriented Multimodal Search Engine

5

Content-based Image Retrieval in Dicoogle • Implements feature extractors, midlevel feature descriptors and distance metrics. • Query by Example • Automatic indexation of features. • Profile-based approach

2016-06-23

A PACS-oriented Multimodal Search Engine

6

Multimodal Search Engine • A Dicoogle plugin • Uses existing query providers • Text • Image • Describes multimodal queries as trees of unimodal queries • Support for uploaded media content

• Implements multi-source retrieval algorithms • Score normalization • Late query fusion: CombSUM, CombMNZ, RRF, …

2016-06-23

A PACS-oriented Multimodal Search Engine

7

Multimodal Search Pipeline

The query is obtained from a web service (or user interface) 2016-06-23

A PACS-oriented Multimodal Search Engine

8

Multimodal Search Pipeline

Transform the input: query refining, expansion, relevance feedback, ... 2016-06-23

A PACS-oriented Multimodal Search Engine

9

Multimodal Search Pipeline

Trigger the unimodal query procedures 2016-06-23

A PACS-oriented Multimodal Search Engine

10

Multimodal Search Pipeline

Normalize scores and use query fusion algorithms to combine the lists of results 2016-06-23

A PACS-oriented Multimodal Search Engine

11

Multimodal Search Pipeline

Inject additional attributes to the results (i.e. DICOM identifiers) 2016-06-23

A PACS-oriented Multimodal Search Engine

12

Multimodal Search Pipeline

Optional DIM-level aggregation (study, series, …) 2016-06-23

A PACS-oriented Multimodal Search Engine

13

Machine-to-Machine Interface • A web service for interaction with the engine; • Existing multimedia retrieval standards? – MRML was considered, but has too many issues

 RESTful service with JSON data schema

2016-06-23

A PACS-oriented Multimodal Search Engine

14

User Interface • Modern, pure web based user interface • Unimodal query boxes • Drag-and-Drop search results and local files • Manual or automatic fusion strategy selection

2016-06-23

A PACS-oriented Multimodal Search Engine

15

Final Remarks • We proposed an architecture for multimodal information retrieval, supporting complex queries and new algorithms without major changes to the software. • Provides a multimodal layer of abstraction over existing retrieval techniques. • The architecture has been evaluated for scalability, for use in real-world scenarios. • Further research over this foundation has already started, aiming to improve and validate the quality of retrieval.

Thank you. 2016-06-23

A PACS-oriented Multimodal Search Engine

16