Multi-Layer Virtual Slide Scanning System with ... - Semantic Scholar

IEICE TRANS. INF. & SYST., VOL.E96–D, NO.4 APRIL 2013

856

PAPER

Special Section on Medical Imaging

Multi-Layer Virtual Slide Scanning System with Multi-Focus Image Fusion for Cytopathology and Image Diagnosis Hiroyuki NOZAKA†a) , Member, Tomisato MIURA† , and Zhongxi ZHENG† , Nonmembers

SUMMARY Objective: The virtual slides are high-magnification whole digital images of histopathological tissue sections. The existing virtual slide system, which is optimized for scanning flat and smooth plane slides such as histopathological paraffin-embedded tissue sections, but is unsuitable for scanning irregular plane slides such as cytological smear slides. This study aims to develop a virtual slide system suitable for cytopathology slide scanning and to evaluate the effectiveness of multi-focus image fusion (MF) in cytopathological diagnosis. Study Design: We developed a multi-layer virtual slide scanning system with MF technology. Tumors for this study were collected from 21 patients diagnosed with primary breast cancer. After surgical extraction, smear slide for cytopathological diagnosis were manufactured by the conventional stamp method, fine needle aspiration method (FNA), and tissue washing method. The stamp slides were fixed in 95% ethanol. FNA and tissue washing samples were fixed in CytoRich RED Preservative Fluid, a liquid-based cytopathology (LBC). These slides were stained with Papanicolaou stain, and scanned by virtual slide system. To evaluate the suitability of MF technology in cytopathological diagnosis, we compared single focus (SF) virtual slide with MF virtual slide. Cytopathological evaluation was carried out by 5 pathologists and cytotechnologists. Results: The virtual slide system with MF provided better results than the conventional SF virtual slide system with regard to viewing inside cell clusters and image file size. Liquid-based cytology was more suitable than the stamp method for virtual slides with MF. Conclusion: The virtual slide system with MF is a useful technique for the digitization in cytopathology, and this technology could be applied to tele-cytology and e-learning by virtual slide system. key words: virtual slide system, liquid-based cytology, multi-focus image fusion, cytopathology

1.

Introduction

Many diagnostic imaging systems have recently been developed for histopathology, including a tele-pathology system, digital microscopy, and a virtual slide system [1], [2]. Virtual slides are high-magnification whole digital images of histopathological tissue sections [1], [2]. The virtual slide system enables perfect digitization of the histopathology diagnosis in the laboratory. In recent years, development of the virtual slide system has advanced to the level of practical application in both clinical [3] and research fields [4], [5]. The virtual slide system is able to be utilized for tele-pathology, electronic medical records (EMR), and immunohistochemical analysis [4], [5]. Development of these digitalized tools may enable a shift from optical microscope diagnosis to display diagnosis in histopathological diagnosis, as has occurred in Manuscript received June 18, 2012. Manuscript revised October 25, 2012. † The authors are with the Graduate School of Health Sciences, Hirosaki University, Hirosaki-shi, 036–8564 Japan. a) E-mail: [email protected] DOI: 10.1587/transinf.E96.D.856

medical imaging with the transition from film- to displaybased diagnosis. Great progress has been made in the development of digital imaging tools for histopathological diagnosis in recent years; however, little progress has been made in the development of digital imaging tools for cytopathological diagnosis because cytopathological smear slides are thicker than histopathological paraffin-embedded tissue sections and because the surface of a cytopathological smear slide is not a flat and smooth plane. The existing virtual slide system is optimized for scanning flat and smooth plane slides such as histopathological paraffin-embedded tissue sections, but is unsuitable for scanning irregular plane slides such as cytopathological smear slides. Additionally, the internal structures of cell clusters are unobservable only single focus (SF) photograph because the depth of focus at the time of high magnification object lens is narrow. Therefore, a virtual slide system for cytopathology requires a mechanical function to scan the three-dimensional structure of cell clusters; it also requires a new imageprocessing technology that is able to display a threedimensional virtual slide. Multi-focus image fusion (MF) is one of technologies which is observable internal structure of cell clusters. MF is a concept of combining multiple layer images into composite images, through which more information than that of individual input images can be revealed [6], [7]. A number of methods have been proposed for image fusion. Laplacian pyramid method and wavelet transform method are known as representative MF methods. In the present study, we developed a multi-layer virtual slide scanning system with MF technology. We compared SF virtual slide with MF virtual slide, and evaluated the usefulness of MF in cytopathological diagnosis. 2. 2.1

Virtual Slide System Overview Hardware Unit

We adopted the area scanning technique in developing the proposed virtual slide system. The virtual slide system consists of a microscope (ScopeA1. Carl Zeiss, Japan) and three-dimensional transfer device (X/Y axis stepping motor, Z axis stepping motor, Oriental motor, Japan) for cytopathological slides. The appearance and structure of the virtual slide system was shown in Fig. 1. The threedimensional transfer device holds the glass slide by air pressure. The transfer accuracy of this system was designed

c 2013 The Institute of Electronics, Information and Communication Engineers Copyright 

NOZAKA et al.: MULTI-LAYER VIRTUAL SLIDE SCANNING SYSTEM WITH MULTI-FOCUS IMAGE FUSION

857

Fig. 1

Virtual slide system.

better than 0.1 µm (Z axis) and 1 µm (X/Y axis). The threedimensional transfer device serves the function of both a microscope slide stage and a glass slide carrier. A 1CCD camera (DFW-910SX, SONY, Japan) was set for macroscopic image capture, and another 3CCD camera (Hitachi Kokusai Electric Inc., Tokyo, Japan) was attached to the microscope for microscopic image scanning. Objective lenses with ×40 and ×20 magnification are available.

Fig. 2

Virtual slide process outline.

processes of microscopic image capture and MF are performed concurrently by parallel pipeline processing. Phase 4: Virtual slide creating. Following the MF, a stitching process is performed on all field images to create a virtual slide data, using a stitching algorithm that combines three different alignment methods based on different imaging characteristics: phase-correlation alignment, feature based alignment, and normalized correlation alignment. The flow chart of stitching process was shown in Fig. 4.

2.2 Software and Image-Processing Technology The software consists of programs for motor control, camera control, and image processing. For virtual slide scanning, we developed three image-processing algorithms employed in (1) a program that detects the scanning area from the macroscopic image, (2) a program that detects the presence of cells on the slide under microscopic image scanning, and (3) the MF program. The appearance of the virtual slide system was shown in Fig. 2. The flow chart of scanning process was shown in Fig. 3. The virtual slide data is created by the following steps. Phase 1: System initialization. Phase 2: Scanning parameter setting. The glass slide is moved under the macroscopic camera by three-dimensional transfer device, and the macroscopic image is captured and displayed on the monitor. The parameters of the microscopic camera (image compression rate, shutter speed, exposure time et al.), microscopy (objective lens) and number of layer in the Z-axis direction are selected in this phase. Phase 3: Scanning phase. Cells smeared on the glass slide are detected automatically as the scanning area. The glass slide is moved to starting X/Y position. The multilayer images are captured by microscopic camera and stored to storage disk. Subsequently, the image analysis with wavelet transformation is carried out, a microscopic image is created by the MF process and stored to storage disk. The same process is executed in all fields of the glass slide. The

1) Phase-correlation alignment (PCA) is a frequency domain method that makes use of the shift property of the Fourier transform — a shift in the spatial domain is equivalent to a phase shift in the frequency domain. Using the rotation and scale properties of the Fourier transform in the logpolar coordinate domain, rotation and scale are also equivalent to a shift in the frequency domain invariant to any translation. The most remarkable property of phase correlation compared with classical cross-correlation is the accuracy with which the peak of the correlation function can be detected; due to whitening of the signals by normalization, phase correlation is also notably robust to the types of noise that are correlated with the image (e.g., uniform variations in illumination and offsets in average intensity). 2) Feature-based alignment (FBA) is a comprehensive approach that extracts interesting features, builds a multilayered pyramid model, and applies different algorithms in different layers. 3) Normalized correlation alignment (NCA) is a mathematical procedure used to locate the reference pattern in the image under consideration. In this method, the reference pattern is shifted to every location in the image, the values are multiplied by the overlaid pixels, and the total value at the overlaid position is stored to form an image showing those regions that are identical or similar to the reference pattern. To normalize the results of the pattern matching or correlation while avoiding bias resulting from the absolute brightness value of the region, the operation is usually cal-

IEICE TRANS. INF. & SYST., VOL.E96–D, NO.4 APRIL 2013

858

Fig. 3

System flow chart of scanning process.

Fig. 4 Stitching process flow chart. ※1 Try pre-defined different overlapping sizes to obtain the corresponded power-spectrum distribution, then estimate the alignment data based on the normalized power spectrum distribution, ※2 Extract pre-defined geometrical features, then align those features in each pair of images, and calculate the weighed confidence; for cases when the confidence is too low, NCA will be used as verification as well ※3 Using standard phase-correlation algorithms here

NOZAKA et al.: MULTI-LAYER VIRTUAL SLIDE SCANNING SYSTEM WITH MULTI-FOCUS IMAGE FUSION

859

culated as the sum of the products of pixel intensity divided by their geometric mean. The virtual slide data created by these image processing methods are then converted into an original data format optimized for original virtual slide viewer. Virtual slide data include system information (machine name, capturing condition, stitching map), a low-magnification image created from the high-magnification image, and the original highmagnification captured image. The stitching map records the position for accurately stitching each high-magnification image. The created virtual slide was shown in Fig. 5. The virtual slide viewer displays the virtual slide image designated by the cytotechnologist in the navigation field; Target area image on viewer is displayed as the specified magnification by user. This process is carried out by real-time image processing in viewer software program.

Fig. 5

Virtual slide by MF.

Table 1

3.

3.1

Evaluation of Focus Fusion Virtual Slide for Cytopathology Materials and Method

Tumors were collected from 21 patients diagnosed with primary breast cancer. Pathological diagnosis and Cytopathological diagnosis by conventional microscope was shown in Table 1. Sample treatment was shown in Fig. 6. After surgical extraction, samples were recovered by the conventional stamp method, fine needle aspiration (FNA) method, and tissue washing method. The stamp slides were fixed in 95% ethanol. FNA and tissue washing samples were fixed in CytoRich RED Preservative Fluid (Becton, Dickinson and Company Inc., NJ, USA), a liquid-based cytopathology (LBC). The LBC slide was manufactured by the manual method, and both the stamp and LBC slides were stained with Papanicolaou stain. Smear thickness of slide was mea-

Fig. 6 Sample treatment process. A: Breast cancer tissue after surgical extraction. B, C: Fine needle biopsy. D, E: LBC process F: Papanicolaou stain

Pathological diagnosis and smear slide thickness.

IEICE TRANS. INF. & SYST., VOL.E96–D, NO.4 APRIL 2013

860

sured by optical microscope (AxioImagerZ1, Carl Zeiss). The stamp and LBC slides were scanned by the virtual slide system after setting of the thickness parameter of each slide. After scanning, both SF virtual slide and MF virtual slide was created, and evaluated effectiveness in cytopathological diagnosis. Evaluation was carried out by 5 pathologists and cytotechnologists. To compare influence of the sample treatment methods, we also evaluated the cytopathological findings of the cells and cell clusters on the virtual slides. Statistical analysis was performed with IBM SPSS Statistics 21 (IBM SPSS Statistics, Chicago, IL, USA). All slides used for this research were made with the technologist blinded to patient information, and individual privacy was protected. 3.2 Results The thickness of the largest cell cluster found on the slides was shown in Table 1. Mann–Whitney’s U-test revealed no difference in size among the sample treatment methods. The Table 2

results of structure and cytopathological findings (nuclear, cytoplasm, and cell cluster) on virtual slide were shown in Table 2. Although visualization of the nuclear and cytoplasm in each method was sufficient, the LBC method was better than the stamp method with respect to visualization of cell clusters. Representative virtual slide was shown in Fig. 7. The nucleus inside of nuclear was clearly observable on MF virtual slide, and the undersurface cell of multilayer cell clusters has appeared by MF. MF virtual slide of the stamp and LBC methods was shown in Fig. 8. On the stamp images, large cell cluster caused a ghosting of the image (Fig. 8-A), and necrosis of cells made diagnosis difficult on the virtual slide (Fig. 8-C). On the LBC images, background materials such as necrotic cells, mucus, and hemorrhage dissolved in the CytoRich RED Preservative Fluid, cytopathological findings are clearly observable inside the cell clusters (Fig. 8-B, D). The results of compare the cytological diagnosis by microscopy and MF virtual

Comparison of cytopathological findings among three methods.

Fig. 7 Comparison of SF virtual slide and MF image fusion virtual slide. The nucleus of a cell was observed clearly and the undersurface cell of a multilayer cell group has appeared by MF.

NOZAKA et al.: MULTI-LAYER VIRTUAL SLIDE SCANNING SYSTEM WITH MULTI-FOCUS IMAGE FUSION

861

Fig. 8 Comparison of virtual slide stamp method and FNA-LBC method. A) Stamp: cluster with extreme thickness (Pap. ×400) B) LBC: same case of 8-A (Pap. ×400) C) Stamp: this case has strong necrotic background (Pap. ×400) D) LBC: same case of 8-C (Pap. ×400) Table 3

Comparison of cytopathological diagnosis microscope and virtual slide.

slides was shown in Table 3. Although the cytopathological features were slightly different between the stamp and FNA-LBC methods, disagreement case in cytological

diagnosis between microscopy and MF virtual slides did not cause. In terms of the ease of observation inside cell clusters, it is clear that the virtual slide system with MF

IEICE TRANS. INF. & SYST., VOL.E96–D, NO.4 APRIL 2013

862

performs better than the SF virtual slide system. The LBC method was more suitable than the stamp method for tissue preparation for MF virtual slides, however MF technology was of limited effectiveness for cell cluster sizes less than 20 µm. 4.

Disscussion

Existing virtual slide systems incorporate only mono-layer scanning or Z-stack scanning [8]. Although it is possible to scan multi-layers on slides by a virtual slide system with Z-stack function, Mori et al. [8] report two problems in using Z-stack function: file size and display efficiency. Increasing the layer number also leads to an increase in the image data, as these factors are related. The increase of virtual slide capacity causes delay of the handling response on the display, and it results in increase of load on the system. Therefore, it is necessary to develop a method of multi-layer imaging that is unaffected by the number of layers. In MF, multi-layer images are converted to a mono-layer, and it is stored as small final file size that is equivalent to the file size of a histopathological virtual slide. Therefore, the file size of an LBC slide can be compressed to less than 200 Megabyte. Display efficiency is the second problem encountered by cytotechnologists performing first screening by using existing virtual slide systems. Cytotechnologists usually carry out first screening by using a low-magnification image, and then check the cells or cluster which have the features of cancer at high magnification. In the case of Z-stack function, the cytotechnologist have to change magnification on each level of layers, and it is required longer time than conventional microscopic screening. Therefore, it is not suitable to perform first screening for cytopathology by using this method. Thomas Kalinski et al. [9] report virtual 3D Microscopy using multiplane whole slide images in diagnostic pathology. 144 gastric biopsy specimens are scanned for detection of Helicobacter pylori (H.pylori) by virtual slide system with Z-stack. They scanned 9 focus planes, and obtain the conclusion that virtual 3D microscopy appropriate for primary diagnosis of H.pylori gastritis and equivalent to conventional microscopy. However, 3D microscopy which they showed is not carried out image processing by volume rendering, they observe 9 layers for detection of H.pylori, it means 9 times observation carried out for diagnosis on display by virtual slide. In the case of observation for H.pylori detection in gastric biopsy, diagnosis by using virtual slide with MF is possible at one times observation. Mori et al. [8] also report that many cytotechnologists regarding viewing by the 10-layered focus image to be “insufficient,” and that this method was “Not usable” or “Usable, but requires effort.” Following the MF technology, there is no limit to the number of layers used to create the image by using MF, and it is not required for the operator to change layers at each level of magnification. Therefore, it is easy for the cytotechnologist to make observations at the requested magnification. In contrast, the effectiveness of focus fusion is lim-

ited for cell clusters less than 20 µm thick, and is easily affected by the background material. It is necessary to resolve these problems by making improvements to the algorithm or the sample treatment process. Although both the Z-stack and MF require time-consuming slide scanning, we consider that the virtual slide system with MF performed better for cytopathological diagnosis than the conventional SF virtual slide system. The sharing of information via a standardized file format is also a problem with virtual slides for cytopathology. At present, virtual slide systems are divided into two main types: area sensor imaging systems and line sensor imaging systems. Virtual slide data created by each system are converted into an original file format. Although preparations for a common format have been advanced by DICOM Committee Working Group 26 [10], these preparations were at the stage of a draft plan as of April 2009; in addition, a draft plan has not been proposed for multi-layer virtual slides. Although virtual slides created by MF are applicable to this format, it is also necessary to consider, at this early stage, a common format for cytopathology. Komatsu et al. [11] evaluated the application of liquidbased cytology to fine needle aspiration cytology in breast cancer, reporting that the cytologic features of LBC were slightly different from those of a conventional aspiration cytology smear. In the present study, the cytopathological features were slightly different between the stamp and FNA-LBC methods, and in some cases it was difficult to evaluate malignancy on the virtual slides. Therefore, although LBC is useful in virtual slides for breast tumor diagnosis, it is necessary to extract the cytopathological features by MF. Although the problems outlined above remain to be solved, we consider that the virtual slide system with MF is useful for the digitization of cytopathology, and could also be applied to tele-cytology and e-learning [12], [13]. 5.

Conclusion

The virtual slide system with MF outperformed the conventional SF virtual slide system with regard to the ease of viewing inside cells and cell clusters and reducing image file size. LBC method was more suitable than the stamp method for virtual slides with MF. We consider that the virtual slide system with MF is a useful technology for the digitization of cytopathology, and could also be applied to tele-cytology and e-learning. Acknowledgments This research was supported by Strategic Information and Communications R&D Promotion Programme (SCOPE) of the Japan Ministry of Internal Affairs and Communications. References [1] R.S. Riley, J.M. Ben-Ezra, D. Massey, R.L. Slyter, and G. Romagnoli, “Digital photography: a primer for pathologists,” J Clin Lab Anal, vol.18, no.2, pp.91–128, 2004.

NOZAKA et al.: MULTI-LAYER VIRTUAL SLIDE SCANNING SYSTEM WITH MULTI-FOCUS IMAGE FUSION

863

[2] W.Y. Liang, C.Y. Hsu, C.R. Lai, D.M. Ho, and I.J. Chiang, “Lowcost telepathology system for intraoperative frozen-section consultation: our experience and review of the literature,” Hum Pathol, vol.39, no.1, pp.56–62, 2008. [3] S. Wienert, M. Beil, K. Saeger, P. Hufnagl, and T. Schrader, “Integration and acceleration of virtual microscopy as the key to successful implementation into the routine diagnostic process,” Diagn Pathol, vol.4, no.3, 2009. [4] G.M. Sharangpani, A.S. Joshi, K. Porter, A.S. Deshpande, S. Keyhani, G.A. Naik, A.S. Gholap, and S.H. Barsky, “Semiautomated imaging system to quantitate estrogen and progesterone receptor immunoreactivity in human breast cancer,” J Microsc, vol.226, no.3, pp.244–255, 2007. [5] A.S. Joshi, G.M. Sharangpani, K. Porter, S. Keyhani, C. Morrison, A.S. Basu, G.A. Gholap, A.S. Gholap, and S.H. Barsky, “Semiautomated imaging system to quantitate Her-2/neu membrane receptor immunoreactivity in human breast cancer,” Cytometry A, vol.71, no.5, pp.273–285, 2007. [6] R. Maruthi and K. Sankarasubramanian, “Multi focus image fusion based on the information level in the regions of the images,” J. Theoretical and Applied Information Technology, vol.3, no.4, pp.80–85, 2007. [7] E.E. Maras, B. Bayram, H.H. Maras, S.S. Maras, and B. Aktug, “Multi-focus image fusion in high precision close-range phaotogrammetry,” Int. J. Physical Sciences, vol.5, no.6, pp.763–767, 2010. [8] I. Mori, O. Nunobiki, T. Ozaki, E. Taniguchi, and K. Kakudo, “Issues for application of virtual microscopy to cytoscreening, perspectives based on questionnaire to Japanese cytotechnologists,” Diagn Pathol, vol.3, suppl.1, S15, 2008. [9] T. Kalinski, R. Zw¨onitzer, S. Sel, M. Evert, T. Guenther, H. Hofmann, J. Bernarding, and A. Roessner, “Virtual 3D microscopy using multiplane whole slide images in diagnostic pathology,” Am J Clin Pathol, vol.130, pp.259–264, 2008. [10] R. Zw¨onitzer, T. Kalinski, H. Hofmann, A. Roessner, and J. Bernarding, “Digital pathology: DICOM-conform draft, testbed, and first results,” Comput Methods Programs Biomed, vol.87, no.3, pp.181–188, 2007. [11] K. Komatsu, Y. Nakanishi, T. Seki, A. Yoshino, F. Fuchinoue, S. Amano, A. Komatsu, M. Sugitani, and N. Nemoto, “Application of liquid-based preparation to fine needle aspiration cytology in breast cancer,” Acta Cytol, vol.52, no.5, pp.591–596, 2008. [12] D.M. Steinberg and S.Z. Ali, “Application of virtual microscopy in clinical cytopathology,” Diagn Cytopathol, vol.25, no.6, pp.389–396, 2001. [13] J. Stewart, 3rd, K. Bevans-Wilkins, A. Bhattacharya, C. Ye, K. Miyazaki, and D.F. Kurtycz, “Virtual microscopy: an educator’s tool for the enhancement of cytotechnology students’ locator skills,” Diagn Cytopathol, vol.36, no.6, pp.363–368, 2008.

Hiroyuki Nozaka received the M.S. degrees in the Open University of Japan in 2005. From 1994 to 1997, he was a medical technologist at Iwate medical university. He is currently a lecturer at Graduate school of health sciences and center of joint research, Hirosaki university.

Tomisato Miura received the M.S. and Ph.D. degrees in the Hirosaki University of Japan in 1993 and 2001. He is currently an associate professor at Graduate school of health sciences and center of joint research, Hirosaki university.

Zhongxi Zheng received the M.S. degrees in the Tsinghua University of china in 1993 and Ph.D. degrees in the Hirosaki University of Japan in 2011. From 2001, he was CTO at CLALO INC. He is currently a CEO at UNIC technologies. INC. and cooperate researcher at Graduate school of health sciences and center of joint research, Hirosaki university.