VANO: A Volume-Object Image Annotation System Hanchuan Peng, Fuhui Long, and Eugene W. Myers Janelia Farm Research Campus, Howard Hughes Medical Institute, Ashburn, Virginia, USA. Email:
[email protected] SUPPLEMENTARY DOCUMENTATION
ABSTRACT
VANO is a cross-platform image annotation system that enables one to conveniently visualize and annotate 3D volume objects including nuclei and cells. An application of VANO typically starts with an initial collection of objects produced by a segmentation computation. The objects can then be labeled, categorized, deleted, added, split, merged, and redefined. VANO has been used to build high-resolution digital atlases of the nuclei of C. elegans at the L1 stage and the nuclei of D. melanogaster’s ventral nerve cord at the late embryonic stage. Availability: Platform independent executables of VANO, a sample dataset, and a detailed description of both its design and usage are available at: research.janelia.org/peng/proj/vano. VANO is open-source for co-development. 1
INTRODUCTION
Image content annotation is one of the basic problems in analysis and mining of threedimensional (3D) high-resolution cellular and molecular images (Peng, 2008). There are several pieces of existing work on developing automatic image-annotation systems for categorizing gene expression patterns (Zhou and Peng, 2007) or predicting cell identities (Long et al, 2008a). Remarkably, to develop all these automatic annotation tools, a critical prerequisite is to have a database of experts’ domain knowledge about the 3D image patterns. Thus we will first need a high-performing tool to input the expert knowledge. We have developed VANO, short for Volume-object Annotation system, to visualize and annotate hundreds of automatically segmented cells, nuclei, and other 3D image objects in cellular images of model organisms including Caenorhabditis elegans (C. elegans), fruit fly (Drosophila melanogaster), etc. In short, VANO is a cross-platform 3D annotator, which provides a spreadsheet of all segmented 3D image objects linked to both the 3D view of the raw image and that of the segmentation mask. This tool enables a user to quickly add or edit the annotations such as cell names/properties in images, and editing the segmentation results such as adding or removing segmented objects. VANO has been used to build digital atlases of C. elegans (Long et al, 2008b) and fruit fly (collaborations with labs of Doe, C., Rubin, G., Simpson, J., et al).
1
Image visualization tools such as Amira (amiravis.com) allow painting of different image objects or refining the boundary of an object. Other image annotation tools, (e.g. the annotator of Open Microscope Environment, www.openmicroscopy.org), may also be used to input text descriptions, but generally not at the single object level, into a database. Different from the existing tools, VANO has the ability to annotate a virtually unlimited number of 3D image objects in a highly interactive and convenient way. 2 2.1
DESIGNS Major GUI components
VANO is designed to maximize the accessibility of the image object contents with a very simple and intuitive GUI, which is arranged as four parts (Fig. 1): 3D image viewer panels, control panel, annotation table viewer, and various dialogs. First, as shown in Fig. 1 (a), there are three image viewer panels for a “subject image”, which is to be annotated. These three panels correspond to the XY, ZY, and XZ cross-sectional planes of the current focus point, indicated by the intersection of dashed lines, drawn on top of the image. This “tri-view” layout is also used for the side-by-side displayed “mask image”, which defines the set of image objects in the subject image. Therefore there are a total of six image viewer panels, which are always anchored together – when the focus point changes, all the 3D views will be altered accordingly. Second, both to the right-side of and underneath the image viewers in Fig. 1 (a), there are many control widgets, which control the focus location, zoom, color channel display, and other functions. For example, VANO implements a special dual-looking-glasses function, which allows simultaneous comparison of local 3D zoom-ins of different image locations. Third, VANO has a build-in spreadsheet called annotation table viewer (Fig. 1 (b)), which displays the annotation information of all image objects together. The 3D image viewers and the annotation viewer can mutually control each other, making it very easy to access image objects and to update the respective annotations. See Section 3.1 for more details. Fourth, VANO has various dialogs to edit the annotations or image objects, such as the pop-up dialog to input annotation for a single object (Fig. 2 (c)). Besides these four major GUI components, VANO has a number of other features such as the pop-up labeler for displaying annotation information without clicking any button, multiple buildin colormaps, etc. The help information contains helpful short-keys and special usage of the software. It is summarized as a one-page of instructions, which can be accessed via the control panel. To maximize the usability of VANO GUI, we implement VANO based on platform-independent QT (trolltech.com) and C/C++, thus VANO runs on all major computer systems.
2
2.2
Data management
VANO manages the image data and associated annotations using four files, i.e. subject image, mask image, annotation file, and linker file.
(a)
(b) Fig.1 GUI of VANO and application of VANO in annotating 3D C. elegans image stacks. (a) The six image viewers (top left), including three for the subject image (in true-color) and three for the mask image (in pseudo-color), and the control panel containing many widgets. The displayed images are confocal image stacks of a C. elegans larva. Raw images: by Xiao Liu and Stuart Kim. The worm body in this stack was straightened using an algorithm in (Peng, et al, 2008). (b) The annotation table viewer, which is a spreadsheet. Each row is an annotation entry for one cell (or other image objects). The columns contain various properties and annotations of an image object.
Both the “subject image” and the “mask image” are 3D image stacks. The “mask image” is an indexed label field, containing information that defines the image objects in the subject image. If at the 3D location (x,y,z) the mask image voxel has the value k, then it means the voxel (x,y,z) in the subject image belongs to the kth image object. When k equals 0, the respective image region is the background. The “annotation file” contains plain text annotation information of all image objects, each of which corresponds to one row in the annotation file. A typical annotation entry has “standard” columns, including the object order, label, standard name, user comments for an image object, the 3D coordinates of the center of this object, the average, standard deviation, and peak voxel
3
intensity of this object collectively. These could indicate the level of gene expression of a cell or associated protein abundance, the size (number of voxels) and mass of this object. The annotation columns can also be customized (see Section 3.4). The annotation file is stored in the simple CSV format (comma separated values), and thus can be easily accessed using other software such as Matlab, Microsoft Excel, or any text editor. It can also be organized into a database effortlessly. Finally, the linker file contains three lines, each points to one of the subject image, mask image and annotation file. In this way, all these files are managed in a convenient yet flexible way. Opening the linker file using VANO will open the subject image, mask image and the annotation file all at once. 3
USAGE
VANO can be used for two different tasks, namely 3D annotating the image objects and correcting 3D image segmentation errors. It also has special usage such as annotation without a mask image, and customization of annotation columns. 3.1
Inputting annotations
There are two modes to input annotations, i.e. from image viewers to annotation table, and from annotation table to image viewers. First, a user can mouse-click anywhere in the six image viewers of subject/mask image. If there is an existing image object at the clicked 3D location, indicated by the respective pixel index value of the mask image, a “Cell Property” dialog will be activated for that detected image object, with which the cell name, user comments, as well as other attributes such as the distribution and shape of the green fluorescence protein expression of the image object. Once the annotation has been entered, the spreadsheet summary of the respective entry is also updated. Second, a user can click any entry in the spreadsheet annotation table to directly input annotation; at the same time, VANO will change the 3D focus and synchronize the currently displayed image contents in all the six image viewers. In this way, a user can always see the respective image while accessing the annotation entry. The combination of these two different input modes make it very convenient to visualize the annotation entries of all image objects, without the risk to leave any image object un-annotated when there are hundreds or thousands of them. 3.2
Correcting segmentation errors
VANO lets a user to easily add, remove and edit image objects in 3D, and respectively updates the spreadsheet annotation table. When the ‘E’ (short for “Edit”) key is pressed while any of the image viewer windows are activated, a pop-up dialog window will allow editing the existing cell at the respective location, or adding a new cell if there is no existing cell at that 3D location. The editing functions include (1) deleting this cell, (2) merging this cell with neighboring cells, (3) splitting this cell as multiple ones, (4) Re-segmenting the cluster of this cell and its neighbors, 4
(5) re-centering and (6) re-sizing this cell. While VANO currently does not provide complicated painting functions to permit a user brush an image object with complicated shape, the aforementioned set of image object editing functions are sufficient for editing and correcting segmentation errors for complex 3D image stacks of globular (and similar) image objects, like cells and nuclei. Note, editing segmentation in the general case requires the ability to paint arbitrary sets of pixels. This is extremely tedious and not really scalable to our scenario. Fortunately, the objects of interest to us are always globular and sufficiently modeled by a sphere. So while VANO supports an arbitrary mask produced by a segmentation program, the simple manual edits aforementioned are indeed very powerful. The restriction to spheres has not been found to be limiting and if somewhat more tailored masks are desired we find combining 2-4 spheres generally suffices to give a ‘snug’ mask. For correcting segmentation error, VANO has two additional features. First, VANO never deletes an annotation entry of an image object, even if this object is deleted in the image viewer; instead, VANO will just mark this image object as “NOUSE” in the spreadsheet, so that the segmentation error statistics can be easily collected later. Second, VANO always stores a hiddencopy of the original mask image, so that even image objects have been edited in an arbitrary way, an user is always able to switch back and forth between the views of the original mask image, where all the initial image objects are visible, and the current mask image, where some image objects have been changed. In this way, a user is always able to recover the critical segmentation information, if some editing goes wrongly. 3.3
Annotation without a segmentation mask image
Usually the mask image can either be automatically produced by a 3D image segmentation program (Long et al 2007), or can be generated only once by an expert. It will serve as a “standard template” that is shared by many different subject images, when all these subject images are registered to a common template. However, VANO provides the function to open and annotate a subject image without an existing mask image. In this case, VANO automatically generates a new linker file, as well as a blank mask image and an empty annotation file. The user is able to add cells and then edit them. In this way, a user may manually segment and annotate an entire 3D image stack without invoking an automatic 3D image segmentation program. This function is suitable for situations where it is computationally difficult to automatically produce such a mask image. 3.4
Customization of the annotation table columns
Besides the standard list of columns used in annotation table viewer, VANO can be customized to edit additional annotation terms, by editing a plain-text schema file called VANO.schema. This schema file serves as a template of the content columns in the annotation table. For example, for annotating the neuronal expression of various fruit fly GAL4 lines, we have considered adding three annotation columns. They describe the intensity, spatial distribution, and shape of neuronal patterns in the compartment of an adult fly brain, where each compartment is a separate image object. This customization is achieved by simply adding three new keywords, as well as the associated categories, in the schema file.
5
(a)
(b)
(c)
Fig.2 Application of VANO in annotating neuronal patterns of fruit fly brain. (a) The “subject image”, which contains NC82 (red) neuropile stains and GFP (green) stain of the atonal. Raw data from Julie Simpson. (b) The “mask image”, which contains a number of compartments in the brain, each has a different color. The label field is from Arnim Jenett. (c) The “Cell Property” dialog which will be popped-up when an image object in the image viewers is activated.
4
APPLICATIONS
VANO allows one to conveniently create or efficiently edit a segmentation and annotation of a collection of globular objects. As such it played a critical support role in building a 3D digital nuclei atlas for C. elegans (Fig. 1 (a) and (b); collaboration with Stuart Kim lab at Stanford) and a 3D digital atlas of the nuclei of the ventral nerve cord of late stage D. melanogaster embryos (collaboration with Chris Doe lab at Oregon). For these two applications, VANO was used to produce the training sets for our automated algorithms (Long, et al, 2007 and 2008a), to assess the quality of the 3D segmentations and annotations of nuclei produced by these algorithms, and to correct errors wherever needed to produce near perfect reference results. VANO can also be used to annotate images in contexts where the image objects have an irregular shape with the caveat that one cannot edit the segmentation in such instances. For example, we have been using VANO to annotate the enervation pattern of target neurons in defined compartments of the adult brain of D. melanogaster (collaborations with Gerry Rubin and Julie Simpson labs at Janelia Farm Research Campus). In Fig. 2, each glia-isolated compartment is defined as a separate object with a complicated 3D shape. VANO enables us to conveniently interact with these 3D objects and annotate the neuronal distribution pattern within each compartment. VANO can be used in the many situations in biomedical imaging where the annotation of automatically segmented images is desired.
6
5
FUTURE WORK
One very useful enhancement of VANO is to incorporate the cell identity prediction algorithm we developed (Long, 2008a), which automatically pre-annotate all nuclei/cells in a newly generated 3D image stack. This will dramatically improve efficiency of manual annotation. Another future work is to combine both VANO and the automatic Image pattern AnNOtation (IANO) tool (Zhou and Peng, 2007), which automatically classifies and predicts the ontological/anatomical categories of image patterns. ACKNOWLEDGMENTS
We thank Xiao Liu and Stuart Kim for feedback in annotating C. elegans data, Mike Layden, Ellie Heckscher and Chris Doe for feedback in annotating fruit fly embryo data, aRnim Jenett, Gerry Rubin, and Julie Simpson for feedback on annotating fruit fly adult brain data, and Frank Midgley for feedback on the design of VANO. We thank Zongcai Ruan for assistance in writing batch-compiling scripts and Margaret Jefferies for editorial review of the manuscript. REFERENCES Long,F., Peng,H., and Myers, E. (2007) Automatic segmentation of nuclei in 3D microscopy images of C. elegans, in Proc. ISBI'2007, 536-539. Long,F., Peng,H., Liu,X., Kim,S., and Myers,E.W. (2008a) Automatic recognition of cells (ARC) for 3D images of C. elegans, Lecture Notes in Computer Science: Research in Computational Molecular Biology, Springer Berlin / Heidelberg, 128-139. Long,F. Peng, H., Liu, X., Kim, S., and Myers, E.W., (2008b) A 3D digital atlas of C. elegans and its application to single-cell expression analysis. HHMI JFRC Technical Report. Peng,H., (2008) "Bioimage informatics: a new area of engineering biology," Bioinformatics, 24(17): 1827-1836. Peng,H., Long, F., Liu X., Kim, S., and Myers, E.W., (2008) Straightening Caenorhabditis elegans images, Bioinformatics, 24(2): 234-242. Zhou,J., and Peng,H., (2007) Automatic recognition and annotation of gene expression patterns of fly embryos. Bioinformatics, 23(5), 589-596.
7
