Speeding up the File Access of Large Compressed NIfTI Neuroimaging Data* Zalán Rajna, Anja Keskinarkaus, Vesa Kiviniemi and Tapio Seppänen
Abstract— A method and implementation are presented to achieve a thousand fold speed-up for seeking of large files in a commonly used compressed neuroimaging data format NIfTI. Such technologies are not currently available in this research field while they would make the everyday work for hundreds of researchers and experts much smoother and faster. The method includes the creation of a novel index structure for the compressed data in order to achieve the speed-up. With random seek simulations, we demonstrate that a speed-up of over hundred up to even five thousand can be reached compared to the currently available implementations. By configuring the index structure properly, one can set an operating point which optimizes the efficiency as speed-up versus index size according to the requirements by the user. For example, a thousand fold speed-up can be achieved with an index size of only about two percent of the original compressed data.
I. INTRODUCTION Recently, the spreading of novel neuroimaging related technologies that sample data ever faster and higher spatial precision have led us to a situation where research groups working on neuroimaging data are regularly running out of disk space. For example, large neuroimaging projects like the human connectome projects (HCP) graciously share highly accurate spatiotemporal data for everyone to explore the brain all over the world. While these datasets are very accurate they also are very large and require several hard drives when obtained. Meanwhile, some fMRI-research centers have started using even higher precisions with ultrafast, 40-100 ms whole brain image sampling rates for critical, i.e. non-aliased physiological sampling of cardiorespiratory signals. These methods are opening views into brain pathology with multiple times more accurate statistical analytics. These kinds of developments require clearly new approaches to data handling and utilization in order to be usable and analyzable. The currently popular Neuroimaging Informatics Technology Initiative (NIfTI) data format [1] in research was designed over ten years ago and was not prepared in many ways for such huge data sets we have today. An optional gzip-compression is available for the format which itself *Research supported by the Academy of Finland Grant 275352, VTR and Medical Research Center of Oulu University Hospital grants. Z. Rajna, A. Keskinarkaus and T. Seppänen are with the Biomedical Engineering Research Group, Department of Computer Science and Engineering, ITEE, University of Oulu, 90014 Finland (Z. Rajna is corresponding author; e-mail addresses:
[email protected],
[email protected],
[email protected]). V. Kiviniemi is with the Oulu Functional Neuroimaging Research Group, Department of Diagnostic Radiology, MRC Oulu, Oulu University Hospital, 90029 Finland (e-mail address:
[email protected]).
978-1-4244-9270-1/15/$31.00 ©2015 IEEE
shows the overhead of the format by reducing the data size with about seventy percent. However, the speed of accessing the compressed format is unacceptably slow, especially for simple tasks like viewing the data and jumping between time points. In this work, we adapt an indexing technique previously unused in this field to dramatically speed-up random access of this compressed data format, thus making the work for a lot of users and experts less tiresome and much faster. II. ZINDEX – A NOVEL INDEXING METHOD A commonly used data format for neuroimaging data is NIfTI. It nowadays becomes necessary to use the NIfTI file format with compression considering the size of neuroimaging data to be stored and analyzed. The compression method supported by the most analysis tools is described by [2], [3] and [4]. We used the zlib format which is an open source, by patents unencumbered and commonly used implementation of these formats [5] which was also used in this work. This compression method has been exclusively used and handled as a standard for NIfTI data, and reduces the necessary disk space by about 70%. However, the performance trade-off or RAM usage for keeping data compressed is currently on an unacceptable level due to the public implementations and interfaces of the NIfTI library. This is especially true for random access properties of compressed files. An implementation initially extracting all the data to RAM would introduce size limits in practice, while being fairly unusable for large data sets (e.g. a group analysis). Therefore, in all current implementations dealing with compressed NIfTI files, the file handling is realized without indexing, which unnecessarily sacrifices the performance. Here, we propose a method to improve this situation. For our application, a strict goal was to keep the existing data compatible with the current formats. This important aspect and the given compression method exclusively used for years limited and determined our choice between random access methods. Certain implementations like BGZF described in [6] were excluded since conversion of all already existing data would be necessary. However, the existence of such an implementation is also a proof how critical random access tools become in cases where data size exceeds the comfortable amount to handle in RAM. A fast random access implementation for the compression format in RAM is publicly available as an example of the zlib library [7]. In this work, we utilized the available example creating a file format to store the in-memory built random access data. For logical and performance reasons, we keep the access points and the necessary data to start
654
uncompressing separately in an index file (.idx) and an uncompressed stream file (.idx.ucs), respectively. These two files (called "zindex") are stored on the disk in the proposed format. Therefore the index needs to be built only once, unlike in [7], which runs in RAM. In this work, we enable creating zindex for files over the size of 4 GB since this application is meant to handle large files. For this reason, the index file contains strictly the following data for each entry point at least at start and end of data.
A 64 bit little endian coded signed integer representing the position in bytes in the uncompressed stream;
a 64 bit little endian coded signed integer representing the position in bytes in the compressed stream;
and a 32 bit little endian coded signed integer holding bit position of compressed stream in reference to byte position above.
For practical reasons, the latter one is kept 32 bit instead of 8 which also enables to store additional information in later formats. Integer types are signed due to file offset representation traditions [8]. For each access point in an index file, exactly 32,768 bytes (32 KB) of raw data sequence is stored after each other in the uncompressed stream file. The 32 KB chunks of the uncompressed stream file is nearly always necessary to initialize the uncompressing algorithm at an access point. In case this data is used, the raw stream contains the uncompressed data exactly before the access point in the original data. Otherwise, for example at the starting point, the raw data sequence stored is undefined which in practice makes the uncompressed stream files byte wise likely different, even if produced from the same original file. For the sake of completeness, note, that due to compression format constraints, access points can only be added at an “end of block” created by the compression algorithm for which the size of a block is coarsely defined and depending on the actual data. Therefore, the access points are in practice created after the first “end of block” point in the compressed stream exceeding the desired access point distance. The creation of zindex files has an important parameter to adjust, the distance between the access points. From the usability point of view, this is not relevant: there exist only a few prerequisites to use the proposed zindex files for speedup: firstly, the access points in the compressed data have to be in growing order and the last access point positioned at the end of the file, and secondly, there should be the same amount of access points and uncompressed 32 KB data chunks in the index file and uncompressed stream file, respectively. From the performance point of view, the distance of access points is a crucial adjustable parameter, and accordingly evaluated in the experiments. Notice, however, that there is a performance increase regardless of this parametric choice.
III. TESTING SETUP A. fMRI data To test the actual performance improvement of using zindex files, a 10 minutes long resting-state fMRI file was tested. The data was created by a Siemens 3T SKYRA MR system with a 32-channel head coil. MREG sequence obtained from Freiburg University via collaboration with Jürgen Hennig group ([9], [10]) was utilized. MREG is a three-dimensional (3D) spiral, single-shot sequence that under-samples 3D k-space trajectory for faster imaging.[11] It samples the brain at 10-Hz frequency (TR=100 ms, TE=1.4 ms, and flip angle=25°) and offers thus about 20–25 times faster scanning than conventional fMRI which also means larger size of data recorded. The recording protocol was approved by the ethics committee of the Northern Ostrobothnia Hospital District. The size of the chosen test file in NIfTI format was about 2 GB (1,970,168,082 bytes) compressed and about 6 GB (6,104,809,824 bytes) uncompressed. B. Testing procedure The testing algorithm was the following for seek speed, i.e. accessing the data at a random point. Many random numbers were generated between the beginning and end of the original data and an output stream (file) was opened. In the runtime measurement, the following steps were repeated:
Safely opening all necessary files: for the reference method, a single compressed file, for the proposed method, the zindex files as well.
Seeking to the desired position (generated random number): without or with zindex files for reference and proposed methods, respectively.
Copying a 16 KB (16,384 bytes) chunk of data from the desired position to the opened output stream.
Proper handling of allocation and I/O errors (no errors occurred during tests though).
Closing all necessary files: for the reference method, a single compressed file, for the proposed method, the zindex files as well.
The algorithm was repeated hundred times with the reference method and average single seek speed was calculated accordingly. Since measurements were a lot faster with the proposed method, the algorithm was run thousand times for each tested access point distance and the average single seek speed for each distance was calculated accordingly. All of these tests were repeated in two completely different hardware environments. Solid state drives (SSD) are outstanding in random access performance, while hard disk drives (HDD) are superior in affordable speed and capacity. For this reason, a modern but average personal hardware environment and a workstation environment with high performance were selected for the tests with SSD and HDD, respectively, with the details in TABLE I.
655
TABLE I. Component
TEST HARDWARE CONFIGURATIONS
SSD configuration
Access point distance
HDD configuration
Zindex size (bytes)
HDD configuration Speed (1/s)
Speed-up
Efficiency (1/MB)
Computer
HP EliteBook 745 G2
Dell PowerEdge R815
4 MB
47345872
62.85
865.63
19.17
Processors
1x AMD 4 Core A10-7350B (2.1 GHz)
4x AMD 8 Core Opteron 6128 (2.0 GHz)
8 MB
23836876
32.20
443.40
19.51
16 GB
128 GB
16 MB
11967620
18.98
261.38
22.90
CentOS 6.6 (Linux RHEL 6 clone) 7 disk RAID-5 conf. of 600GB Seagate SAS disks (ST3600002SS)
32 MB
6000204
10.53
145.03
25.35
Memory Operating system Storage
Linux Mint 17.1 HP 256GB SED Opal 2 Solid State Drive (K1Z11AA)
IV. RESULTS We calculated the average single seek speed with the conventional (reference) method with the SSD and HDD configuration and they were 0.09257/s and 0.07261/s, respectively (successful runs of the measured tasks per second). The testing algorithm with the proposed method was run with zindex files for nine different access point distances from 32 MB down to 128 KB, with the evaluation results presented in TABLE II and TABLE III. We present the speed as the average number of successful runs of the test algorithm (random seek) per second for direct comparison. We also show the speed-up values defined as the speed multiplication rate of the new method compared to the reference method. The efficiency is given by the measured speed-up over the zindex size, i.e. speed multiplication for every megabyte (MB) of zindex data compared to the reference method. TABLE II. Access point distance
Zindex size (bytes)
128 KB
1118234740
256 KB
TEST RESULTS WITH SSD SSD configuration Speed-up
Efficiency (1/MB)
270.16
2918.39
2.74
629267296
427.46
4617.56
7.69
512 KB
354962888
495.48
5352.33
15.81
1 MB
185088260
316.25
3416.23
19.35
2 MB
94626168
236.15
2550.93
28.27
4 MB
47345872
114.19
1233.48
27.32
8 MB
23836876
55.91
603.97
26.57
16 MB
11967620
23.52
254.10
22.26
32 MB
6000204
13.25
143.17
25.02
TABLE III. Access point distance
Zindex size (bytes)
128 KB
1118234740
256 KB
We illustrate the performance increase (relative speed-up) with using the proposed technique in Fig. 1 and 2. In Fig. 2, also the average efficiency of the two test configurations is shown. For our implementation of creating zindex files, the access point distance of 4 MB was selected. This reaches the second highest average efficiency. The highest efficiency was not chosen because of the much less speed-up and the utilizing limitations: the practical original data size for a 32 MB of access point distance would be at least hundreds of megabytes. However, with less efficient hard disk usage, around five and two thousand fold of speed-up could be reached with SSD and HDD, respectively. It is important to observe that the desired speed-up and zindex size trade-off may be chosen freely in different applications.
Speed (1/s)
Figure 1. Random access speed-up reached with zindex data and size of zindex files for different access point distances. On the left vertical axis, the relative speed-up compared to the reference implementation is shown for the SSD and the HDD configurations. On the right vertical axis, the relative size of zindex files is shown compared to the original compressed data.
TEST RESULTS WITH HDD HDD configuration Speed (1/s)
Speed-up
Efficiency (1/MB)
98.72
1359.54
1.27
629267296
127.88
1761.14
2.93
512 KB
354962888
140.85
1939.73
5.73
1 MB
185088260
130.21
1793.24
10.16
2 MB
94626168
96.62
1330.64
14.75
Figure 2. Efficiency of zindex data for different access point distances. On the vertical axis, the relative random access speed-up per megabytes of zindex data is given.
656
V. DISCUSSION We proposed a novel indexing technique to provide a dramatic speed-up in seeking large fMRI data. As reference we used the file format without zindex, as this is how all currently available implementations handle compressed NIfTI data. The speed-up can be from hundreds-fold up to thousands-fold depending on the user-configurable setup. The test results demonstrate the practical capabilities of this novel technology in the neuroimaging research field. Software to create the above described zindex (index and uncompressed stream files) for any zlib compatible compressed file is provided by us at [12]. The neuroimaging data specific software is a modified library (libznz) to handle additional zindexes while reading NIfTI files are also provided at [12] which can be replaced within NIfTI C library [1]. To demonstrate performance and to avoid changing the interface of the libznz library created by [13], only reading such files with restricted zindex file names is possible now. If writing the file is attempted or no zindex files are present, the library automatically falls back to the legacy behavior without further notice. Please, see provided source codes at [12] for details. One of the most commonly used tools of the popular FSL neuroimaging analysis software package described in [14] is FSLView. FSL uses the NIfTI C library for I/O operations, thus FSLView can make use of our modified libznz library as well. FSLView can visualize functional MRI and diffusion MRI data and provides comfortable tools to have a look on NIfTI data for researchers and experts. Our proposed method can be later on implemented for other popular neuroimaging software packages like AFNI[15], FreeSurfer[16] or SPM[17]. For our earlier described test data jumping to a frame close to the end of the file in FSLView takes tens of seconds every time even with an SSD. However, with our new implementation, the time needed to have a look on such a frame is unnoticeable. Therefore, we assume that in various applications using NIfTI C library, the speed-up is significant and it is worth to optimize these for files improved by our method. A video illustrating the described use case is available at [12].
hope to make the everyday work for hundreds of researchers and experts much smoother and faster in their work with modern neuroimaging data. ACKNOWLEDGMENT We are grateful to Mark Adler who implemented and made available a random access tool as an example for the zlib library in 2005. The authors acknowledge the support of the Academy of Finland Grant 275352, VTR and Medical Research Center of Oulu University Hospital grants. REFERENCES [1] [2]
[3]
[4] [5] [6]
[7]
[8] [9] [10]
[11]
VI. CONCLUSION We showed that thousand folds of seek speed-up can be reached with the proposed zindex technique for large compressed NIfTI neuroimaging data. Keeping neuroimaging data compressed is nowadays essential due to the ever growing size and number of such recordings. Depending on the particular application, the access point distance of zindex files can be freely chosen, which enables setting of an optimal compromise between the index size and achieved speed-up. For example, a thousand fold speedup can be achieved with an index size of only about two percent of the original compressed data. We have made available a software implementation on the NIfTI C library by providing a modified version of libznz library. By distributing the implementation of the proposed method, we
[12]
[13] [14] [15] [16] [17]
657
“NIfTI documentation”, available at http://nifti.nimh.nih.gov/nifti1/documentation (accessed 23 March 2015), June 2004. “RFC (Request for Comments) 1950” (ZLIB Compressed Data Format Specification version 3.3), available at http://tools.ietf.org/html/rfc1950 (accessed 23 March 2015), May 1996. “RFC (Request for Comments) 1951” (DEFLATE Compressed Data Format Specification version 1.3), available at http://tools.ietf.org/html/rfc1951 (accessed 23 March 2015), May 1996. “RFC (Request for Comments) 1952” (GZIP file format specification version 4.3). Available at http://tools.ietf.org/html/rfc1952 (accessed 23 March 2015), May 1996. J. Gailly and M. Adler “Zlib general purpose compression library” (Version 1.2.8) [program code], available at http://zlib.net/ (accessed 23 March 2015), April 2013. H. Li, B. Handsaker, A. Wysoker, T. Fennell, J. Ruan, N. Homer, G. Marth, G. Abecasis and R. Durbin, “The Sequence Alignment/Map format and SAMtools”, Bioinformatics, vol. 25, no. 16, pp. 2078– 2079, Aug. 2009. M. Adler (2005) “Example of zlib/gzip stream indexing and random access” (Version 1.0) [program code], available at https://github.com/madler/zlib/blob/master/examples/zran.c (accessed 23 March 2015), Sep. 2012. “POSIX.1-2008: IEEE Std 1003.1™-2008 and The Open Group Technical Standard Base Specifications, Issue 7”, available at http://pubs.opengroup.org/stage7tc1/ (accessed 23 March 2015), 2013. H.L. Lee, B. Zahneisen, T. Hugger, P. LeVan, J. Hennig, “Tracking dynamic resting-state networks at higher frequencies using MRencephalography”, NeuroImage, vol. 65, pp. 216-222, 2013. B. Zahneisen, T. Hugger, K.J. Lee, P. LeVan, M. Reisert, H.L. Lee, J. Assländer, M. Zaitsev, J. Hennig, “Single shot concentric shells trajectories for ultra fast fMRI”, Magnetic Resonance in Medicine, vol. 68, pp. 484–494, 2012. J. Assländer, B. Zahneisen, T. Hugger, M. Reisert, H.L. Lee, P. LeVan, J. Hennig, “Single shot whole brain imaging using spherical stack of spirals trajectories”, NeuroImage vol. 73, pp. 59–70, 2013. Z. Rajna, “A fast version of libznz used in NIfTI C library: libznz with zindex” [program code], available at http://sourceforge.net/projects/libznzwithzindex/ (accessed 23 March 2015), March 2015. M. Jenkinson, “Zipped or non-zipped library” [program code], available at http://sourceforge.net/projects/niftilib/ (accessed 23 March 2015), Sep. 2004. M. Jenkinson, C.F. Beckmann, T.E.J. Behrens, M.W. Woolrich and S.M. Smith, “FSL”, NeuroImage, vol. 62, no. 2, pp. 782-790, 2012. R.W. Cox, “AFNI: software for analysis and visualization of functional magnetic resonance neuroimages”, Computers and Biomedical Research, vol. 29, no. 3, pp. 162-173, 1996. A.M. Dale, B. Fischl and M.I. Sereno, “Cortical surface analysis: I. Segmentation and surface reconstruction”, NeuroImage, vol. 9, no.2, pp. 179-194, 1999. J. Ashburner, K.J. Friston, “Unified segmentation”, NeuroImage, vol. 26, no. 3, pp. 839-851, 2005.
