Magn Reson Mater Phy (2012) 25:313–320 DOI 10.1007/s10334-011-0290-7
RESEARCH ARTICLE
A highly parallelized framework for computationally intensive MR data analysis Roland N. Boubela • Wolfgang Huf • Klaudius Kalcher • Ronald Sladky • Peter Filzmoser Lukas Pezawas • Siegfried Kasper • Christian Windischberger • Ewald Moser
•
Received: 27 May 2011 / Revised: 11 October 2011 / Accepted: 13 October 2011 / Published online: 16 November 2011 Ó ESMRMB 2011
Abstract Object The goal of this study was to develop a comprehensive magnetic resonance (MR) data analysis framework for handling very large datasets with user-friendly tools for parallelization and to provide an example implementation. Materials and methods Commonly used software packages (AFNI, FSL, SPM) were connected via a framework based on the free software environment R, with the possibility of using Nvidia CUDA GPU processing integrated for high-speed linear algebra operations in R. Three hundred single-subject datasets from the 1,000 Functional Connectomes project were used to demonstrate the capabilities of the framework. Results A framework for easy implementation of processing pipelines was developed and an R package for the example implementation of Fully Exploratory Network ICA was compiled. Test runs on data from 300 subjects demonstrated the computational advantages of a processing Roland N. Boubela and Wolfgang Huf contributed equally to this paper. R. N. Boubela W. Huf K. Kalcher R. Sladky C. Windischberger (&) E. Moser Center for Medical Physics and Biomedical Engineering, Medical University of Vienna, Wa¨hringer Gu¨rtel 18-20, 1090 Vienna, Austria e-mail:
[email protected] R. N. Boubela W. Huf K. Kalcher L. Pezawas S. Kasper Department of Psychiatry and Psychotherapy, Medical University of Vienna, Wa¨hringer Gu¨rtel 18-20, 1090 Vienna, Austria R. N. Boubela W. Huf K. Kalcher P. Filzmoser Department of Statistics and Probability Theory, Vienna University of Technology, Wiedner Hauptstraße 8-10, 1040 Vienna, Austria
pipeline developed using the framework compared to nonparallelized processing, reducing computation time by a factor of 15. Conclusion The feasibility of computationally intensive exploratory analyses allows broader access to the tools for discovery science. Keywords Magnetic resonance imaging fMRI High performance computing Statistical computing
Introduction With the development and application of new statistical methods, analysis of magnetic resonance imaging (MRI) data is becoming increasingly complex and computationally intensive. In addition, larger sample sizes and higher resolution used in more recent MR studies further increase computational demands for data analysis. Both of these trends lead to the need for more efficient computing strategies. In particular, the analysis of functional MRI (fMRI) data is a subfield that relies on a host of different methodological approaches, each specialized for particular study designs and research questions. Among these are general linear models (GLMs) for model-based analysis of fMRI time series, seed-based connectivity analyses to identify regions in the brain exhibiting similar patterns of activation over time [1, 2] and independent component analysis (ICA) as an exploratory approach to identify independent sources of activation. In practice, many analyses are not simply straightforward applications of a single procedure but consist of multiple levels. Raw data as acquired by the MR scanner have to undergo reconstruction, preprocessing [3, 4]
123
314
including motion correction and spatial smoothing, before higher-level statistical procedures such as ICA can be applied. All of these analyses, yielding only single-subject results, still have to be combined at a group level, the third level of processing. Multiple software packages are available at each level to perform the numerous tasks involved. For various reasons, investigators might prefer different software packages for different processing steps, e.g. AFNI [5] or SPM [6] for preprocessing and FSL MELODIC [7, 8] for ICA. The interface from one software package to the other, however, is often non-straightforward, and to many researchers, the barrier to including new, individually adapted procedures into the processing pipeline is prohibitive. Therefore, we herein propose a framework based on R [9] for the creation of smoothly integrated processing pipelines with the additional benefit of easy customization. The idea of this framework is to provide an application programming interface (API) aimed at MR researchers willing to implement efficient analysis pipelines for their data, possibly combining both external tools and their own custom-developed analysis routines. The R language for statistical computing was chosen as it is a powerful environment for combining and extending multiple software packages to perform all necessary steps of fMRI analysis within a single comprehensive framework. Its modular structure and the plethora of packages available on the comprehensive R archive network (CRAN; http://cran.r-project.org/) makes it particularly appealing for this purpose. One increasingly important requirement for such a framework is the capacity to handle very large datasets, since the number of studies with individual or multicenter datasets of 100 subjects or more is constantly increasing. Currently, one outstanding product of this development is the 1,000 Functional Connectomes project, which has aggregated resting-state fMRI datasets from 35 international centers including more than 1,400 subjects [1]. Due to this development, an important issue to be addressed is the efficiency of computation on datasets with large sample sizes. Currently, several studies have used parts of the 1,000 Functional Connectomes dataset [10, 11] but only a few have used the entirety of the available dataset [1, 12] probably due to the computational burden involved. In general, two things are important for large scale analyses of MR data: first, memory usage must be adapted to conform both to hardware limitations as well as to requirements of the particular analysis method. Second, processing time should not be too long, for week-long computations may negatively affect research efficiency. With current hardware development in mind, parallelization is the most promising approach for a framework to cope with these issues. On the one hand, distributing calculation to multiple machines can split memory
123
Magn Reson Mater Phy (2012) 25:313–320
requirements between computers and thus reduce the memory usage for each individual machine. On the other hand, distribution of calculations between multiple computing units helps reduce the overall time needed for the completion of the whole calculation, where the computing time needed for individual chunks of calculation cannot be arbitrarily reduced. In this paper, we present a conceptual framework for highly parallelized MR data analysis in R, including the possibility of using different software packages at different steps of the analysis. As past experience has shown, it is not advisable to attempt re-implementing the functionality of existing software packages from scratch: changes in the implementation of the original software packages would invariably obsolete their re-implementations too quickly for them to be of any practical use. The project PSPM [13], for example, tried to parallelize SPM routines, but development was discontinued at an early stage due to the impossibility of keeping up with the pace of development of the SPM software. Rather, functionalities available in one of the large analysis software packages should be fully embraced and directly called using a framework designed to automatically distribute these calls among the computing resources available. A powerful way to accelerate computations is to use graphics processing units (GPUs) for massive parallelization of simple computations. To date, this resource is best tapped using the CUDA framework [14], which provides specialized libraries that can be integrated using C and FORTRAN code. Such tailored software routines may then be accessed via R’s foreign language interface [15] . In this way, researchers can develop their analysis routines in CUDA and integrate these functions smoothly with standard calculations performed in external programs. While CUDA-enabled standard analysis functions in external software packages would certainly enhance performance, this needs to be done by the developers of the external software packages themselves and cannot be performed by third parties. We demonstrate the feasibility of our concept by implementing an analysis pipeline integrating the parallelized execution of functions of different external software packages as well as calculations within R. Finally, to assess the practical usefulness of such an analysis pipeline, we applied our framework to resting-state fMRI data [16]. This kind of data, which was a focus of a recent special issue of MAGMA [17] , is particularly suitable to illustrate the benefits of integrated processing because of the robust and generally accepted resting-state networks, which have been described in multiple publications that can be used as references for our results [18–20] . To show the advantages of parallel computing for computationally intensive group analysis algorithms, Fully Exploratory Network ICA
Magn Reson Mater Phy (2012) 25:313–320
(FENICA) [21]—a method for finding consistent spatial ICA components on a group level by combining singlesubject ICA results—was implemented and applied within the R framework to examine a sample of 300 subjects from the 1,000 Functional Connectomes database.
Materials and methods The analysis framework presented herein was developed to provide an easy-to-use environment for the analysis of large-size MR data, including calls to multiple external software packages, and distributing computing tasks to different processing units—distributing them to multiple CPU cores and other different computing devices, e.g. GPUs, as well as distributing tasks across different computers—within one analysis pipeline. The basis of the framework is the free software environment for statistical computing, R [9]. One advantage of this particular choice is the ability to call and use external programs with a command line interface directly from within R using the function system, thus offering a natural interface to existing software packages. Furthermore, the modular structure of the R system, using specialized packages for many different tasks, enables the user to benefit from a large number of pre-existing and well-tested resources (see Fig. 1). Accordingly, a number of R packages are used in the framework. First, data input from a standard MRI format (NIfTI-1 in this case) are read using the functions of the package Rniftilib [22]. The NIfTI-1 file format can also be used as an interface to external software routines used during the various steps of the analysis process. Second, the handling of big data objects is performed by the package bigmemory [23], which allows R to access data objects directly from the hard drive as opposed to the usual method of storing all objects in memory. Third, the snowfall [24] package was used to distribute computation among multiple computers as well as to parallelize calculations using multiple processor cores on a single machine. Finally, statistical computations can be performed directly in R, including the
315
possibility of using GPUs in specific functions to perform highly parallelized computations, taking advantage of the high data throughput of current graphics hardware accessible via the CUDA programming model [14]. More specifically, preprocessing is performed using one of the widely used neuroimaging analysis software packages (e.g. SPM, AFNI, FSL), and is parallelized by the framework using the R function sfapply from the snowfall package. In the sfapply call, the user specifies the analysis parameters by defining a shell command that invokes an external software package via the R function system. snowfall then creates multiple instances of R, each starting its own instance of the external software, using a seperate process for each subject, and manages the different R instances to always execute a prespecified number of processes simultaneously. For example, if preprocessing is to be performed using SPM on two computers with four processor cores each, snowfall may start eight instances of Matlab to subsequently distribute them to the eight processor cores available. The same procedure can be applied to all single-subject analyses, e.g. using FSL MELODIC for computing single-subject ICA in parallel. At the group level, analyses are most efficiently performed directly in R, building analyses on the heap of functions available in the core packages of the R distribution or in additional packages downloaded from CRAN. These group-level analyses can also be parallelized using snowfall, but it is equally possible to perform these analyses on GPUs by implementing some or all analysis functions in CUDA or by resorting to existing GPU-enabled R packages, e.g. gputools or cudaBayesreg [25, 26]. As an example, the FENICA algorithm was used to demonstrate an application of the framework. It combines single-subject independent components in a group analysis to exploratorily identify components consistent over a population of subjects [21]. After preprocessing, the data undergo two stages of analysis: first, the computation of independent components for each individual subject, and second, the comparison and combination of the resulting components into consistent group components.
Fig. 1 Schematic of the elements of the R framework presented here and their interaction. External programs can be parallelized from within R by using its internal parallelization capabilities. Calculations within R can be further parallelized by using GPU computing power
123
316
Magn Reson Mater Phy (2012) 25:313–320
For our purpose, the most important distinction is between the two analysis steps: first, the individual subjectlevel, which includes preprocessing and single-subject ICA, the main feature being the independence of the calculations for each subject from the calculations of all other subjects. In second-level analysis, these first-level singlesubject results need to be considered simultaneously in order to calculate grouped spatial maps for the common components, i.e., it is necessary to handle the calculation of three large correlation matrices requiring all single-subject results to be available during processing. Since analyses on the single-subject level can be performed independently for each subject, this stage of analysis was distributed to multiple machines, each using the data for one subject only, with no need to communicate with other processing units. This was performed using the R package snowfall which distributes the subjects between the CPUs and CPU cores on one machine as well as between multiple different computers in a cluster environment. On each computer, single-subject data were preprocessed using AFNI, and single-subject ICA was performed using FSL MELODIC in multiple instances of R, one for each subject processed on this computer. In contrast to the first level of the analysis, group-level calculations require all components from all subjects, and
access to the entire set of first-level results is therefore necessary to perform comparison and grouping operations. Moreover, the computation time is proportional to the square of the number of subjects, as opposed to the first stage where computational time increases linearly with the number of subjects. Therefore, the second level of the analysis was the main focus of this work (see Fig. 2). To handle the computational requirements of grouplevel analysis, efficient use of parallelization can help reduce overall time needed for the necessary correlation matrix calculations. Even on standard desktop hardware, calculation can be parallelized on multiple cores, but this approach is limited by memory bandwidth which, on current hardware, can feed only three to four processor cores adequately for these calculations. In order to push the possibilities for computation further on a single machine, additional hardware in the form of high memory bandwidth graphics hardware can be employed to significantly increase computational power and thus reduce the overall time needed. Indeed, in the FENICA algorithm, the calculation of the matrix of correlation coefficients between all pairs of single-subject components and average component maps is very time-consuming—not least due to the necessity of transferring petabytes of data between memory and CPU when analyzing the entire 1,000 Functional
Fig. 2 Schematic of the implementation of the FENICA algorithm in the framework. The four steps of the analysis pipeline are shown along with their interaction with the data handling package
bigmemory and external programs. For a more detailed description of the FENICA algorithm see [28]
123
Magn Reson Mater Phy (2012) 25:313–320
Connectomes dataset, this data transfer alone taking over 10 days on current hardware—and therefore can be seen as a viable target for further optimization. Due to the highly parallelizable nature of the correlation matrix computation, and due to data throughput in the computing memory being the bottleneck of these computations, the transfer of these computations from the CPU to the GPU promises to significantly reduce the time needed for the calculation of the correlation matrix. Due to the very specialized nature of GPU code optimization, gains in computation speed compared to CPU implementations are possible only with code particularly designed for GPU computing. Thus, the advantages of GPUs can be harnessed only by employing external programs including specialized GPU routines or by using GPU calculations within R. GPU-based packages are already available on CRAN, but to fully benefit from of the high memory throughput and parallelization, specialized GPU functions have to be written. To this end, two CUDA tools for fMRI calculation were implemented and included in the framework. First, the calculation of spatial (Pearson) correlations between activation maps was implemented and used in the FENICA example pipeline to compare ICA maps of different subjects. Second, a CUDA function for calculation of whole-brain t tests was written for the framework and used in the example analysis for computation of group mean maps and t statistics. It is also important to note that even the mere ability to process datasets in the order of magnitude of hundreds of gigabytes—regardless of time needed—is not to be taken for granted. On the one hand, the matrix of all component maps of all subjects as used in the FENICA algorithm easily exceeds memory size available in any single computer. On the other hand, even the number of cells in such a matrix is beyond 232 and therefore exceeds the address space of many software packages used to date. In the framework presented here, the R package bigmemory is used to comfortably enable the system to do calculations on the whole component matrix even if it does not fit into memory. To minimize the amount of (slow) hard drive access, the package bigmemory automatically performs some caching of parts of the matrix in RAM. In addition, for calculating correlation matrices, we implemented a tiling algorithm that stores parts of the input matrices in RAM or graphics memory (depending on whether the calculation at hand is performed on CPU or GPU) for calculating the largest possible tiles before new data have to be read from the hard drive [27]. As an example, an exploratory analysis of 300 subjects randomly sampled from seven centers of the 1,000 Functional Connectomes project was performed using the FENICA algorithm, 300 being the order of magnitude of the largest fMRI studies currently available. The FENICA
317
algorithm has previously been employed only on sample sizes below 30 [28], and its application to a larger sample can show the feasibility of exploratory analysis of larger fMRI studies. The software and its source code are freely available to interested individuals from the corresponding author. All analyses presented were computed on an Intel Core i7 PC (4 physical cores) with 8 gigabytes of RAM and an Nvidia GTX 285 graphics card (240 CUDA cores, 1 gigabyte of RAM) running R 2.10.1 on Ubuntu Linux 10.04.2 (Lucid Lynx).
Results The main result of our work is the concept and successful application of a comprehensive scalable framework for fMRI data analysis using R, covering various calculations from preprocessing to final statistical group analysis. Due to similarity of computational requirements of their data processing tasks, this concept can easily be extended to structural MRI and MR spectroscopy (MRS) analyses. The R framework is able to efficiently handle problems arising from the sheer size of datasets in large MR studies, taking into account both memory requirements and computation times. The basis of its efficiency is highly parallelized computation on three levels; parallelization between different personal computers, between CPU cores, and parallelization on GPUs within a single machine are implemented. Beyond accelerating analyses that have already been possible with current methods (see Table 1), Table 1 Computation time (in hours) for FENICA stage 2 depending on sample size N
Computation time (hours) No parallelization
4 CPU cores
GPU
10
0.03
0.01
0.01
20
0.08
0.03
0.02
30
0.18
0.07
0.03
40
0.32
0.12
0.03
50
0.45
0.17
0.05
60
0.65
0.25
0.06
70
0.87
0.34
0.08
80 90
1.14 1.48
0.45 0.58
0.12 0.15
100
1.84
0.73
0.19
150
3.99
1.52
0.36
200
6.91
2.64
0.63
250
11.06
4.12
0.93
300
16.01
5.76
1.23
Isotropic voxels of 3 mm in whole brain datasets of 61 9 73 9 61 voxels were used, number of subjects varying between 10 and 300
123
318
Magn Reson Mater Phy (2012) 25:313–320
algorithms on massive MR datasets such as the 1,000 Functional Connectomes dataset (see Fig. 3). The result of the example analysis is a set of 15 restingstate networks consistent in 300 subjects (see Fig. 5), which correspond to the networks identified in different individual papers including [20, 21]. Note that most components in Fig. 5 represent known brain networks, e.g. sensorimotor (components 1, 2, 3), default-mode (4, 11), visual (10), working memory (7, 9), and dorsal pathway (6) networks. However, consistent artifacts, e.g. from eye movements (15), white matter (5), or CSF signals (14), are also obtained. A full account of brain networks statistics exceeds the scope of this work and will be published
Fig. 3 Computation time depending on sample size for isotropic voxels of 3 mm in a whole brain analysis (61 9 73 9 61 voxels). The smaller inner figure shows computation time (in hours) of the test runs performed with 10–300 subjects. The larger figure shows predicted computing time (in days) for larger sample sizes up to 1,500 subjects
analysis on previously prohibitively large datasets can be tackled in a reasonable amount of time (see Fig. 4), thus showing the computational feasibility of using complex
Fig. 4 Computation time depending on voxel size. Voxels are isotropic with edge lengths given in mm, ranging from 2–5 mm in a whole brain dataset (183 9 219 9 183 mm3). Runs were performed on 30 subjects
123
Fig. 5 Resting-state networks identified in the example analysis. Results are presented as t values, thresholded at t = 18 (P = 4 9 10-44, FWE-corrected). Each row corresponds to one highly significant network showing nine slices spaced 15 mm in the z direction. Networks are both actual brain networks and consistent artifacts (e.g. component 15 showing consistent eye movements). Red color represents highest t values, images are shown in radiological convention (the right side of the brain is displayed on the left)
Magn Reson Mater Phy (2012) 25:313–320
elsewhere, using the full Functional Connectomes dataset currently available.
Discussion The conceptual framework presented here has proven useful in developing a feasible analysis pipeline for exploratory resting-state fMRI analysis by combining multiple tools for the analysis of large datasets into an easily adaptable processing tool. We compiled an R package and applied it to replicate the findings of restingstate networks [18–21] on a larger sample using a fully exploratory analysis approach. In addition to typically 6–10 networks described (where later studies tend to report more networks, including networks with frequency power spectra closer to those of artifacts [20]), 15 networks have been identified using data from 300 subjects indicating the increasing statistical power. Our implementation of the framework allows for easy use of the resources of standard workstations as well as dedicated servers. But even beyond the scope of computation on single or multiple PC workstations, the use of R as a computation framework provides readily accessible means to transfer MR processing pipelines to a supercomputing environment (which was tested on the Vienna Scientific Cluster), including the packages Rmpi [29] and snow/snowfall [24]. Here, we have presented a framework for large-scale analysis of MR datasets, lowering the barrier of entry to the field of efficient analysis utilising parallel computing for a broader community of MR-scientists. As a side effect, this study allows for the examination of a wide range of sample sizes, particularly useful for a robustness check of results produced by computationally intensive procedures such as the FENICA algorithm and sensitivity analyses in general. The newly acquired range of possibilities of performing such analyses in an acceptable time frame furthermore paves the way for computationally intensive exploratory statistical methods. Furthermore, corresponding/complementary information from genetic analyses, physiological or biological parameters may be included in such a database and analyzed (correlated) easily to identify potential biomarkers [30]. Exploratory approaches are especially useful for generating hypotheses [31] in the analysis of fMRI data where datasets are commonly several gigabytes per subject, but are also gaining importance in the analysis of structural MRI (e.g. voxel-based morphometry, diffusion tensor imaging) and MRS (e.g. 3D spectroscopic imaging, multinuclear) data due to the combination of increasing resolution with larger sample sizes available to comprehensive analysis. For instance, computation time for the analysis of multi-voxel data from chemical shift imaging (CSI) studies
319
can easily be reduced by parallelizing calls to programs that perform voxel-wise analysis (e.g. LCModel [32]) on multiple CPU cores or different computers using snowfall. In this context, one especially interesting aspect is the possibility of combining information from different modalities into one study [33], both in human and in animal studies. Additionally, the prohibitive time and memory demands of exploratory methods are alleviated by the efficiency of parallelized computing. This shows that discovery science using exploratory analysis can now be undertaken in a feasible time-frame, adding to the attractiveness of data-driven approaches on large datasets.
Conclusion We provided a framework for parallelized analysis of large MR datasets along with a conceptual implementation of the FENICA algorithm, demonstrating the feasibility of performing exploratory analysis even on a sample as large as the 1,000 Functional Connectomes dataset. The choice of R as a basis for our framework allows for easy creation of data analysis pipelines, including the ability to handle the computational demands of large datasets concerning both memory requirements and computation time, the latter being reduced to a practically acceptable level due to massive parallelization of calculations. These extended possibilities in MR data processing allow for a wider range of statistical methods to be employed, including most notably the computationally intensive exploratory methods needed for data-driven discovery science. Acknowledgments This study was funded by the Institute for the Study of Affective Neuroscience (ISAN), by the Oesterreichische Nationalbank (OeNB P13890, OeNB P13903, OeNB P12982) and by the Austrian Science Fund (FWF) as part of the Special Research Program 35 (SFB-35). The computational results presented have been achieved in part using the Vienna Scientific Cluster (VSC).
References 1. Biswal BB, Mennes M, Zuo XN, Gohel S, Kelly C, Smith SM, Beckmann CF, Adelstein JS, Buckner RL, Colcombe S, Dogonowski AM, Ernst M, Fair D, Hampson M, Hoptman MJ, Hyde JS, Kiviniemi VJ, Ko¨tter R, Li SJ, Lin CP, Lowe MJ, Mackay C, Madden DJ, Madsen KH, Margulies DS, Mayberg HS, McMahon K, Monk CS, Mostofsky SH, Nagel BJ, Pekar JJ, Peltier SJ, Petersen SE, Riedl V, Rombouts SARB, Rypma B, Schlaggar BL, Schmidt S, Seidler RD, Siegle GJ, Sorg C, Teng GJ, Veijola J, Villringer A, Walter M, Wang L, Weng XC, Whitfield-Gabrieli S, Williamson P, Windischberger C, Zang YF, Zhang HY, Castellanos FX, Milham MP (2010) Toward discovery science of human brain function. Proc Natl Acad Sci USA 107:4734–4739 2. van den Heuvel MP, Pol HEH (2010) Exploring the brain network: a review on resting-state fMRI functional connectivity. Eur Neuropsychopharmacol 20:519–534
123
320 3. Weissenbacher A, Kasess C, Gerstl F, Lanzenberger R, Moser E, Windischberger C (2009) Correlations and anticorrelations in resting-state functional connectivity MRI: a quantitative comparison of preprocessing strategies. Neuroimage 47:1408–1416 4. Sladky R, Friston KJ, Tro¨stl J, Cunnington R, Moser E, Windischberger C (2011) Slice-timing effects and their correction in functional MRI. Neuroimage 58:588–594 5. Cox RW (1996) AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Comput Biomed Res 29:162–173 6. Penny, WD, Friston, KJ, Ashburner, JT, Kiebel, SJ, Nichols, TE (eds) (2007) Statistical parametric mapping: the analysis of functional brain images. Academic Press, London 7. Beckmann CF, Smith SM (2004) Probabilistic independent component analysis for functional magnetic resonance imaging. IEEE Trans Med Imaging 23:137–152 8. Beckmann CF, DeLuca M, Devlin JT, Smith SM (2005) Investigations into resting-state connectivity using independent component analysis. Philos Trans R Soc Lond B Biol Sci 360: 1001–1013 9. R Development Core Team (2011) R: a language and environment for statistical computing. ISBN 3-900051-07-0 10. Tomasi D, Volkow ND (2011) Gender differences in brain functional connectivity density. Hum Brain Mapp. doi:10.1002/ hbm.21252 11. Rubinov M, Sporns O (2011) Weight-conserving characterization of complex functional brain networks. Neuroimage 56:2068– 2079 12. Tomasi D, Volkow ND (2010) Functional connectivity density mapping. Proc Natl Acad Sci USA 107:9885–9890 13. Parallel SPM. http://sourceforge.net/projects/parallelspm/ 14. NVIDIA Corporation (2011) CUDA API Reference Manual 15. R Development Core Team (2011) Writing R extensions. R foundation for statistical computing. Austria, Vienna 16. Lowe MJ (2010) A historical perspective on the evolution of resting-state functional connectivity with MRI. Magn Reson Mater Phy 23:279–288 17. Moser E, Ranjeva JP (2010) In vivo MR imaging of brain networks: illusion or revolution?. Magn Reson Mater Phy 23: 275–277 18. Damoiseaux JS, Rombouts SARB, Barkhof F, Scheltens P, Stam CJ, Smith SM, Beckmann CF (2006) Consistent resting-state networks across healthy subjects. Proc Natl Acad Sci USA 103: 13848–13853
123
Magn Reson Mater Phy (2012) 25:313–320 19. Fox MD, Snyder AZ, Vincent JL, Corbetta M, Essen DCV, Raichle ME (2005) The human brain is intrinsically organized into dynamic, anticorrelated functional networks. Proc Natl Acad Sci USA 102:9673–9678 20. Robinson S, Basso G, Soldati N, Sailer U, Jovicich J, Bruzzone L, Kryspin-Exner I, Bauer H, Moser E (2009) A resting state network in the motor control circuit of the basal ganglia. BMC Neurosci 10:137 21. Scho¨pf V, Kasess CH, Lanzenberger R, Fischmeister F, Windischberger C, Moser E (2010) Fully exploratory network ICA (FENICA) on resting-state fMRI data. J Neurosci Methods 192:207–213 22. Granert O (2010) Rniftilib: Rniftilib—R Interface to NIFTICLIB (V1.1.0). R package version 0.0-29 23. Kane MJ, Emerson JW (2010) bigmemory: manage massive matrices with shared memory and memory-mapped files. R package version 4.2.3 24. Knaus J (2010) snowfall: easier cluster computing (based on snow). R package version 1.84 25. da Silva ARF (2010) cudaBayesreg: Bayesian computation in CUDA. The R Journal 2/2:48–55 26. da Silva ARF (2011) A Bayesian multilevel model for fMRI data analysis. Comput Methods Programs Biomed 102:238–252 27. NVIDIA Corporation (2011) NVIDIA CUDA C Programming Guide 28. Scho¨pf V, Windischberger C, Robinson S, Kasess CH, Fischmeister FPhS, Lanzenberger R, Albrecht J, Kleemann AM, Kopietz R, Wiesmann M, Moser E (2011) Model-free fMRI group analysis using FENICA. Neuroimage 55:185–193 29. Yu H (2010). Interface (wrapper) to MPI (message-passing interface). Package version 0.5-9 30. Eidelberg D (2009) Metabolic brain networks in neurodegenerative disorders: a functional imaging approach. Trends Neurosci 32:548–557 31. Huf W, Kalcher K, Pail G, Friedrich ME, Filzmoser P, Kasper S (2011) Meta-analysis: fact or fiction? How to interpret metaanalyses. World J Biol Psychiatry 12:188–200 32. Provencher SW (1993) Estimation of metabolite concentrations from localized in vivo proton NMR spectra. Magn Reson Med 30:672–679 33. Damoiseaux JS, Greicius MD (2009) Greater than the sum of its parts: a review of studies combining structural connectivity and resting-state functional connectivity. Brain Struct Funct 213: 525–533
