ATHENA Optimization: The Effect of Initial Parameter Settings across Different Genetic Models Emily R. Holzinger, Scott M. Dudek, Eric C. Torstenson, and Marylyn D. Ritchie Center for Human Genetics Research, Department of Molecular Physiology & Biophysics, Vanderbilt University, Nashville, TN, USA
[email protected], {dudek,torstenson,ritchie}@chgr.vanderbilt.edu
Abstract. Rapidly advancing technology has allowed for the generation of massive amounts data assessing variation across the human genome. One analysis method for this type of data is the genome-wide association study (GWAS) where each variation is assessed individually for association to disease. While these studies have elucidated novel etiology, much of the variation due to genetics remains unexplained. One hypothesis is that some of the variation lies in gene-gene interactions. An impediment to testing for interactions is the infeasibility of exhaustively searching all multi-locus models. Novel methods are being developed that perform a non-exhaustive search. Because these methods are new to genetic studies, rigorous parameter optimization is necessary. Here, we assess genotype encodings, function sets, and cross-over in two algorithms which use grammatical evolution to optimize neural networks or symbolic regression formulas in the ATHENA software package. Our results show that the effect of these parameters is highly dependent on the underlying disease model.
1 Introduction One of the main goals of modern human geneticists is to identify genetic variants that alter risk for common, complex disease. The realization of this goal has been facilitated by the rapidly advancing genotyping technologies that measure common variation across the human genome. The most popular statistical technique used to analyze this influx of data has been single-locus genome-wide association studies (GWAS). In GWAS, each variant is individually tested for association to disease. These studies have successfully elucidated novel genetic architectures for many complex human diseases [1, 2]. However, much of the estimated variability in disease state attributable to genetics, or the heritability, remains elusive. One hypothesis is that some of the missing heritability lies in gene-gene, or epistatic, interactions. [3, 4]. Due to the computational burden of exhaustively testing all possible multi-locus models, many scientists have begun to focus on developing novel data-mining techniques. Several of these methodologies utilize machine learning algorithms to stochastically search for genetic models that accurately predict a phenotypic state [5]. In order to adapt these methods to the analysis of genetic data, rigorous optimization of parameter settings across different multi-locus interaction models is essential. In this study, we perform a parameter sweep across different initial values for two machine C. Pizzuti, M.D. Ritchie, and M. Giacobini (Eds.): EvoBIO 2011, LNCS 6623, pp. 48–58, 2011. © Springer-Verlag Berlin Heidelberg 2011
ATHENA Optimization: The Effect of Initial Parameter Settings
49
learning algorithms that operate within the Analysis Tool for Heritable and Environmental Network Associations (ATHENA): Grammatical Evolution Neural Networks (GENN) and Grammatical Evolution Symbolic Regression (GESR). We then test the effects of the parameter value combinations across various disease models to determine optimal parameter settings. 1.1 Epistasis in Complex Human Disease Epistasis is both a biological and a statistical concept. In strict biological terms, epistasis is one gene altering the effect of another via physical interaction [6]. Statistically, we attempt to detect biological epistasis by measuring deviations from additivity. Epistasis has long been accepted as playing an important role in the effect of genetic variation on phenotype [7]. It is logical then to conclude that a portion of the missing heritability in complex human diseases is attributable to interaction effects that would be missed by methods that only test for marginal effects. There are several hindrances to testing for interactions using traditional GWAS methods such as linear or logistic regression. First, as mentioned in the previous section, exhaustively testing all multi-locus interactions in data that contain hundreds of thousands of independent variables is computationally impractical. Second, due to the complexity of biology, it is impossible to pre-specify how the independent variables are modeled. Third, the data becomes sparse as the number of loci in the interaction model increases. This is commonly referred to as “the curse of dimensionality [8].” Finally, when performing multiple comparisons in a parametric analysis, the p-value required to reject the null hypothesis must be altered to correct for the increased probability of making a Type I error with each additional test [5]. In an interaction analysis, the number of comparisons explodes, greatly reducing the test’s ability to detect a true positive. Collectively, these issues have prompted many genetic researchers to develop novel methodology. Machine learning algorithms have become one area of interest in this field due to their stochastic data-mining nature. Although these methods are nonparametric, it is unlikely that initial parameter settings will perform equally given different underlying disease models [9]. It will be important in going forward with these algorithms to carry out rigorous optimization to determine the specific parameters that have the greatest impact on loci detection across various genetic models. 1.2 Optimization of Neural Networks and Symbolic Regression Formulas Neural networks (NN) were originally designed to mimic the functioning of a neuron in order to take advantage of the brain’s ability to process information in parallel. Modern scientific research utilizes NNs as a statistical method to detect patterns in the data that accurately predict an outcome of interest. In brief, a NN accepts independent variables as inputs, multiplies each variable by a coefficient, and then processes the weighted inputs according to some function to generate an output signal [10]. Traditionally, the feed-forward NN is trained using a gradient descent algorithm such as back-propagation (BPNN). BPNN optimizes NNs by randomly initializing the weights and adjusting the values with each run in order to minimize an error function [11]. BPNN is not sufficient optimization, however, if the appropriate architecture of the network is unknown. In genetic studies of complex disease, this is virtually always the
50
E.R. Holzinger et al.
case. In order to overcome the issue of finding the appropriate architecture, Koza and Rice proposed a method that applies genetic programming to the optimization of both the weights and structure of the NN [12]. Genetic programming of neural networks (GPNN) was developed specifically for application to genetic association studies [13]. Symbolic regression (SR) is a method that utilizes evolutionary computation to search for a mathematical formula that maps a set of input variables to an output with minimal error. This is different from linear or logistic regression which only search for the coefficients in a pre-specified model [14]. SR has been successfully applied as a data mining technique in other scientific domains. For example, Schmidt and Lipson were able to re-discover certain laws of physics using SR analysis of raw experimental data [15]. Symbolic discriminant analysis (SDA), a method very similar to SR, uses GP to mathematically model patterns in the data that discriminate between values of a dichotomous outcome [16]. This method has been applied primarily to microarray data to mine gene expression patterns that predict disease outcome. Our lab incorporates the application of both NNs and SR to genetic data analysis via the ATHENA software package [17]. Briefly, ATHENA is a multi-functional tool designed to execute the three main functions essential to determining the genetic architecture of complex diseases: 1. performing variable selection from categorical or continuous independent variables, 2. modeling main and interaction effects that best predict categorical or continuous outcome data, and 3. interpreting the significant models for use in further biomedical research. Grammatical Evolution Neural Networks (GENN) and Grammatical Evolution Symbolic Regression (GESR) are two methods available in ATHENA that operate to primarily to achieve the second of the above listed goals. Both methods use grammatical evolution, a variation on genetic programming, to optimize NNs or SRs, respectively. Grammatical evolution has previously been described in detail [18]. A number of parameter optimization studies have been done for NN and SR based methods. A parameter optimization sweep in GENN using simulated data showed that genotype encoding, activation node function set, population size, and running the algorithm in parallel using the “island model” all influenced detection of the functional model [19]. Another study in SDA showed that more complex function sets resulted in improved performance by reducing the model size and, in some instances, reducing classification error [20]. Additionally, a parameter optimization study using biological data that consisted of polymorphisms in a specific gene and disease status showed that tree depth, function set, and population size all had a significant impact on the accuracy of the SDA models [9]. In light of these findings, the goal of this study is to assess the specific effects of different function sets, genotype encodings, and cross-over strategies on the performance of GENN and GESR across various simulated genetic models. We will also compare GENN and GESR to examine the effect of the different modelling strategies on loci detection.
2 Methods 2.1 Data Simulation All data sets were simulated using genomeSIMLA, a forward-time population-based method previously described in detail [21, 22]. The details of our data simulation are
ATHENA Optimization: The Effect of Initial Parameter Settings
51
Table 1. Characteristics of the simulated data sets Parameter Allele frequencies Number of SNPs Cases/controls Heritability Interaction Models
Value(s) 0.4 / 0.6 25 (2 functional / 23 non-functional) 1000 / 1000 0.01, 0.05, 0.1 1. Small additive interaction/ modest marginal effects 2. Modest additive interaction/ modest marginal effects 3. Non-additive interaction/ modest marginal effects 4. Non-additive interaction / small marginal effects
Table 2. Penetrance functions for h2 = 0.05. Each cell denotes probability of disease given genotype. Genotype frequencies used to calculate the marginal penetrance (MP) values are: AA/BB = 0.36; Aa/Bb = 0.48; aa/bb = 0.16.
shown in Table 1. In summary, we generated data sets with 1000 unaffected controls (i.e. people without disease) and 1000 affected cases (i.e. people with disease), each with genotype values for 25 single nucleotide polymorphisms (SNPs). Each SNP can take on one of three values representing the presence of 0, 1, or 2 minor alleles at that locus. Two of the 25 SNPs were functional components of the disease model and had a forced minor allele frequency of 0.4. The 23 non-function SNPs had minor allele frequencies between 0.1 and 0.5. We simulated the data using twelve different penetrance functions of three broad sense heritabilities (or the proportion of outcome variability attributable to all genetic effects) and four different epistatic models. The penetrance tables for the four different epistatic models under our medium effect size (h2 = 0.05) are displayed in Table 2. These values represent the probability of having the disease given a particular genotype at the two functional loci. Here, A/B represent the major alleles and a/b represent the minor alleles for the two funcional SNPs. The genotype frequencies were calculated under the assumption of Hardy-Weinberg
52
E.R. Holzinger et al.
equilibrium. The tables for the other two heritabilities are available upon request from the authors. For each of the twelve models, we generated 100 data sets in order to calculate detection power across different parameter settings. Here we define detection power as the number of data sets out of 100 that only the two functional loci were selected as the best model, and no false positive loci were included. 2.2 Data Analysis For the optimization analysis, we evaluated multiple specific values for three different initialization parameters in both GESR and GENN. The details of the parameter sweep are shown in Table 3. GENN and GESR utilize the same algorithm to search for solutions with variations in the production rules allowing for the generation of either SR formulas or NNs. The steps of the algorithm are as follows: 1. The data set is divided into 5 equal parts for 5-fold cross-validation (4/5 for training and 1/5 for testing). 2. Training begins by generating a population of random binary strings initialized to be functional NNs or SR formulas using sensible initialization [18]. The population is divided across nodes into sub-populations, or demes, on a computer cluster to allow for parallelization. 3. The NNs or SR formulas in the population are evaluated on the training data, and balanced accuracy, or fitness, is recorded. The best solutions are selected for crossover and reproduction, and a new population is generated. 4. Step 3 is repeated for a pre-defined number of generations. For each generation, the newly “evolved” population is tested on the training data with an optimal solution being selected. Migration of best solutions occurs between demes every nnumber of generations, as specified by the user. 5. The overall best solution across generations is run on 1/5 of the data left out for testing, and prediction accuracy is recorded. 6. Steps 2-5 are repeated 5 times, each time using different 4/5 of data for training and 1/5 for testing. The best model is defined as the model that was chosen the most over the 5 cross-validations. Ties are decided based on prediction accuracy. All of the aforementioned optimization studies found the function set to play an important role in the overall performance of the algorithm [9, 19, 20]. For this reason, we decided to assess simple and complex function sets in GESR as well as additiononly and all-function activation nodes in GENN. In the GENN optimization analysis, it was demonstrated that genotype encoding played a large role in detection power. Specifically, the detection power of GENN went up substantially with the linear component of the dummy-encoding method developed for NN analysis in genetic studies [23]. The simulated disease model in this analysis consisted of an additive interaction effect and additive marginal effects that predicted a quantitative outcome. For our study, we wish to find out if this finding is robust to different disease models and to a dichotomous outcome. We are also interested to determine the effect of genotype encoding in GESR, as it has not yet been tested.
ATHENA Optimization: The Effect of Initial Parameter Settings
53
Table 3. Initialization parameter values for optimization runs Parameter GE Parameters Demes Population size / deme Generations Migrations Probability of cross-over Probability of mutation Fitness Metric Function Sets GESR GENN Genotype Encoding (AA, Aa, aa) Cross-over Strategy
Value(s) 5 200 100 4 (every 25 generations) 0.9 0.01 Balanced accuracy = (sensitivity + specificity) / 2 Simple {+, -, *, /} Complex {+, -, *, /, sin, cos, tan, log, ^} All {+, -, *, /} Add-only {+} None (0,1,2) Dummy-encoding (-1, 0, 1) and (-1, 2, -1) Linear Dummy (-1, 0, 1) Single-point binary cross-over (SBPXO) Tree-based cross-over (TBXO)
Finally, we will be examining the effect of two different types of cross-over—treebased cross-over (TBXO) and single-point binary cross-over (SBPXO). Traditionally, GE uses SBPXO where cross-over occurs at the level of the binary string. However, this can potentially be destructive to the overall NN or SR formula functionality [24]. Alternatively, TBXO translates the binary string into a NN or SR formula, allowing for cross-over to occur between functionally similar parts of the model. This type of cross-over makes the algorithm more similar to a GP-based method [25]. One previous study found no significant difference in power between the two cross-over strategies under disease models with no marginal effects [26]. Another study, however, found that using TBXO yields improved sensitivity under a disease model with additive main and additive interaction effects for quantitative outcomes [27]. We want to assess the impact of the two cross-over strategies across several different disease models.
3 Results The results from our parameter optimization analysis are presented in Figure 1. Table 4 defines the abbreviations shown on the x-axis. Our results indicate that the effect of different parameter combinations on detection power is highly variable across underlying disease models. First, we show that under an additive interaction effect (Models 1 and 2), TBXO confers higher detection power overall than SPBXO. However, when the interaction effect is non-additive (Models 3 and 4), SPBXO has an advantage, especially when complex functions and full dummy-encoded genotypes are used. This effect is most pronounced when the marginal effects are very small (Model 4).
54
E.R. Holzinger et al.
Fig. 1. Results from the parameter optimization sweep. The actual detection power for each parameter combination is shown on the y-axis and is defined as the number of data sets where both functional loci were identified as the best model. The parameter combinations for algorithm type, function set, and genotype-encoding method are abbreviated along the x-axis. Abbreviation definitions are given in Table 7. The solid dark gray line represents TBXO, and the dotted light gray line represents SPBXO. The four gene-gene interaction models are displayed by row (defined in Table 1), and the three effect sizes are displayed by column.
A considerable increase in detection power is also observed when complex function sets are used to search for non-additive interaction effects (Models 3 and 4). This effect is most noticeable in GENN where the parameter combinations that include allactivation nodes (NN4-6) have clear improvement over those with add-only activation nodes (NN1-3). The specific effect of genotype-encoding on detection power is less evident. It appears that there is an interaction between the encoding and the function set specific
ATHENA Optimization: The Effect of Initial Parameter Settings
55
to the underlying disease model. For example, when the interaction effect is additive (Models 1 and 2), the linear dummy-encoding method with simpler function sets (SR3 and NN3) are beneficial. However, when the interaction effect is non-additive (Models 3 and 4), full dummy-encoding with complex function sets (SR4 and NN4) confers a higher average detection power. We also observed a drastic decrease in detection power when no dummy-encoding and add-only activation nodes are used in GENN (NN-2). In addition, we ran Kruskal-Wallis rank tests to assess the statistical significance of the effects of each parameter on detection power within the different disease models averaged across heritabilities. The results shown in Table 5 suggest that the parameters we tested have a greater impact on detection power in GENN. Notably, genotype encoding was the only parameter value to have a marginally significant effect in both GENN and GESR. The only significant effect observed after correcting for the 24 tests using the Bonferroni method with an experiment-wide type I error rate set at 0.05 was function set in GENN under the disease model with a non-additive interaction effect and modest marginal effects (Model 3). Table 4. Abbreviation definitions for parameter combinations shown in Figure 1 Abbrev. SR-1 SR-2 SR-3 SR-4 SR-5 SR-6 NN-1 NN-2 NN-3 NN-4 NN-5 NN-6
Algorithm GESR GESR GESR GESR GESR GESR GENN GENN GENN GENN GENN GENN
Function Set Simple Simple Simple Complex Complex Complex Add-only nodes Add-only nodes Add-only nodes All nodes All nodes All nodes
Genotype-Encoding Dummy No Dummy Linear Dummy Dummy No Dummy Linear Dummy Dummy No Dummy Linear Dummy Dummy No Dummy Linear Dummy
Table 5. P-values for Kruskal-Wallis test assessing effect of specific parameter values on detection power within different disease models for GENN and GESR. Bold values are significant before correction for multiple tests. Bold value with asterisks indicates significance after correction for multiple tests. GENN Interaction Model 1 2 3 4
Crossover 0.6014 0.7996 0.5370 0.2608
GESR
Function Set 0.9191
Encoding
0.9494 0.0003* 0.0575
0.0079 0.8245 0.0245
0.0550
Crossover 0.2533 0.1493 0.6805 0.5796
Function Set 0.2600 0.2956 0.9369 0.3928
Encoding 0.0791 0.8026 0.6912 0.0482
56
E.R. Holzinger et al.
Importantly, GENN and GESR use two distinct modeling techniques. In order to evaluate the performances of the two methods, we ran statistical analyses comparing the prediction accuracies of the best models that correctly identified the functional loci averaged across the 100 data sets and the detection powers for all 144 parameter combinations. GESR and GENN both calculate prediction accuracy using the formula for balanced accuracy as shown in Table 3. The expected prediction accuracy under null data would be 50%. Figure 2 displays box plots of the average prediction accuracies and detection powers for GENN and GESR. Because neither the average prediction accuracy nor the detection power values were normally distributed, we used the Wilcoxon rank-sum test to determine if there was a significant difference between the two methods. After correcting for multiple tests using the Bonferroni method as before, we found a statistically significant difference for detection power (p = 0.001), but not for average prediction accuracy (p=0.158).
Fig. 2. Box plots comparing the distribution of average prediction accuracies for the correct best models and the detection powers across all parameter combinations for GENN and GESR
4 Discussion In this study, we demonstrate that, although GENN and GESR are model-free techniques, the ability to detect the true gene-gene interaction model is contingent upon initial parameter settings. The statistical analyses suggest that the detection power of GENN may be more sensitive to specific parameter settings than GESR for the tested scenarios. The performance of both methods appears to be dependent upon genotype encoding when the disease model contains a non-additive interaction effect with very small marginal effects. Interestingly, on average, GESR has significantly greater detection power than GENN. Nevertheless, as illustrated by Figure 1, under specific disease models and parameter setting, GENN appears optimal. One of the principle motivations behind the development of model-free analysis methods is the concept that in complex human diseases, the “true” genetic etiology is almost never known. Our results suggest that even for data-mining methods such as GENN and GESR, we still must take the underlying genetic model into account when setting the initial search parameters. One option, as demonstrated by Moore et al., is to perform parameter optimization in the actual data before analysis using parameters that have been shown to have the greatest impact on detection power [9]. One
ATHENA Optimization: The Effect of Initial Parameter Settings
57
limitation to this technique is the potential for over-fitting, especially in small data sets. This problem could be addressed via cross-validation or by incorporating parsimony in to the overall fitness evaluation. Another potential solution is to allow evolution itself to find the optimal parameter values. For example, the initial random population of solutions could be generated with equal proportions of all three genotype encodings. Ideally, as evolution progresses, the fittest models will be those with the optimal genotype encoding. One limitation of this method would be the need to generate larger population sizes to allow for the different parameter values to be represented. Because increasing the population size inflates the run time, this may in turn be a computational burden. The true value of these methods can only be accurately assessed according to their performance in biological data. As stated previously, three tasks that an effective method for elucidating the genetic architecture of complex disease must complete are variable selection, modeling, and biological interpretation. GESR and GENN are both limited in their capacity to perform variable selection due to the overwhelming number of variables that currently constitute a genetic association study. For this analysis, our in silico data only consisted of 25 SNPs, a tiny fraction of the one-million genotypes captured on the standard platforms. Also, the NN or SR models that are generated by either algorithm do not give rise to a simple biological interpretation of the underlying mechanism. These challenges highlight the necessity of incorporating GESR and GENN into ATHENA, an analysis framework from which one can perform the tasks of variable selection and model interpretation separately. Currently, ATHENA addresses these issues with Biofilter, which makes use of publicly available biological domain knowledge in order to filter out statistical noise in favor of signals that have true biological relevance [28]. Integrating effective filtering techniques with optimized analysis methods will allow us to efficiently search for complex genetic models and elucidate novel human disease etiology.
References 1. Hindorff, L.A., Sethupathy, P., Junkins, H.A., et al.: Potential etiologic and functional implications of genome-wide association loci for human diseases and traits. Proc. Natl. Acad. Sci. U.S.A 106(23), 9362–9367 (2009) 2. Nicolae, D.L., Gamazon, E., Zhang, W., et al.: Trait-associated SNPs are more likely to be eQTLs: annotation to enhance discovery from GWAS. PLoS Genet. 6(4), e1000888 (2010) 3. Manolio, T.A., Collins, F.S., Cox, N.J., et al.: Finding the missing heritability of complex diseases. Nature 461(7265), 747–753 (2009) 4. Maher, B.: Personal genomes: The case of the missing heritability. Nature 456(7218), 18–21 (2008) 5. Motsinger, A.A., Ritchie, M.D., Reif, D.M.: Novel methods for detecting epistasis in pharmacogenomics studies. Pharmacogenomics 8(9), 1229–1241 (2007) 6. Bateson, W.: Mendel’s Principles of Heredity. Cambridge University Press, Cambridge (1909) 7. Wright, S.: The roles of mutation, inbreeding, crossbreeding, and selection in evolution. In: Proc 6th Intl.Congress of Genetics, vol. 1, pp. 356–366 (1932) 8. Moore, J.H.: The ubiquitous nature of epistasis in determining susceptibility to common human diseases. Hum. Hered. 56(1-3), 73–82 (2003) 9. Moore, J.H., Barney, N., Tsai, C.T., et al.: Symbolic modeling of epistasis. Hum. Hered. 63(2), 120–133 (2007)
58
E.R. Holzinger et al.
10. Bishop, C.M.: Neural Networks for Pattern Recognition, pp. 1–482. Oxford University Press, London (1995) 11. Krogh, A.: What are artificial neural networks? Nat. Biotechnol. 26(2), 195–197 (2008) 12. Koza, J.R., Rice, J.P.: Genetic generation of both the weights and architecture for a neural network. IEEE Transactions II (1991) 13. Ritchie, M.D., White, B.C., Parker, J.S., et al.: Optimization of neural network architecture using genetic programming improves detection and modeling of gene-gene interactions in studies of human diseases. BMC Bioinformatics 4(1), 28 (2003) 14. Koza, J.: Genetic Programmming: On the Programming of Computers by Means of Natural Selection, pp. 1–819. MIT Press, Cambridge (1992) 15. Schmidt, M., Lipson, H.: Distilling free-form natural laws from experimental data. Science 324(5923), 81–85 (2009) 16. Moore, J.H., Parker, J.S., Olsen, N.J., et al.: Symbolic discriminant analysis of microarray data in autoimmune disease. Genet. Epidemiol. 23, 57–69 (2002) 17. Motsinger-Reif, A.A., Dudek, S.M., Hahn, L.W., et al.: Comparison of approaches for machine-learning optimization of neural networks for detecting gene-gene interactions in genetic epidemiology. Genetic Epidemiology 32(4), 325–340 (2008) 18. O’Neill, M., Ryan, C.: Grammatical Evolution. IEEE Transactions on Evolutionary Computation 5(4) (2001) 19. Holzinger, E.R., Buchanan, C.C., Dudek, S.M., et al.: Initialization Parameter Sweep in ATHENA: Optimizing Neural Networks for Detecting Gene-Gene Interactions in the Presence of Small Main Effects. In: Genet. Evol. Comput. Conf., pp. 203–210 (2010) 20. Reif, D.M., White, B.C., Olsen, N.J., et al.: Complex function sets improve symbolic discriminant analysis of microarray data. In: Cantú-Paz, E., Foster, J.A., Deb, K., Davis, L., Roy, R., O’Reilly, U.-M., Beyer, H.-G., Kendall, G., Wilson, S.W., Harman, M., Wegener, J., Dasgupta, D., Potter, M.A., Schultz, A., Dowsland, K.A., Jonoska, N., Miller, J., Standish, R.K. (eds.) GECCO 2003. LNCS, vol. 2724. Springer, Heidelberg (2003) 21. Dudek, S.M., Motsinger, A.A., Velez, D.R., et al.: Data simulation software for wholegenome association and other studies in human genetics. In: Pac. Symp. Biocomput., vol. 11, pp. 499–510 (2006) 22. Edwards, T.L., Bush, W.S., Turner, S.D., et al.: Generating Linkage Disequilibrium Patterns in Data Simulations Using genomeSIMLA, LNCS, vol. 4793. pp. 24–35 (2008) 23. Ott, J.: Neural networks and disease association studies. American Journal of Medical Genetics (Neuropsychiatric Genetics) 105(60), 61 (2001) 24. O’Neill, M., Ryan, C.: Grammatical Evolution: Evolutionary Automatic Programming in an Arbitrary Language, First. Kluwer Academic Publishers, Norwell (2003) 25. Turner, S.D., Dudek, S.M., Ritchie, M.D.: Grammatical Evolution of Neural Networks for Discovering Epistasis among Quantitative Trait Loci. In: Pizzuti, C., Ritchie, M.D., Giacobini, M. (eds.) EvoBIO 2010. LNCS, vol. 6023, pp. 86–97. Springer, Heidelberg (2010) 26. Motsinger, A.A., Hahn, L.W., Dudek, S.M., et al.: Alternative Cross-Over Strategies and Selection Techniques for Grammatical Evolution Optimized Neural Networks. In: Genet. Evol. Comput. Conf., pp. 947–948 (2006) 27. Turner, S.D., Dudek, S.M., Ritchie, M.D.: ATHENA: A knowledge-based hybrid backpropagation-grammatical evolution neural network algorithm for discovering epistasis among quantitative trait Loci. BioData. Min. 3(1), 5 (2010) 28. Bush, W.S., Dudek, S.M., Ritchie, M.D.: Biofilter: a knowledge-integration system for the multi-locus analysis of genome-wide association studies. In: Pac. Symp. Biocomput., pp. 368–379 (2009)
