Hybridizing Statistical Computation with SAS and MATLAB J. Andrew Howe
Istanbul, Turkey
[email protected]
Eylem Deniz Howe Mimar Sinan Fine Arts University Department Of Statistics, Faculty of Arts and Letters Istanbul, Turkey
[email protected] January 25, 2011 Abstract
There is a wide variety of computation resources available to statisticians, ranging from the simple Microsoft Excel, to the complex MATLAB. Each program has its own set of advantages and disadvantages, and sometimes the ability to use more than one application for modeling can be advantageous. Several researchers have developed methods for this. In this paper we propose a very general method for integrating MATLAB and SAS, which allows us to perform basically any type of statistical modeling so desired. We demonstrate this interface with a very interesting example. We have a large cross-sectional time series database for which we want to identify a parsimonious model to predict retail sales in the next time-step. We use a stochastic search algorithm in MATLAB and fit a series of mixed models in SAS. Keywords: Statistical computing, Platform independence.
1
Introduction
Over the past century, statisticians have evolved from using pen-and-paper calculations for analysis, to dependence upon increasingly complex computer programs. As the sophistication of these programs has increased, so has the types of modeling we can perform. For example, we can use computer software to perform the required numerical optimization for a non-Gaussian mixture model with many populations, Bayesian regression model with non-conjugate priors, or generalized linear models with various link functions. Of course, while this has definitely opened more frontiers, it’s not without its disadvantages. The universe of statistical modeling / analysis programs has also grown quite complex. According to [20], there are almost 100 software packages for statistical modeling. Additionally, the authors are aware of several others not listed. There may be many reasons for preferring one software package over others. For example, add-ins to Excel1 can be preferred in a business environment, where most modeling is performed in spreadsheets anyway. Several organizations standardize on R for the sole reason that it is freely available under the GNU General Public License. Many experienced programmers prefer the strong matrix-based programming environment provided by MATLAB2 - especially those that came from APL. Another Wikipedia page [19] compares several of these packages based on characteristics such as price and license. Ideally, however, software packages would instead be compared based mostly on two characteristics: • computational efficacy - “Can it do what I want?” • computational efficiency - “Does it make good use of resources?” 1 Copyright 2 Copyright
Microsoft The Mathworks
Logically, there should be complete sources comparing statistical software packages based on these important characteristics; one example would be the listing at [4]. This paper, however, is concerned with addressing this issue from a different perspective - we want toavoid choosing a single software package. Instead, we can consider the two items shown above in choosing how to combine software packages. Effectively, we want to hybridize the computational aspect of statistical modeling. In Section 2, we briefly review a subset of existing interfaces between several software packages that are available for “batch”or automated analysis. We then provide details of our interface between SAS3 and MATLAB, by way of a real-life example in Section 3. We finish up with concluding remarks in 4.
2
Computational Environments and Interfaces
2.1
Computational Environments
R is a popular open source statistical modeling package [13]; it was initially available in 1997 with the first published alpha version 0.16. R is freely available under the GNU General Public License, and
maintained by the R Core Development Team. However, anybody can develop and distribute routines for R. Its widespread usage and popularity probably stem mostly from the fact that it is free. R is also known to have flexible graphical capabilities. As a downside, it has been criticized for having a very opaque, overly complex, and verbose programming language. Additionally, many users of software are generally skeptical that open source software packages are as reliable and bug-free as commerciallydeveloped software. Microsoft’s Excel is the most widely used spreadsheet program, and is used extensively in business environments for modeling. While Excel has several formulas for statistical modeling, many statisticians consider it to be weak in this aspect. In fact, several versions of Excel have been severely criticized for imprecise calculations, misleading results, and poor algorithms [21, 10, 2]. MATLAB, short for “MATrix LABoratory”, was initially developed by Cleve Moler as an easy environ-
ment in which to perform linear algebra routines [such as those included in LINPACK and EISPACK 3]. Due to increasing popularity among students, engineers, and researchers, he and Jack Little formed The Mathworks to commercialize and distribute the software. As Chief Scientist, Dr. Moler has overseen the development of MATLAB as it has earned a following of a wide variety of scientists. Its biggest strength is how MATLAB treats all variables as matrices, thus allowing much code to be written without inefficient loops. While MATLAB is primarily used by engineers, the company has been paying more attention lately to how statisticians use its software; information regarding the statistics toolbox can be found here [17]. WinBUGS is an open source software package developed to perform Bayesian inference Using Gibbs
Sampling, developed initially by researchers at the Cambridge University MRC Biostatistics Unit [11, 9]. Freely available, the primary strength of WinBUGS (and other versions of BUGS) is that it is very specialized to Markov Chain Monte Carlo (MCMC) sampling. SAS is an expensive modeling environment that is used heavily in the business environment [18]. SAS
is very fast, and has a wide variety of statistical modeling techniques available. Its biggest strength is that it handles very large datasets well; there are also tools, such as Enterprise Miner, that provide nice graphical interface for statistical modeling and data mining. For many experienced computer programmers, the SAS scripting language is not a strong point. 3 Copyright
SAS Institute
Howe and Howe
Page 2 of 9
2.2
Interfaces
A free Excel add-in called RExcel was developed. RExcel makes all the functionality of R available in Excel, integrated with the familiar and powerful spreadsheet modeling capabilities of Excel. For more information about RExcel, see [1] and [5].
R - Excel
MATLAB - Excel Like RExcel, The Mathworks developed Spreadsheet Link EX to allow its analytical tools to be used in the spreadsheet environment directly [16]. This interface allows the researcher to either use Excel-type formulas or a GUI for analysis. R - WinBUGS With the popularity of R, an interface with WinBUGS is very useful. R2WinBUGS was originally written by Andrew Gelman [15]. With this package, the researcher can call a BUGS model, summarize inferences and convergence, and save the simulations in a format appropriate for R. MATBUGS is an interface which allows the researcher to combine the powerful matrix programming capabilities of MATLAB with the Bayesian and MCMC modeling functionality of WinBUGS [12]. Once the user correctly specifies the model parameters, the MATBUGS function creates and calls the WinBUGS script. Results, saved in a delimited text file, can be read easily by MATLAB. Lee, in his book [8], develops structural equation models under the Bayesian framework, as implemented in MATBUGS. MATLAB - WinBUGS
Several interfaces between SAS and R have been developed. The paper at [6] details methods and code to call R from SAS, then pass a plot from R back to SAS. In the Summer of 2009, The SAS Institute released version 3.2 of the SAS/IML Studio which included the ability to execute R code within SAS. SAS - R
SAS - WinBUGS [22] showed how to implement Bayesian analysis with WinBUGS using a set of standard SAS routines. This allows the researcher to use the strong data processing capabilities of SAS and the specialized Bayesian methods in WinBUGS. SAS - MATLAB We aren’t the first researchers to develop an interface between SAS and MATLAB. A white paper [14] posted on The Mathworks’ site details a method for interfacing the two applications through a MySQL database. This is clearly a step in the correct direction, but seems quite limited, in that it does not allow either application to call or run commands in the other.
3
Our
MATLAB
-
SAS
Interface
Unlike [14], our interface between SAS and MATLAB relies on passing data through simple (.csv or any format) text files. The real strength lies in the ability to dynamically write and run SAS scripts from MATLAB. We use the DOS command in MATLAB to call SAS, relying on SAS’s ability to execute a script passed as a command line parameter. Generalized, the algorithm for our approach is as follows: 1. In MATLAB: Perform whatever preliminary modeling steps are required 2. In MATLAB: Generate SAS script file and save to disk 3. In MATLAB: Call SAS executable, passing the script file as a command line parameter 4. In SAS: Execute script file, which should save some results to output file(s) 5. In MATLAB: Read output file(s) from SAS and continue as appropriate (it is easy to make MATLAB check if SAS ran successfully)
Howe and Howe
Page 3 of 9
The remainder of this narrative details how we applied this method to a complex data mining problem [7], with code excerpts for each of these steps. In this study, we had a large dataset of retail sales data. The data is composed of 27, 103 observations gathered over t = 1, . . . , 6 time periods (repeated observations) of 4, 517 salespoints (panels). The prediction variable of interest Y is the economic outcome in millions of Euros. The first variable is binary, indicating the presence or absence of a marketing campaign. The next 9 variables X2 , . . . , X10 (called pr1 through pr9) measure sales values of specific product categories in millions of Euros. The last five variables, X11 through X15 , were originally categorical, but we transformed then to continuous. They are time-invariant panel attributes. Finally, the first lag of the outcome variable was included as a potential predictor. Our goal is to identify a subset regression model which we can use to predict sales in the next time step. Preliminary analysis showed that some of the pr variables exhibit high colinearity, which could lead to inflated standard errors of the regression coefficients- see Table 1. Table 1: Correlation Matrix for pr Variables. pr1 pr2 pr3 pr4 pr5 pr6 pr7 pr8 pr9
pr1 1.000
pr2 0.597 1.000
pr3 0.753 0.659 1.000
pr4 0.653 0.750 0.688 1.000
pr5 0.573 0.476 0.449 0.393 1.000
pr6 0.595 0.526 0.505 0.426 0.637 1.000
pr7 0.702 0.501 0.553 0.549 0.726 0.602 1.000
pr8 0.670 0.681 0.617 0.753 0.593 0.508 0.711 1.000
pr9 0.697 0.315 0.581 0.320 0.540 0.552 0.628 0.414 1.000
Thus, we need to fit a subset regression model; for this problem, there are q = 16 variables 215 − 1 = 65, 535 possible subset models. Typical methods for selecting a subset regression model include: stepwise analysis, partial sums-of-squares and sequential F-tests, and consideration of reduced rank regression models. In practice, these methods may not work well in the presence of multicollinearity or model misspecification. Additionally, the hypothesis tests all require somewhat arbitrary selection of significance levels. We decided to use the genetic algorithm (GA), coded in MATLAB to search the subset model space. With five restarts, 50 generations, and a population size of 50, a maximum of 12, 500 panel regression models on over 25, 000 observations would to be run. Initially, we evaluated several MATLAB packages to perform the panel regression, but it was orders of magnitude slower than SAS (with proc TSCSREG). Hence, we needed a method to call and run SAS for each subset generated by the GA. To make the problem clear - each time we run TSCSREG, the variables in the dataset will be different. Our approach is to generate a new SAS script file for each subset evaluated. To do so, before the GA function starts iterating, we set up several global variables that hold the static SAS code, as shown in Listing 1. The variable pred names is a cell array that lists all the variable names, and StrPad is our MATLAB function that pads strings with any character desired, to a specific length. You can see that each of the string matrices created here are coding for a different proc in SAS.
Howe and Howe
Page 4 of 9
Listing 1: MATLAB code for generating static SAS code. 2
4
6
8
10
12
14
16
18
20
22
24
26
28
30
32
34
global pred names procimp p r o c s o r t da ta 2 . . . pr o cexp maxlen da ta 3 pr o cexp2 ; pred names = { ’ t r e a t ’ , ’ pr1 ’ , ’ pr2 ’ , ’ pr3 ’ , ’ pr4 ’ , ’ pr5 ’ , . . . ’ pr6 ’ , ’ pr7 ’ , ’ pr8 ’ , ’ pr9 ’ , ’ x1 ’ , ’ x2 ’ , ’ x3 ’ , ’ x4 ’ , ’ x5 ’ } ; % p u t t h e s e h e r e so o n l y have t o d e f i n e them a l l g l o b a l l y once maxlen = 1 5 0 ; procimp = [ StrPad ( ’PROC IMPORT OUT= work . c l a d a g i n ’ , maxlen , ’R ’ ) ; . . . StrPad ( [ ’ DATAFILE= ” ’ , mydir , . . . ’ \ c l a d a g v a r s n o t e m p o 7 . c s v ” DBMS=CSV REPLACE; ’ ] , maxlen , ’R ’ ) ; . . . StrPad ( ’ GETNAMES=YES ; ’ , maxlen , ’R ’ ) ; . . . StrPad ( ’ DATAROW=2; ’ , maxlen , ’R ’ ) ; . . . StrPad ( ’RUN; ’ , maxlen , ’R ’ ) ] ; p r o c s o r t = [ StrPad ( ’PROC SORT ’ , maxlen , ’R ’ ) ; . . . StrPad ( ’ DATA = work . c l a d a g i n ’ , maxlen , ’R ’ ) ; . . . StrPad ( ’ OUT = work . c l a d a g s o r t ; ’ , maxlen , ’R ’ ) ; . . . StrPad ( ’ BY i d tempo ; ’ , maxlen , ’R ’ ) ; . . . StrPad ( ’RUN; ’ , maxlen , ’R ’ ) ] ; da ta 2 = [ StrPad ( ’ da ta work . modelcov ; ’ , maxlen , ’R ’ ) ; . . . StrPad ( ’ SET work . c l a d a g o u t ; ’ , maxlen , ’R ’ ) ; . . . StrPad ( ’ WHERE TYPE =”COVB” ; ’ , maxlen , ’R ’ ) ; . . . StrPad ( ’ DROP TYPE NAME ; ’ , maxlen , ’R ’ ) ; . . . StrPad ( ’RUN; ’ , maxlen , ’R ’ ) ] ; da ta 3 = [ StrPad ( ’ da ta work . modelparms ; ’ , maxlen , ’R ’ ) ; . . . StrPad ( ’ SET work . c l a d a g o u t ; ’ , maxlen , ’R ’ ) ; . . . StrPad ( ’ WHERE TYPE =”PARMS” ; ’ , maxlen , ’R ’ ) ; . . . StrPad ( ’ DROP TYPE NAME ; ’ , maxlen , ’R ’ ) ; . . . StrPad ( ’RUN; ’ , maxlen , ’R ’ ) ] ; pr o cexp = [ StrPad ( ’PROC EXPORT DATA = work . modelcov ’ , maxlen , ’R ’ ) ; . . . StrPad ( [ ’ OUTFILE = ” ’ , mydir , ’ \ modelcov . c s v ” ’ ] , maxlen , ’R ’ ) ; . . . StrPad ( ’ DBMS = CSV REPLACE; ’ , maxlen , ’R ’ ) ; . . . StrPad ( ’RUN; ’ , maxlen , ’R ’ ) ] ; pr o cexp2 = [ StrPad ( ’PROC EXPORT DATA = work . modelparms ’ , maxlen , ’R ’ ) ; . . . StrPad ( [ ’ OUTFILE = ” ’ , mydir , ’ \ modelparms . c s v ” ’ ] , maxlen , ’R ’ ) ; . . . StrPad ( ’ DBMS = CSV REPLACE; ’ , maxlen , ’R ’ ) ; . . . StrPad ( ’RUN; ’ , maxlen , ’R ’ ) ] ; The reason why these variables are defined as global is that we are using a “template” file we created as a general genetic algorithm for any kind of subset modeling. The specific modeling is all coded in a separate function that fits the model, then computes and returns some score - the objective function for the GA. Thus, we want all of these variables available in the other function, but don’t want the overhead of passing them around so many times. The GA passes a binary word encoding the presence (1) or absence (0) of each predictor in the regression model. In the code excerpt Listing 2, we see how the binary string is used with the pred names global variable to create a string with the variables names that TSCSREG will use. Listing 2: MATLAB code converting binary word into variable names for SAS.
2
% c o n v e r t b i n a r y t o s t r i n g o f column h e a d i n g s global pred names ; p r e d s u s e = c e l l 2 m a t ( pred names ( b i n ==1)); % f o r c e i n p u t t o be l o g i c a l
Howe and Howe
Page 5 of 9
In Listing 3, the only dynamic code in the SAS script is created, using the preds use variable. We simply vertically catenate each of the string matrices into one big matrix, then use dlmwrite to save the script file to the disk for SAS to use. Listing 3: MATLAB code for generating subset-specific SAS code. 2
4
6
8
10
% c r e a t e t h e SAS f i l e global procimp p r o c s o r t da ta 2 pr o cexp maxlen da ta 3 pr o cexp2 ; p r o c t s c r = [ StrPad ( ’PROC TSCSREG DATA = work . c l a d a g s o r t . . . COVOUT OUTEST = c l a d a g o u t ; ’ , maxlen , ’R ’ ) ; . . . StrPad ( [ ’ MODEL outcome = ’ , p r e d s u s e , ’ / ’ ] , maxlen , ’R ’ ) ; . . . StrPad ( ’ RANTWO COVB; ’ , maxlen , ’R ’ ) ; . . . StrPad ( ’ ID i d tempo ; ’ , maxlen , ’R ’ ) ] ; dlmwrite( ’ c l a d a g a n a l S U B . s a s ’ , [ procimp ; p r o c s o r t ; p r o c t s c r ; . . . da ta 2 ; da ta 3 ; pr o cexp ; pr o cexp2 ] , ’ ’ ) ; Now (Listing 4) we finally call SAS using the DOS command, which allows MATLAB code to call any executable file. The mydir variable was defined globally, and it stores the current MATLAB directory, which is where the script file was stored. Recall in the procexp and procexp2 variables, SAS was instructed to save the model covariance matrix and estimated parameters as .csv files. Thus, after SAS is run, the objective function imports the files and computes the fitness score of the model. Listing 4: MATLAB code to execute the SAS script and extract the results.
2
4
6
8
% run t h e SAS f i l e SASpath = ’ ”C: \ SAS\ s a s . exe ” ’ ; global mydir ; [ s ,w] = dos ( [ SASpath , ’ ” ’ , mydir , ’ \ c l a d a g a n a l S U B . s a s ” ’ ] ) ; % read i n t h e parms f i l e & g e t MSE, and . . . n u s e = n∗(6− b i n ( 1 ) ) / 6 ; MSE = csvread ( ’ modelparms . c s v ’ , 1 , 3 ) ; l a c k o f f i t = n u s e ∗ log ( 2 ∗ pi ) + n u s e ∗ log (MSE(1 ))+ n u s e ;
10
12
% . . . read i n t h e r e s u l t i n g model c o v a r i a n c e mat rix , and . . . covmat = csvread ( ’ modelcov . c s v ’ , 1 , 9 ) ; It has been suggested that our method is inefficient, but this is not the case. On a normal desktop computer, one replication of the GA in MATLAB called SAS 3, 317 times, requiring just short of 80 minutes. Over 3, 000 different panel regression models were performed in less than an hour and a half. Finally, we have the SAS script that generated the best model, shown in Listing 5.
Howe and Howe
Page 6 of 9
Listing 5: Partial output from a single replication of the GA. 2
4
6
8
10
12
14
16
18
20
22
24
26
28
30
32
34
PROC IMPORT OUT= work . c l a d a g i n DATAFILE= ”J : \ p e r s \ c l a d a g \ q2 \ c l a d a g v a r s n o t e m p o 7 . c s v ” DBMS=CSV REPLACE; GETNAMES=YES ; DATAROW=2; RUN; PROC SORT DATA = work . c l a d a g i n OUT = work . c l a d a g s o r t ; BY i d tempo ; RUN; PROC TSCSREG DATA = work . c l a d a g s o r t COVOUT OUTEST = c l a d a g o u t ; MODEL outcome = t r e a t pr2 pr6 pr8 pr9 x1 x2 x3 x4 x5 / RANTWO COVB; ID i d tempo ; RUN; da ta work . modelcov ; SET work . c l a d a g o u t ; WHERE TYPE =”COVB” ; DROP TYPE NAME ; RUN; da ta work . modelparms ; SET work . c l a d a g o u t ; WHERE TYPE =”PARMS” ; DROP TYPE NAME ; RUN; PROC EXPORT DATA = work . modelcov OUTFILE = ” J : \ p e r s \ c l a d a g \ q2 \ modelcov . c s v ” DBMS = CSV REPLACE; RUN; PROC EXPORT DATA = work . modelparms OUTFILE = ” J : \ p e r s \ c l a d a g \ q2 \ modelparms . c s v ” DBMS = CSV REPLACE; RUN; This research was performed and submitted as part of the solution for the international 1st CLADAG Young Researcher Data Mining competition, sponsored by the SAS Institute and the Classification and Data Analysis Group of the Italian Statistical Society. The research methodology and results earned the “Best Operative Solution” [7].
4
Concluding Remarks
In this paper, we have detailed a general method for integrating statistical software packages SAS and MATLAB. The proposed method uses basic built-in capabilities of both programs, and does not require advanced programming. This allows us to use the strengths of both packages - specifically the speed of SAS and the superior matrix development environment of MATLAB. Because this method allows us to run arbitrary code in SAS automatically, it is more flexible than others that have been proposed. We used this interface to allow MATLAB to use proc TSCSREG to actually fit all the subset regression models generated by our genetic algorithm encoded in MATLAB. This was a rather complex example, showing the power and flexibility of our method. Along with this article, we have distributed the
Howe and Howe
Page 7 of 9
excerpted MATLAB functions (GASubsetSelect CLADAGq2.m and SAS ModelCovComp.m) used to generate the documented results. While this interface is a very useful addition to our computational abilities (we have used it in more than one research project), one weakness is that the SAS code needs to be created manually beforehand, or tailored specifically for each usage. Thus, we propose this work could be further improved by developing a full MATLAB toolbox with templates for all the SAS procedures. This would make implementation of the SAS - MATLAB interface even more general and useful.
References [1] Baier, T., Neuwirth, E., 2007. Excel :: Com :: R. Computational Statistics 22. [2] Cryer, J., 2001. Problems with microsoft excel for statistics, presented at the Joint Statistical Meetings, American Statistical Association. [3] Forsythe, G., Malcom, M., Moler, C., 1977. Computer Methods for Mathematical Programming. Prentice-Hall, Englewood Cliffs, New Jersey, USA. [4] Free Statistics, 2010. Freestatistics.info - free statistical software, data and resources. [Online; accessed 02-March-2010]. URL http://en.freestatistics.info/comp.php [5] Heiberger, R., Neuwirth, E., 2009. R Through Excel. Springer Verlag. [6] Holland Numerics, 2010. Sas to r to sas. [Online; accessed 06-February-2010]. URL http://www.hollandnumerics.co.uk/pdf/SAS2R2SAS_paper.pdf [7] Howe, J., Deniz, E., Turan, E., September 2009. Retail data mining under misspecification: 1st cladag young researcher data mining prize solution. Presented at the 2009 CLADAG meeting. [8] Lee, S., 2007. Structural equation modelling: A Bayesian approach. John Wiley & Sons, New York, NY. [9] Lunn, D., Thomas, A., Best, N., Spiegelhalter, D., 2000. Winbugs – a bayesian mframework: Concepts, sdtructure, and extensibility. Statistics and Computing 10, 325–337. [10] McCullough, B., Wilson, B., 2009. On the accuracy of statistical procedures in microsoft excel 2003. Computational Statistics and Data Analysis 49, 1244–1252. [11] MRC Biostatistics Unit, Cambridge, 2010. The bugs project. [Online; accessed 01-February-2010]. URL http://www.mrc-bsu.cam.ac.uk/bugs/ [12] Murphy, K., Mahdaviani, M., 2005. Matbugs software. URL http://www.cs.ubc.ca/murphyk/Software/MATBUGS/matbugs.html [13] R Core Development Team, 2010. The comprehensive r archive network. [Online; accessed 01February-2010]. URL http://cran.r-project.org/ [14] Shvorob, D., 2006. The twain shall meet: Facilitating data exchange between sas and matlab. White paper, Vanderbilt University. [15] Sturtz, S., Ligges, U., Gelman, A., 2005. R2winbugs: A package for running winbugs from r. Journal of Statistical Software 12 (3), 1–16.
Howe and Howe
Page 8 of 9
[16] The Mathworks, 2010. Spreadsheet link ex 3.1 for microsoft excel. [Online; accessed 01-February2010]. URL http://www.mathworks.com/products/excellink/ [17] The Mathworks, 2010. Statistics toolbox 7.2. [Online; accessed 06-February-2010]. URL http://www.mathworks.com/products/statistics/ [18] The SAS Institute, 2010. Sas 9.2 software. [Online; accessed 01-February-2010]. URL http://www.sas.com/software/sas9/ [19] Wikipedia, 2010. Comparison of statistical packages — wikipedia, the free encyclopedia. [Online; accessed 30-January-2010]. URL http://en.wikipedia.org/w/index.php?title=Comparison_of_statistical_ packages&oldid=339869296 [20] Wikipedia, 2010. List of statistical packages — wikipedia, the free encyclopedia. [Online; accessed 30-January-2010]. URL http://en.wikipedia.org/w/index.php?title=List_of_statistical_ packages&oldid=340604209 [21] Yalta, A., 2008. The accuracy of statistical distributions in microsoft excel 2007. Computational Statistics and Data Analysis 52, 4579–4586. [22] Zhang, Z., McArdle, J., Wang, L., Hamagami, F., 2008. A sas interface for bayesian analysis with winbugs. Structural Equation Modeling 15, 705–728.
Howe and Howe
Page 9 of 9
