An Alternative to SAS-Intrnet Using Base SAS, ASP.NET and SQL

An Alternative to SAS®/Intrnet® Using Base SAS, ASP.NET and SQL Server Yuying Zhang, Leslie Hill, Nick Kinsey, Susan Eversole, Bill Savage, David Uglow, George Grubbs, Annette Green, Jeniffer Iriondo-Perez RTI International, Research Triangle Park, NC

ABSTRACT Dynamic, web-based SAS reports are usually generated using SAS/IntrNet or other technologies such as the SAS 9 Business Intelligence/Enterprise Business Intelligence (BI/EBI) Architecture. This paper describes a method to create SAS reports dynamically on the web, using SAS, ASP.NET and Microsoft SQL Server technologies. By leveraging what you already have with your base SAS license and other technical tools, you can achieve web-based SAS reports if SAS/IntrNet is not an available option for your report delivery. RTI devised this alternative way of creating dynamic SAS reports using ASP.NET, Microsoft SQL Server and an existing base SAS server. We used SQL Server to store report requests and developed a Windows Service that monitors user requests continuously. Users are able to submit report requests via a dynamic web user interface developed using ASP.NET. The resulting SAS reports can be either emailed or posted on the web site on, essentially, a real-time basis. What is required in this solution is base SAS knowledge and software expertise with no additional investment in SAS technologies.

INTRODUCTION We piloted our approach for a project that had a need for web-based access to certain SAS reports using data that had been archived. A very limited number of users would infrequently access these reports, so purchase of the SAS/IntrNet component to support this feature was cost-prohibitive. A workaround to implement these reports was needed. The fact that these reports can be produced with the base SAS package, and the web user interface can be created through ASP.NET pages led us to the workaround. We created a solution that utilized our expertise in software technologies and our experience with base SAS to provide web-based, user-driven, dynamic SAS reports.

SYSTEM ARCHITECTURE The application that supports the project is a complex web-based application developed using ASP.NET with a back-end SQL Server 2005 database. Users must be authenticated to access the website and only certain privileged users can access the archived data reports. After the user selects the desired report from the menu, the report options are presented. The user selects the desired options and submits the request. Depending on user selections, some reports can take a long time to run and could cause the user to experience session timeout errors. This led us to look for a solution that did not require the web page to wait for the report to run. In our implementation, the report requests are validated in real time and then saved to the back-end database. Users are notified immediately about the status of their request, and when SAS finishes its job, the requested reports are automatically emailed to the users. Users do not experience session time-out problems, and the reports are available in their email inbox, which they can access anytime. An alternative approach would have been to update the website to provide links to the reports when they became available. We preferred the email approach to push the report to the user as quickly as possible, rather than requiring the user to pull the information later.

SYSTEM PROCESS The following steps, depicted in Figure 1, are used to produce and deliver the reports: 1.

The user requests the report and selects the desired parameters on the website.

2.

The website logs the request details in the database.

3.

The report invoker, a Windows Service, monitors the database for new requests.

4.

The report invoker invokes SAS on the command line.

5.

The SAS report server runs the report, prepares the report file and saves it to the file system.

1

6.

The report invoker retrieves the report file.

7.

The report invoker emails the report to the user.

8.

The report invoker logs all actions in the database.

9.

The report invoker could update the website with the report file, but this is not implemented in our project.

Figure 1. Process diagram WINDOWS SERVICE Authorized users can log into the website at any time of the day and submit a report request. To monitor user requests continuously, we installed Report Invoker as a Windows Service in the production environment. In our testing environment, Report Invoker is executed on demand as a windows console application. As a Windows Service, Report Invoker runs in the background. It is configured to start automatically when the operating system is rebooted, so no user intervention is required to keep the service running. There were two technical challenges we had to resolve. The first challenge was to find out when a user makes a report request so we can invoke the appropriate SAS program in a timely manner; the second challenge was to find out when SAS finishes its job so that the system can email the final report to the user or update the website as soon as the report is available. To solve these two technical problems, we initially decided to use a database watcher to alert us to new report requests and a file watcher to alert us to completed SAS report runs. Since both watchers would be running as part of a Windows Service, we would never miss a user request. But in the end, for our project, the traditional database polling approach and the sequentially processing of requests proved to be better than the planned database notification and concurrent processing of files. For the benefit of our potential readers, we describe both approaches in the following two sections.

2

DATABASE WATCHER A database watcher, using SQL Server Notifications Service, raises an event whenever new requests for reports are inserted in the database. This event handler is used to invoke SAS to run the report. SQL Server Query Notifications allow applications to receive a notice when the results of a query have changed. This improves performance by not having to query the database periodically for changes. SQL Server Query Notifications is a powerful new feature in SQL Server 2005. The “sweet spot” for this feature is for read-mostly tables. For this application, since there is a very limited number of users that will request archive reports, the underlying table that stores the requests is updated very infrequently. Therefore, Query Notifications seemed to be a good fit for this solution. We learned that a design policy of the SQL Server Query Notifications is that it is better to notify the client too often rather than to have the client miss a notification. During testing, the solution worked as expected, but when we deployed it to production, we saw far too many events fired. We also learned that Microsoft will discontinue SQL Server Notification in SQL Server 2008. We didn’t think it was worthwhile to pursue development with technologies that would not be supported in the near future. In the end, we used the proven database polling approach. The database watcher polls SQL server database on a periodic basis to see if there are any new requests, and if there are, it immediately invokes SAS to run the requested SAS report.

FILE WATCHER The time required to generate reports in SAS varies by report. The archive reports range anywhere from just under a minute to more than 20 minutes. In order to monitor SAS jobs, we originally created a file system watcher using the .NET Framework. The watcher raised an event when a new report file was created in the report folder, and the event handler emailed the report to the requested user, using an SMTP server. Our initial reason for using the file watcher was so we could run the reports simultaneously, rather than sequentially, when multiple requests were queued. In testing, we discovered that this approach did not result in significantly faster processing times. Processing the reports sequentially enabled us to send the email without a file watcher notification. Once we determined that we would not use the database watcher, we also discontinued using the file watcher. The Windows service now waits for the SAS process to finish and then locates the report and emails it to the user

SYSTEM FLOW When the Report Invoker Windows Service starts, the following methods, depicted in Figure 2, are used to watch for and handle report requests: 1.

HandleRequests calls IsNewRequest periodically to determine if there is a new report request.

2.

IsNewRequest polls the database to determine if there is a new report request. a.

If so, control is passed to RunReport.

b.

If not, control is passed back to HandleRequests.

3.

RunReport calls SAS on the command line to run the report request.

4.

LogReport updates the report request in the database to record that the report was created and the program has finished processing/running.

5.

EmailReport sends the report to the requestor.

6.

LogEmail updates the report request in the database to record that the report was emailed.

7.

Control is returned to HandleRequests.

3

Figure 2. High-level flow chart. ENVIRONMENTS In order to deliver high quality archive reports to our clients, two independent environments, testing and production, were established. These two environments have identical configuration, including folder structure, SAS datasets, SAS programs and other files. All bug fixes and testing are performed in the testing environment, and production is used exclusively to serve our clients with verified functionality. The advantages of these independent environments are that they: 1.

Support the availability of production reports

2.

Facilitate the development of bug fixes and report enhancements

3.

Provide a clearly named and centralized folder structure

4.

Decrease the changes required for deployment and thus reduce the possibility of error

The content and purpose of each folder within the environments is described in Table 1.

Table 1. Folder structure. Folder

Purpose

Development

Environment for development and testing of reports. Contains the subfolder tree below.

Production

Environment for production reports. Contains the subfolder tree below.

Subfolders Programs

Contains the SAS report programs, which are structured so they function from the reports folder in either environment without modification.

Requests

Contains the sysparams used by SAS to generate the reports. These files are created for each request and named using the request ID.

Reports

Contains the report output files created by SAS. These files are created for each request and named using the request ID.

Logs

Contains the .log and .lst files created by SAS. These files are created for each request and named using the request ID.

Console

Contains a copy of the console application that runs the reports. The console application is used to run reports in development. Production reports are run by a Windows Service.

Datasets

Contains a dedicated set of SAS datasets. If datasets are shared between environments reports may fail due to file locking. This is not a problem within the environments because the reports are run sequentially.

Formats

SAS format files

4

SAS SOLUTION A critical component of the reporting process is the passing of user-specified inputs, or report parameters, to the various SAS programs used to generate real-time, dynamic reports comparable to those produced via SAS/IntrNet. To specify these inputs, users access the reporting web interface, select one of the four types of available reports, and select from available inputs (e.g., race, ethnicity, age, gender, outcome) to create a customized version of the report. Each user request is assigned a unique request ID that is passed via the command line to the selected report program. This is accomplished using the SAS SYSPARM option. The request ID is also used in both the name of the parameter file passed to the reports program and in the output file returned to the user. The following command and switches are used to invoke SAS: “C:\Program Files\SAS\SAS 9.1\sas.exe” -SYSIN “SasProgramName” -SYSPARM “EnvironmentFilePath, RequestID” -LOG “SasProgramName_RequestID.log” -PRINT “SasProgramName_RequestID.lst” -ICON -NOSPLASH In order to run the SAS code in each environment without making code changes, the environment file path to the upper-level code is sent in as a command-line variable. Within the SAS code, this path is appended to relative file paths. The request ID, also passed as a command-line variable, is used to locate the report parameter selections file and to name logs, print files, and report files. Note that it is necessary to specify unique log and print file names to avoid file locking. To determine the location of the SAS datasets and request ID associated with the current request, the following SAS code is used to extract the file path and request ID. data _null_; LENGTH PATH $150 REQ $20; dlm = “,”; parmstr = strip(SYMGET(‘SYSPARM’)); PATH = STRIP(SCAN(PARMSTR,1,DLM)); REQ = STRIP(SCAN(PARMSTR,-1,DLM)); CALL SYMPUT(‘FILEPATH’,STRIP(PATH)); CALL SYMPUT(‘REQID’,STRIP(REQ)); run; Because the SAS SYSPARM option limited the length of text strings passed to the SAS program, the report parameters specified by the user are stored in an external text file. Parameters in the file are stored using a comma-delimited list of name-value pairs. This maximizes the number of inputs a user could specify for a given report. For example, for Request ID 192, the file SYSPARMS_192.txt has all the SAS parameters that the user specified to run the one report. Here is the content of the SYSPARMS_192.txt file: Inact=1,selABC=1,FFY=0,FFYQTR=0,baseLine=1,AgeGroup=,Gender=,HispanicLatino=,Ethni city=,Race=,Employment=,matchType=1,xVar=AgeGroup,FreqVars=Gender HispanicLatino AgeGroup To generate the report, the text file is read by the selected report program, and the SAS SCAN function is used to parse each parameter and assign it as a macro variable. This step allows the same code to be used repeatedly to generate reports with differing inputs. The SAS code then uses the macro variables to produce the requested report. FILENAME INPUT “&FILEPATH.20-Requests\SYSPARMS_&REQID.txt”; data _null_; INFILE INPUT LRECL=500 TRUNCOVER PAD; LENGTH P_VAR P_VAL $300; INPUT @001 PARMSTR

$CHAR500.;

5

dlm = “,”; PARM_NUM = 1; P_VAR = strip(scan(parmstr,1,dlm)); DO WHILE(P_VAR ^= ‘ ‘); P_VAL = STRIP(SCAN(P_VAR,-1,’=‘)); P_VAR = STRIP(SCAN(P_VAR,1,’=‘)); CALL SYMPUT(P_VAR,STRIP(P_VAL)); PARM_NUM+1; P_VAR = STRIP(SCAN(parmstr,PARM_NUM,dlm)); END; run; The output report file is saved using the same request ID. The report is saved as a Word (.doc) file and either emailed to the user or posted to the website for real-time access. In the above example, the SAS-generated Report-Name_192.doc will be sent as an attachment to the user who made the request 192.

CONCLUSION This low cost technical solution meets our needs and has been in production for a few months. We have found it to be reliable and efficient. It is probably not as powerful as using the SAS/Intrnet, but given our special situation; it is a good alternative.

ACKNOWLEDGMENTS The authors acknowledge the editing assistance of Susanna Cantor, RTI International.

RECOMMENDED READING 1.

Stanislaw Furdal, Quick Windows Batches to Control SAS Programs Running Under Windows and UNIX

2.

SQL Server Notification http://www.microsoft.com/Sqlserver/2005/en/us/notification-services.aspx

CONTACT INFORMATION Your comments and questions are valued and encouraged. Contact the author at: Yuying Zhang Research Triangle Institute, International 3040 Cornwallis Road, PO Box 12194 Research Triangle Park, NC, 27709-2194 Work phone: (919) 316– 3844 Fax: (919)541-6178 Email: [email protected] Web: http://www.rti.org

Leslie Hill Research Triangle Institute, International 3040 Cornwallis Road, PO Box 12194 Research Triangle Park, NC, 27709-2194 Work phone: (919) 541 – 8749 Fax: (919)541-6178 Email: [email protected] Web: http://www.rti.org

SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration. Other brand and product names are trademarks of their respective companies.

6