the above mentioned concepts to a nonstationary data set as a means to motivate ... correlation analysis, and linear filtering, using MATLAB. They are asked to ...
Session 13b4 Making Probabilistic Methods Real, Relevant, and Interesting Using MATLAB Jose Ramos and Charles Yokomoto Department of Electrical Engineering Indiana University Purdue University at Indianapolis 723 West Michigan Street, SL 160 Indianapolis, IN 46202-5132
Abstract - Teaching probabilistic methods (PM) to undergraduate electrical engineering students is often difficult because of the increased conceptual level of the material compared to their previous learning experiences. However, with the help of software tools like MATLAB, an instructor can unravel the mysteries of probability theory and its marvelous array of applications. A typical PM course normally covers the following topics: set theory and Venn diagrams, probability axioms and probability distributions, basic statistics, hypothesis testing, regression analysis, random processes, correlation analysis, power spectrum, and optimum linear systems. If we were to teach these concepts in this particular order with traditional pencil and paper learning, we could end up with students who have merely learned to put a puzzle together by numbers instead of from the actual picture. In this paper we will describe how MATLAB exercises and projects can be used to make a course in probability interesting, real, and relevant to the electrical engineering student. We will describe a teaching strategy that calls for students to apply the above mentioned concepts to a nonstationary data set as a means to motivate them, to promote active learning and to help them integrate the concepts. Finally, we will also describe a sample term project in which students are given a nonstationary data set that represents the output of some unknown system. Using this data set, students are asked to convert the data set to white noise through the application of a sequence of transformations using concepts from regression analysis, Fourier and spectral analysis, correlation analysis, and linear filtering, using MATLAB. They are asked to validate the model, which requires hypothesis testing, confidence intervals, cumulative periodogram test, probability plots, etc. Assessment of student satisfaction has shown that this strategy of using MATLAB helped students grasp the concepts and learned to integrate them.
I. Introduction
previous learning experiences. A traditional pencil and paper approach can lead to serious learning conflicts if the instructor does not properly blend the theory with practical applications. Our experience is that students tend to reject deep abstract concepts when they cannot associate them with anything practical from their field of study. Another difficulty the instructor may encounter is the fact that although the typical gambling examples are good for introducing the concepts, many students find themselves clueless when are asked to extrapolate these concepts to their areas of interest. Thus, the instructor should always keep in mind who the audience is and what are the applications relevant to such audience. One then has to find alternative teaching strategies without compromising both the theoretical and applied aspects of the course. From our experience we have learned that the use of MATLAB has a positive influence on the students understanding of the concepts and can lead to a very interesting learning experience. In this paper we will describe a teaching strategy that calls for students to apply a graphical user interface (GUI) to a real-life data set as a means to motivate them, to promote active learning, and to help them integrate the concepts into a final project. Finally, we will also describe a sample term project in which students are given a nonstationary data set that represents the output of a system taken from a collection of applications they are allowed to choose from. Using this data set, students are asked to build a mathematical model that represents the data. The students are then asked to validate the model, and finally to use the model for its intended purpose, i.e., forecasting or simulation. Assessment of student satisfaction has shown that this strategy of using MATLAB helped students grasp the concepts and integrate them into their curriculum.
2. Designing a MATLAB-Based PM Course
In order to introduce MATLAB into a PM course one has to make specific changes to the course content compared with a Teaching an undergraduate course such as probabilistic traditional course. Starting from set theory, one should methods (PM) in electrical engineering can often be a stress the notion of vectors and matrices, since the basic challenging task for the instructor because of the increased computational unit in MATLAB is a matrix, thus a scalar level of abstraction required, compared to the student’s would be treated as a (1 x 1) matrix. The first part of the 0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13b4-2
Session 13b4 course should be a basic introduction to sets, functions, and graphs as they apply to basic signal analysis. The ultimate goal is to be able to relate signals as functions on sets (i.e., audio signals, radio signals, voltages, synchronous circuits, images, etc.), functions as tables, and finally, tables as vectors and matrices. One could talk about sets as they relate to the elements of a vector or a matrix. This approach leads to the development of a data set and how it relates to MATLAB and the probabilistic and signal analysis GUI. Venn diagrams can be discussed in the context of signals, vectors, matrices, and data sets. Following set theory and Venn diagrams, the instructor may choose to introduce probability axioms. This topic relates very well to constraints, inequalities, and proportions. Probability distributions and moments can be introduced from a rather practical point of view. From our experience, students seem to favor the definition of proportions as a means of defining probability. This ties very well with histograms and can be easily computed from a data set. Moments can be introduced with examples from mechanics (i.e., center of gravity, moments about a point, moment of inertia, and parallel axis theorem). These analogies bring physical meaning to the understanding of moments. One application where moments are used is to test for stationarity of a data set (lack of trends in the data). At this point the instructor should consider exercises geared towards testing the students’ data for what is called “second-order stationarity”, using tests based on first and second-order moments. Later on the instructor may want to verify stationarity using the concept of autocorrelations. The next major topic is statistics. Here the instructor may consider the use of MATLAB to compute basic statistics such as mean, median, mode, range, standard deviation, skewness, kurtosis, correlation coefficient, and explain how these relate to the distribution properties of the data. The central limit theorem can also be explained from a data analysis point of view. Hypothesis testing should then be introduced keeping in mind that this will be a useful tool for model validation later on. Finally, estimation and regression analysis can be combined into a single topic since they are somewhat related. However, we suggest the use of nonlinear models that are intrinsically linear since many engineering applications deal with nonlinear functions that can be converted to linear ones. Montgomery and Peck [1] and Draper and Smith [2] are excellent sources for these types of problems. This will be a useful tool for removing trends from the data. The final major topic is on random processes. This is definitely the most difficult part of the course and the instructor should always bear that in mind. Fortunately, with a given data set and the use of a special-purpose GUI, students will enjoy analyzing their data and visualizing the results. We suggest having live classroom demos - they can be fun and very informative. Here the instructor can explain autocorrelation with scatter diagrams between a data set and the same data set shifted by a certain lag (i.e., z0 vs. z1, z0 vs.
z2, z0 vs. z3, etc). Fitting a straight line and calculating the correlation coefficient will give the student a good feel for what autocorrelation is. Having eight or nine such scatter diagrams leads to a nice explanation of why autocorrelations tend to decay (scatter diagram becomes fatter) as the number of lags increase. Once autocorrelation is explained the instructor can then show how a random process can be decomposed as a sum of polynomial trends, periodic trends, and a stationary random component (autocorrelated process). The fact that there are trends explains why the data may be nonstationary. Likewise, nonstationarity can be observed from the autocorrelation function if it decays very slowly. The relationship between power spectrum as the Fourier transform of the autocorrelation function may be a logical way of studying the periodic behavior of the data. Introducing the cumulative periodogram as an analysis of variance Table is a direct consequence of Parseval’s theorem. From this the students can observe the frequencies that explain the most power in the data (variance). Finally, the next step would be to introduce the concept of linear systems such as the family of autoregressive moving average filters ARMA (n, m) (see Box and Jenkins [3]). Given the level of difficulty of this material, we suggest combining it as part of a class project. A more detailed description of topics and project material for a MATLAB-based PM course is shown in Table 1.
3. Integrating Probability, Statistics, Random Processes into a Project
and
In order to introduce practical applications into a PM course, one has to blend theory with practice. Our approach is to give the students the opportunity to choose a data set from a number of sources from the Internet. Three specific sites are: http://www.esat.kuleuven.ac.be/sista/daisy, http://wwwpersonal.buseco.monash.edu.au/~hyndman/TSDL http://bos.business.uab.edu/browse
0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13b4-3
Session 13b4
Major Topics • • • •
•
Set theory Venn diagrams Probability axioms Discrete probability distributions Continuous probability distributions Moments
• • • •
Basic statistics Hypothesis testing Estimation Regression analysis
• • •
Correlation analysis Spectral analysis Linear systems
•
Table 1.
Used in Project (P) or Homework (H) (H) (H) (H) (H)
(H), (P)
(H), (P) (P) (P) (P) (P) (P) (P) (P)
Application
GUI Functions
• • • • • • •
Data sets Vectors Matrices and tables Histograms Probability paper Probability plotting Testing stationarity
•
Exploratory analysis
• • • •
Data analysis MLE, MOM, LS Confidence Intervals Curve fitting
• • • • •
Exploratory analysis Detrending Model building Model validation Spectral analysis
• • • • • •
Scatter diagrams Autocorrelations Fourier analysis Box & Jenkins Whiteness tests Stationarity tests
•
Nonstationary analysis Spectral analysis Model building Model validation Exploratory analysis
• • • •
Outline of suggested topics, homework, and project material for a MATLAB-based PM course.
These sites have a wide variety of data available, ranging from electrical and mechanical systems to environmental and business time series. We prefer to use output data only (no inputs), otherwise the analysis would require more specialized system identification concepts. Also we suggest data that has periodic components and polynomial behavior so that the students can assess these as part of the analysis. The idea is to characterize the data as the output of some model driven by a white noise input. In the absence of inputs, the best one can do is to assume the inputs are white noise. The problem then is to find out what type of white noise generated such outputs. The solution is obtained by converting the output process into a white noise process via a series of transformations. This may sound a bit too random process oriented, however, elements from probability and statistics play an important role in the
conversion process as will be seen shortly. The exact picture is presented in Figure 1. Here a cause-effect relationship between inputs and outputs from some unknown system can be approximated by passing a white noise input through a mathematical model to produce the noisy output data. A more realistic situation is shown in Figure 2, where the output is known but both the system and white noise inputs are unknown. The system then becomes a series of transformations put together into the form of a model. The tools used for building such mathematical models are a combination from statistics, time series analysis, and linear systems theory. It is quite unlikely to have a PM course that combines all these tools together. However, with the use of MATLAB and a special-purpose GUI, our approach is feasible. We should point out that the use of a specialpurpose GUI is the most important element for the development of such PM course.
0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13b4-4
Session 13b4
Figure 1. Cause and effect relationship between I/O and its mathematical approximation.
Figure 2. Block diagram of the process of converting from white noise to outputs.
In the process of converting white noise into output, one has to work backwards, that is from the output data to white noise. The first thing to do is to plot the original output data and observe any peculiar trends such as polynomial growth, decay, or periodic patterns, which are an indication of nonstationarity in the data. A histogram of the data is essential, along with a probability fit superimposed. Departure from a Gaussian distribution is observed from a histogram plot and with the help of basic statistics such as mean, mode, median, variance, skewness, kurtosis, and range. If the data departs from a Gaussian distribution, one should consider a Box-Cox transformation as presented in Box and Jenkins [3]. This type of transformation is shown in Figure 3, where the original data Y is converted to Gaussian data and then labeled X data. Once the data is nearly Gaussian (the above transformation can also be used to suppress an increase in variance and to properly scale the data), one should plot X and observe any polynomial behavior in the data. This can be corrected by fitting a polynomial function to the data (thus regression analysis) as shown in Figure 4. Here the X data is detrended and converted to Z data. At this point the Z data should have a constant mean and it should start looking more Gaussian. A good practice is to compute as many statistics as possible and to study them as part of the process. The entire exercise
has good educational value since the students will be able to visualize dramatic changes after each transformation and at the same time will observe the data becoming more and more towards a white noise process. The next step is to plot the Z data and observe if there are any periodic trends. If so, then a Fourier fit is required in order to remove all periodic components as shown in Figure 5. After the Fourier analysis is done, the new data set is labeled W. It is interesting to observe this data set, as it will show no signs of nonstationarity (polynomial and periodic trends). However, it will be an autocorrelated process with constant mean and standard deviation, also called secondorder stationary. At this point one has to observe the autocorrelation and partial autocorrelation in order to determine the orders m and n of the ARMA (m, n) model. Then to convert from W data to white noise, one has to try different model orders and test the residuals for whiteness (hypothesis testing). Finally, one should verify that the residuals from the ARMA model, labeled E data, forms a white noise process. The general procedure is shown in Figure 6. There are numerous tests for these and are all integrated into the probabilistic and signal analysis GUI (see Figure 7)..
0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13b4-5
Session 13b4
Figure 3. Transformation to Gaussian via a Box-Cox transformation.
Figure 4. Transformation to a constant mean process by removing polynomial trends.
Figure 5. Transformation to a non-periodic process by removing all periodic components.
Figure 6. Converting a stationary process into white noise via an ARMA (m, n) filter.
0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13b4-6
Session 13b4
Figure 7. Main features of the probabilistic and signal analysis GUI.
4. Conclusions Teaching probabilistic methods (PM) to undergraduate electrical engineering students can indeed be a fun experience if properly done with sufficient examples from real world applications to back up the theory. With the help of MATLAB, we have presented an application oriented PM course syllabus, which exploits the mysteries of probability theory and its marvelous array of applications. We have presented a teaching strategy that calls for students to apply concepts learned in class to a non-stationary data set as a means to motivate them, to promote active learning and to help them integrate the concepts into a class project. Finally, a sample term project was discussed, which converts the data set to white noise through the application of a
sequence of transformations using concepts from regression analysis, Fourier and spectral analysis, correlation analysis, and linear filtering, all using the MATLAB GUI.
5. References 1.
2. 3.
D.C. Montgomery and E.A. Peck, Introduction to Linear Regression Analysis, second edition, Wiley, New York, 1992. N.R. Draper and H. Smith, Applied Regression Analysis, third edition, Wiley, New York, 1998. G.P. Box and G.M. Jenkins, Time Series Analysis: Forecasting and Control, Holden Day, San Francisco, 1976.
0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13b4-7
