THE FOURTH ANNUAL 2008 MLSP COMPETITION Kenneth E. Hild III, Vince D. Calhoun 2,3
IDept. of Biomedical Engineering, Oregon Health and Science University, Portland, OR, USA 2Dept. of Electrical and Computer Engineering, University of New Mexico, Albuquerque, NM, USA 3The Mind Research Network, Albuquerque, NM, USA
[email protected],
[email protected] ABSTRACT For the Fourth Annual 2008 Machine Learning for Signal Processing competition entrants were asked to develop a machine learning algorithm that maximizes the rate of return by trading (buying, selling, shorting, or covering) stocks over a six-month time period. Each entrant began with a (fictional) SI00,000 USD. Both the training and the test set include the daily price and volume for a total of 2929 stocks that are traded in American stock markets and a total of 41 monthly indices. Stock valuations are based on real (historical) stock prices. This year there were 5 algorithms submitted. The highest annual rate of return of an astonishing 150% was obtained by Peng and Ji of the Rensselaer Polytechnic Institute/Shanghai Maritime University team.
1. INTRODUCTION The Machine Learning for Signal Processing (MLSP) Technical Committee has hosted a machine learning competition, held in conjunction with the MLSP conference, for each of the last four years. In previous years the competition consisted of blindly separating synthetic instantaneous mixtures and real (convolutive) mixtures of speech, denoising magnetoencephalographic (MEG) data, processing functional magnetic resonance image (fMRI) data obtained from an oddball task, and rejecting artifacts in eventrelated electroencephalographic (EEG) data. This year's competition consists of maximizing the annual rate of return by trading stocks over a six-month period. In 2006 the New York Times reported that artificial intelligence is already being used in the financial sector and that its role is increasing [1]. In this article the author reports that quantitative funds (or quant funds for short), which are mutual funds that buy and sell stocks based on proprietary computer models, have "outperformed their benchmarks in the last three years" and that "investors have noticed." This year's competition centers on developing new algorithms for stock trading. Aas and Dimakos have written an introductory paper on this topic [2]. Several published approaches include a direct reinforcement method [3], neural
network methods [4], [5], and a method based on independent components analysis [6]. Dempster et al. have also written a nice paper on computational techniques for trading [7]. In this paper we explain the rules and details of this year's competition, we briefly describe the 5 submitted algorithms, and we present the results of all 5 algorithms.
2. TRAINING DATA The data, at the time this article went to press, may be downloaded from the following URL, http://www.bme.ogi.edu/ hildk/mlspO8TrainingData.mat. The training data, which is in Matlab® format, consist of daily stock prices and monthly indices dating from Sept. 29, 2003 until Sept. 28, 2007. Daily stock data are given for a total of 2929 stocks. The value reported for each stock is adjusted for dividends and splits and it represents the value at the moment the market closed. The total volume of shares traded is also provided for each stock. The monthly index data are given for a total of 41 different indices. The index data were obtained from http://www.economy.com/freelunch. Some examples of the indices provided include: gross domestic product, income, personal savings, bankruptcy, employment, and trade. The variables in the training set are defined in Table I. 1"',.1
TABLE 1. VARIABLES INCLUDED IN TRAINING SET
closeval volume dateval
ticker stockname index monthval
indexname
Daily closing price of the 2929 stocks Daily volume of the 2929 stocks Date associated with each column of closeval and volume using the "YYYYMMDD" format Ticker associated with each stock Name of each stock Monthly value of the 41 indices Date associated with each column of index using the "YYYYMMOO" format (in all cases the day is arbitrarily set to 00) Name of each index
1-4244-2376-7/08/$20.00 ©2008 IEEE
38
3. TESTING METHODOLOGY Stock and index data for dates that differ from the training set were used to compare the algorithms. Each algorithm started with $100,000 USD, which was used to purchase, sell, cover, and short stocks. The teams submitted Matlab code to us on or before the May 30, 2008 deadline. We then ran each algorithm on the (real) test set. On the first day each algorithm was presented the price of 2929 stocks and 41 indices for 62 contiguous trading days (approximately 3 months). The submitted code was required to return a list of stocks to purchase, sell, cover, and short for the current trading day and the associated number of shares for each transaction. On the second day each algorithm was presented with the previous data except that the information for the oldest date was removed and the information for the next trading day was added. This process was repeated for 125 consecutive trading days, which corresponds to 6 calendar months. On the last day of trading all remaining securities were automatically liquidated and all shorts were automatically covered at the current market value. Only market orders were allowed, as opposed to limit orders, conditional orders, puts, and futures. Both (integervalued) short and long positions were acceptable. The uncommitted cash reserves (total cash reserves minus a transaction fee minus the sum of all short values) and the price of the stock on the specified day was used to determine the maximum number of stocks that could be purchased, shorted, and covered (as opposed to sold; the maximum number of stocks that could be sold is limited by the number of shares currently owned). No calls were enforced nor was any credit extended aside from the implicit credit that may occur after a stock is shorted when the price of that stock rises. A stock could not be sold and shorted in a single transaction. Likewise, covering a short and making a purchase required two transactions. Orders to purchase, sell, cover, or short a stock that did not exist on the current trading day and purchase, short, and cover orders that exceed an entrant's current uncommitted cash reserves were charged a transaction fee, but were otherwise ignored. The transaction fee was $10 for each (successful or attempted) purchase, sale, cover, and short transaction. The code used for testing each algorithm may be obtained from http://www.bme.ogi.edu/l"',.Jhildk/mlsp08Test.m. The validity of each transaction was checked using http://www.bme.ogi.edu/ hildk/updateVariables.m. Each entrant supplied their algorithm in the form of uncompiled Matlab code with the filename myFunction.nl. The actual test data used for the competition were not made available to the entrants prior to testing. However, a sample function and sample test data were provided to the entrants for debugging purposes. The sample code, myFunction.m, may be downloaded from I"',.J
http://www.bme.ogi.edu/l"...Jhildk/myFunction.m.This code is not based on machine learning principles. It was used to acquaint the entrants with the testing procedure. The sample data may be downloaded from http://www.bme.ogi.edu/l"...Jhildk/testdata.mat.This data file was generated using stockPrices = closeval(:,end-185:end); volun1es = volume(:,end-185:end); indices = index(:,end8:end-l );. To use the sample code download all 3 m-files (mlsp08Test.m, updateVariables.m, and myFunction.m) and the data file (testdata.m), place them in the current directory, and run [cashReserves,totaIM,totallnvalid] = mlsp08Test; from the Matlab command line. The variables used in these programs and data files, which also match the variable names of the true test set, are defined in Table II. TABLE II. VARIABLES INCLUDED IN TEST SET
Daily stock price of the 2929 stocks (the current market value appears in the final column) volumes Daily volume of the 2929 stocks indices Monthly value of the 41 indices cashReserves Current (committed plus uncommitted) cash reserves positions The (integer) number of shares owned (positive values correspond to long positions and negative values correspond to short positions) short Value Total value of each shorted stock stocklndex (M xl) vector that denotes (using integer indices) each stock for the M desired transactions shares (M x 1) vector containing the (integer) number of shares to purchase/cover (positive values) or sell/short (negative values) for the M desired transactions userDefined An optional (single) variable (we suggest using a struct variable) that is entirely userdefined and may be used to pass previ0us decisions to the user supplied function, myFunction.m (which is responsible for properly initializing userDefined since it will initially be empty) stockPrices
4. ENTRANTS A total of 4 teams submitted 5 algorithms for this year's competition. The 5 submitted algorithms are briefly described in this section. In the subsequent section we show the performance of all 5 algorithms for both a sample and a real test set. We then pick a winner and a runner-up based on the performance on the real test set.
39
4.1. RPI/SMU The first team, which represents a collaboration between researchers from Rensselaer Polytechnic Institute and Shanghai Maritime University (RPI/SMU), consists of Qingsong Peng and Qiang Ii. The algorithm developed by Peng and Ii uses a Bayesian Network to predict stockPrices. Both the stockPrices and volumes variables are discretized to form the nodes of a Bayesian Network. The Bayesian network is constructed each day to predict the probability that the stock will rise in the following day. The putative number of shares to purchase for a specified stock is given by,
n(t) == In(t)J where
n(t) = (c( t) - 10) p(t)
(1)
* (1 -
r (t) )
(2)
c( t) is the total cash on hand, $10 represents the purchase fee, p( t) is the current price of the given stock, and the relative price is given by,
r-(t) ==
I
p(t) ~t-I
5" L.Jr=t-5 p
(3)
() T
Each of the 10 most highly rated stocks are purchased if the rating produced by the Bayesian Network exceeds a predefined threshold and 17, (t) ~ 50 for the associated stock. Stocks are sold once the profit margin exceeds 5(Xl of the original purchase price.
Edinburgh University is represented by Vipperla Ravichander. The method developed by Ravichander is based on volatility, v(t), opportunity, o(t), and prediction confidence, c( t), which are given by,
- *
+ PVST
(-kr L~~~_NP(T))
o(t) -
c(t)
"'1
vST
_ () t ()
_
t -
Cardiff University is represented Hamid R. Mohseni. The algorithm developed by Mohseni uses a Kalman filter to create fJ( t), which is the denoised price data. A line is then fit to the 10 most recent values of p( t) for a specified stock, the slope of which is designated rn. The putative number of shares to purchase of a specified stock is given by,
n(t) == In(t)J
(8)
where n( t) is defined here as,
n(t) =
(c(t) - 10) p(t)
*b -
v(t))
* r(t)
(9)
c( t) is the cash on hand, p( t) is the current price, , is a userdefined scalar, r(t) is the relative price given by Equation (3), and the volatility, v(t), is defined here as, 1
N
v(t) = N L(p(t) - p(t)).
(10)
- p(t)
I:~-==~-N p( T) == c(t - 1) + 7/ * g(t)
The relative volume of a specified stock is given by,
_ v(t) ==
v(t) I
t-I
5 Lr=t-5 v( T)
(11 )
(5)
where v ( t) is the total trading volume of the stock. A stock is purchased if it has one of the 100 largest values of m and n(t) > 50 and v(t) > 6 and v(t) < 1. A given stock is sold if rn < 0 or if the current price is at least 150/0 larger than the original purchase price.
(6)
4.4. Cambridge University
(4)
where 0' and /i are positive-valued scale parameters, p( t) is the current price, c( 0) is initialized to 1 for each stock, g(t) E {-I, 0,1}, "/ is a step size parameter, and
VLT
4.3. Cardiff University
t=1
4.2. Edinburgh University
v(t) == aVLT
are the long-term and short-term volatility measures, respectively. The values of v (t), o( t), and c( t) are calculated for each stock on each trading day. The set of stocks selected for purchasing and shorting equals the set of stocks having the largest volatility intersected with the set having the largest confidence value. The volume of stocks purchased and shorted is proportional to lo(t) I and the decision to buy or short is based on the sign of o(t). Stocks are sold and covered once a predefined profit margin is exceeded.
Cambridge University is represented by a collaboration between Yunus Saatci and Ryan Turner. Saatci and Turner submitted two algorithms. Both algorithms create a matrix of returns the elements of which are given by,
~ (P(T)-P(T-l))2 L
T=-oo
p(T _ 1)
~ (P(T) - p(T -
L
T=t-N
p(T - 1)
1))2
(7) where p( s, t) is the price of the stock, which explicitly includes the stock index, s. The matrix of returns was reduced
40
to 2 dimensions for algorithm #1 and to 30 dimensions for algorithm #2. The dimension reduction was performed using a matrix constructed of the 2/30 eigenvectors associated with the largest eigenvalues. In algorithm # 1 both signals of the 2-dimensional data were fed into separate Gaussian Process Autoregression (GPA) routines [8], which use the first 4 order statistics of the data to find the predictive distribution. Likewise, in algorithm #2 the 30-dimensional data was fed into a Kalman filter in order to estimate the parameters of the predictive distribution. Afterwards the low-dimensional data are transformed back into the original high-dimensional space using the transpose of the respective orthonormal dimension-reduction matrix. The portfolio selection is restricted to the set of 30 stocks, designated S, that maximized profits for a cross validation set. On each trading day a set of weights, w( s), was found that maximized the expected profit, 1f (t), which is given by,
1f(t) ==
L 11J(s)R(s, t)
(13)
sES
where w( s) == 0 Vs ~ S, 2:s=ES 11J( s) == 1, and 11.'( s) 2: o Vs. Values of w( s) below a user-defined threshold are set to zero and Equation (13) is recomputed. This is repeated until all non-zero values of w (s) exceed the threshold. The set ofweights, w (s), represents the desired number of shares of the associated stocks. The number of shares purchased/sold equals the difference between the desired number of shares and the current holdings (positive values result in a purchase and negative values result in a sale).
5. RESULTS Results are reported for both the sample test data (corresponding to a portion of the training data having the same dimensionality as the testing data) and the real test data. The results are given in terms of the annual rate of return,which is computed daily and uses annual compounding. The definition of the annual rate of return is given by,
a(t) == (d(t))2 _ 1 a(O)
(14)
where a(O) == $100,000, the exponent of 2 (the inverse of 1/2) is used since trading occurs over the course of 125 days of trading, which corresponds to half of a calendar year and d( t) is the total net worth for a specified entrant, which is computed daily. Notice that we (arbitrarily) chose to use a fixed exponent, as opposed to a time-varying exponent. Hence, a( t) implicitly assumes that all assets are tied up for the duration of the testing. The annual rate of return of the training set and test set are 18% and 25%, respectively. Providing the results below for the sample test data allows us to see, to a small extent, how well the training data
represents the test data, although the generalization performance necessarily depends on the set of features that are used. Hence, the generalization performance of each algorithm is expected to differ.
5.1. Sample Test Set The final value of the annual rate of return, a(125), the total number of transactions, T, and the total number of different stocks traded, S, for each of the 5 submitted algorithms on the sample test set are given in Table III. For the sake of reference, the sample program that was freely distributed to the entrants yields an annual rate of return of 28% based on 30 total transactions. Notice that, for these data, the sample program performs exceedingly well and that all 5 ofthe submitted algorithms outperform the sample algorithm. Figure 1 shows a plot of the annual rate ofreturn of the 5 algorithms for the sample test set. TABLE III. SAMPLE TEST SET
I Algorithm
I a(125)
RPI/SMU Cambridge University #2 Cambridge University #1 Edinburgh University Cardiff University
~
170% 64% 59% 32% 32%
40 569 919 139 40
13 24 18 20 19
5.2. Real Test Set The final value of the annual rate of return, a(125), the total number of transactions, T, and the total number of different stocks traded, S, for each ofthe 5 algorithms on the real test set are given in Table IV. Figure 2 shows a plot of the annual rate of return of the 5 algorithms for the real test set. TABLE IV. REAL TEST SET
I Algorithm
I a(125)
RPI/SMU Cardiff University Cambridge University #2 Cambridge University #1 Edinburgh University
150°A> 116% 67% 21% -88%
~ 48 46 774 956 33
8 20 29 20 4
6. DISCUSSION We chose not to enforce calls to simplify testing, which has a downside when a significant portion of one's cash is used to short a stock the price of which subsequently increases. Since the cash associated with the short is committed, it cannot be used for other trades. Additionally, it is not possible to take a loss on the short in order to remove the commitment. The Edinbugh algorithm shorted 22,548 shares
41
200
C'
250
RPI/SMU C'
c
c
Q) ()
l-
Q)
0.
c
... ;:,
Q)
100
Q)
Cambridge #1
0.
...........
... C ;:,
SO
o
...co Q)
0::
a::
50
...co
0
Q)
\t-
0
0
Q)
0::
-50 Edinburgh -100
100
I-
a:: \t-
150
Q) ()
I-
........... l-
Cardiff
200
150
0
0.1
0.2 0.3 Time (years)
0.4
-50 -100
0.5
0
0.1
0.3 0.2 Time (years)
0.4
0.5
Fig. 1. Annual rate of return using the sample test set.
Fig. 2. Annual rate of return using the real test set.
of AMR Corp. at a mean price of $4.74. After the trade the price rose steadily until the end of the trading period, at which point the stock was automatically covered at $11.22. We modified the code to to see how well it would perform if it had not shorted the stock from AMR Corp. The performance of the modified Edinburgh algorithm yielded a very respectable 20% rate of return. The annual rate of return of the S&P 500 and the composite NASDAQ indices from February 1971 until April 2008 is approximately 70/0 and 9°h>, respectively. The training and testing data correspond to a period of economic recovery and, therefore, the annual rates of return of both (18°h> and 250/0, respectively) are higher than normal. The performance of each of the 5 submitted algorithms on the sample test set significantly surpasses the historicallynormal rates of return and that of the market during the sample-test period. The performance of 3 of these algorithms, RPI/SMU, Cardiff University, and Cambridge University #2~ also significantly surpasses that of the market during the real-test period. The RPI/SMU team team is the winner of this year's competition. Their algorithm had a rate of return of 150%. The Cardiff University team is the runner-up. Their algorithm had the highest instantaneous rate of return, but it was not as consistent as the algorithm from RPI/SMU. We would like to congratulate our winners and once again thank all the entrants for submitting their algorithms.
[2] K.Aas andX.K. Dimakos, "Statistical modelling of financial time series: An introduction," Technical report SAMBAl08/04, Norwegian Computing Center Applied Research and Development, 8 March, 2004.
7. REFERENCES
[I] Z. Jelveh, "How a Computer Knows What Many Managers Don't," New York Times, July 9, 2006.
[3] 1. Moody and M. Saffell, "Learning to Trade via Direct Reinforcement," IEEE Trans. on Neural Networks, Vol. 12, No.4, pp. 875-889, July, 2001. [4] R. Gencay and M. Qi, "Pricing and Hedging Derivative Securities with Neural Networks: Bayesian Regularization, Early Stopping, and Bagging," IEEE Trans. on Neural Networks, Vol. 12, No.4, pp. 726-734, July, 2001. [5] H. White and 1. Racine, "Statistical Inference, The Bootstrap, and Neural-Network Modeling with Application to Foreign Exchange Rates," IEEE Trans. on Neural Networks, Vol. 12, No.4, pp. 657-673, July, 2001. [6] A.D. Back, A.S. Weigend, "A First Application of Independent Component Analysis to Extracting Structure from Stock Returns," International Journal of Neural Systems, Vol. 8, No.4, pp. 473-84, August, 1997. [7] M.A.H. Dempster, T.W. Payne, Y. Romahi, and G.W.P. Thompson, "Computational Learning Techniques for Intraday FX Trading Using Popular Technical Indicators," IEEE Trans. on Neural Networks, Vol. 12, No.4, pp. 744-754, July, 2001. [8] C.E. Rasmussen and C.K.I. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
42
