Restricted Ridge Estimator in the Logistic Regression

Communications in Statistics - Simulation and Computation

ISSN: 0361-0918 (Print) 1532-4141 (Online) Journal homepage: http://www.tandfonline.com/loi/lssp20

Restricted Ridge Estimator in the Logistic Regression Model Yasin Asar, Mohammad Arashi & Jibo Wu To cite this article: Yasin Asar, Mohammad Arashi & Jibo Wu (2016): Restricted Ridge Estimator in the Logistic Regression Model, Communications in Statistics - Simulation and Computation, DOI: 10.1080/03610918.2016.1206932 To link to this article: http://dx.doi.org/10.1080/03610918.2016.1206932

Accepted author version posted online: 26 Aug 2016. Published online: 26 Aug 2016. Submit your article to this journal

Article views: 14

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalInformation?journalCode=lssp20 Download by: [Necmettin Erbakan University]

Date: 04 September 2016, At: 23:48

ACCEPTED MANUSCRIPT Restricted Ridge Estimator in the Logistic Regression Model Yasin Asar10 , Mohammad Arashi2 , Jibo Wu3 1 Department 2 Department

of Mathematics-Computer Sciences, Necmettin Erbakan University, Konya, 42090, Turkey

of Statistics, School of Mathematical Sciences, Shahrood University of Technology, Shahrood, Iran

3 Key

Laboratory of Group & Graph Theories and Applications, Chongqing University of Arts and Sciences, Chongqing, 402160, China Abstract It is known that when the multicollinearity exists in the logistic regression model, variance

of maximum likelihood estimator is unstable. As a remedy, Schaefer et al. (1984) presented a ridge estimator in logistic regression model. Making use of the ridge estimator, when some linear restrictions are also present, we introduce a restricted ridge estimator in the logistic regression model. Statistical properties of this newly defined estimator will be studied and comparisons are done in the simulation study in the sense of mean squared error criterion. A real data example and a simulation study are introduced to discuss the performance of this estimator.

Keywords: Ridge estimator; Mean squared error; Restricted ridge estimator AMS Subject Classification: Primary 62J02, Secondary 62J07

0

Corresponding author (Email: [email protected], [email protected])

ACCEPTED MANUSCRIPT 1

ACCEPTED MANUSCRIPT 1

Introduction

Let us consider the following logistic regression model yi = πi + εi , i = 1, . . . , n,

(1)

where exp(xi0 β) , i = 1, 2, . . . , n πi = Pr(yi = 1) = 1 + exp(xi0 β)

(2)

is the expectation of yi when the ith value of the dependent variable is Bernoulli with parameter πi , β = (β0 , β1 , . . . , β p )0 shows the unknown (p + 1)-vector of regression coefficients, xi = (1, x1i , x2i , . . . , x pi )0 presents the ith row of X, the n × (p + 1) data matrix, and εi ’s are independent, with zero mean and variance wi = πi (1 − πi ). In the estimation process of coefficient β, one often uses the maximum likelihood (ML) approach. Making use of the iteratively re-weighted least squares (IRLS) algorithm, the MLE is obtained as ˆ −1 X 0 WZ, ˆ βˆ MLE = (X 0 WX) where Z = (Z1 , . . . , Zn )0 , with Zi = log(ˆπi ) +

(3) yi −ˆπi πˆ i (1−ˆπi )

ˆ = Diag(ˆπi (1 − πˆ i )). and W

In the context of the linear regression model, when the multicollinearity exists, the ordinary least squares (OLS) estimator is no longer a good estimator. To combat the problem of multicollinearity, new biased shrinkage estimators have been presented. As such, the ridge estimator presented by Hoerl and Kennard (1970), biased estimators of Akdeniz et al. (1999), Akdeniz and Erol (2003), and Akdeniz (2004), the restricted ridge estimator presented by Zhong and Yang (2007) and Akdeniz and Tabakan (2009), and the restricted almost unbiased ridge estimator proposed by Xu and Yang (2011a,b). In the logistic regression model, when the explanatory variables are highly correlated, the variance of the MLE becomes inflated. Handling such a problem, some estimators have been introduced. Some of them are the ridge estimator of Schaefer et al. (1984), Liu estimator defined in

ACCEPTED MANUSCRIPT 2

ACCEPTED MANUSCRIPT Mansson et al. (2012), Liu-type estimator studied by Asar (2015) and Asar and Genc (2016) etc. In the presence of restrictions on regression coefficients, Duffy and Santner (1989) suggested to use the restricted MLE (RMLE). Now, let us consider the following linear restrictions Hβ = h

(4)

where H denotes a q × (p + 1) (q ≤ p + 1) known matrix and h shows a q × 1 vector of pre-specified known constants. Considering such a restriction, Duffy and Santner (1989) defined the RMLE with the following form βˆ RMLE = βˆ MLE − C −1 H 0 (HC −1 H 0 )−1 (H βˆ MLE − h),

(5)

ˆ where C = X 0 WX. Schaefer et al. (1984) defined the ridge estimator (RE) in the logistic regression model given by ˆ ˆ = Ck−1 X 0 WZ β(k)

(6)

where k > 0 is the biasing parameter and Ck = C + kI p such that I p is the identity matrix of order p. Based on the RMLE and the ridge estimator, we propose a restricted ridge estimator (RRE) which is given as follows: h i−1 ˆ − Ck−1 H 0 HCk−1 H 0 βˆ H (k) = Ck−1 X 0 WZ h i ˆ −h ∙ HCk−1 X 0 WZ h i−1 h i ˆ −h ˆ − Ck−1 H 0 HCk−1 H 0 H β(k) = β(k)

(7)

where k > 0 is the biasing parameter. It is easy to see that when k = 0, the restricted ridge estimator becomes the RMLE, that is βˆ H (0) = βˆ RMLE = βˆ MLE − C −1 H 0 (HC −1 H 0 )−1 (H βˆ MLE − h).

(8)

ACCEPTED MANUSCRIPT 3

ACCEPTED MANUSCRIPT The paper is organized as follows: Some methods are proposed to choose the parameter k to be used in RRE in section 2 and some methods to choose the biasing parameters are presented in section 3. A simulation study is designed and a real example is demonstrated in section 4. The study is concluded in section 5.

2

Proposed methods to choose the biasing parameter k

In this section, following Muniz and Kibria (2009), we propose some methods to estimate the ridge parameter and use them for our simulation study. Now, let αˆ = γ0 βˆ 2RMLE and m j = followings are our proposed estimators of the biasing parameter k: Following Kibria (2003), we suggest to use mean of 1/m j to get ! p 1X 1 , k1 = p j=1 m j The following estimators are the maximum and minimum elements of 1/m j , respectively ! 1 , k2 = max mj ! 1 . k3 = min mj

b σ2j b α2j

, the

(9)

(10)

(11)

Following Kibria (2003), we use geometric mean and median functions to obtain the following estimators:  p  Y 1 1/p  , k4 =   m j j=1

(12)

! 1 k5 = median . mj

(13)

Finally, following Hoerl et al. (1975) and using harmonic mean of 1/m j , we propose the

ACCEPTED MANUSCRIPT 4

ACCEPTED MANUSCRIPT following estimator: k6 =

p . m1 + ... + m p

(14)

Interested readers may consider Roozbeh et al.(2015), and references therein for more recent related researches.

3

Monte Carlo Simulation

In this section, a Monte Carlo simulation is conducted to study and compare the performance of our new estimator with the proposed methods of selecting the biasing parameter.

3.1

Design of the Simulation

There are important factors in designing the simulation. The main effective factor of this study is the degree of correlation ρ among the explanatory variables. In the present experiment, four different values of ρ as 0.9, 0.95, 0.99 and 0.999 are considered. We implemented a model with p = 4 and p = 8 and sample sizes 50, 100, 200. Following McDonald and Galarneau (1975), we generate the data sets with different strengths of correlation by using xi j = (1 − ρ2 )1/2 zi j + ρzip , i = 1, ..., n, j = 1, ..., p

(15)

where zi j are independent standard normal pseudo-random numbers and ρ is specified so that the correlation between any two explanatory variables is given by ρ2 . The coefficient vector β is chosen as as the normalized eigenvector corresponding to the largest eigenvalue of the matrix C due to Schaefer (1986) such that β0 β = 1. Furthermore, following Månsson et al. (2016), we use different restriction matrices, to study the effect of imposing restrictions on estimators, as follows:

ACCEPTED MANUSCRIPT 5

ACCEPTED MANUSCRIPT   1  I) H1 =   1   1  II) H2 =   1

 1   with h1 = (0, 0)0 when p = 4 and  −1 1 −1   0 −2 1 −3 1 1 1   with h2 = (0, 0)0 when p = 8.  1 0 1 −3 1 −2 1  By this setting, we may understand the effect of restrictions on the performances of the estima0

−2

tors.

In simulations, the estimated mean squared error (MSE) for each estimator is evaluated based on 2000 replications and the whole process is run 2000 times to compute the simulated MSE as follows:

P2000 ˜ 0 ˜ ˜ = r=1 (βr − β) (βr − β) MSE(β) 2000

where β˜ r is any estimator considered in the study in the rth repetition.

3.2

Results and Discussions

We present the result of Monte Carlo simulation in this part. Tables 1-2 contain the simulated MSE values of the estimators when p = 4 and p = 8 respectively. One can observe from tables that the new proposed estimator RRE performs better than RMLE, for all different methods of estimating k. Also, it can be observed from tables that for a given n, although there is some degeneracy in the monotonicity of the situation, increasing the degree of correlation makes an increase in the MSE values in general. More especially, the MSE of RMLE is inflated when sample size is small and degree of correlation is high. Only the estimating methods k3 and k6 have a little inflation when compared to RMLE by an increase in the correlation. However, the other methods work fine in this case. Furthermore, increasing the sample size affects the performances of the estimators, most of the

ACCEPTED MANUSCRIPT 6

ACCEPTED MANUSCRIPT time. There are some exceptions when p = 4, n = 200 and ρ = 0.90, 0.95 and when p = 8, n = 200 and ρ = 0.90. Thus, we can say that high correlation makes a positive effect on the asymptotic properties of the estimator. Usually, k3 and k4 have quite better performances than the others.

4

Real Data Application

In order to see superiority of our estimator compared to the MLE and RMLE in real applications, we implemented our result in a real data example. The dataset is about a study of myopia taken from Hosmer et al. (2013). There are 618 subjects who had at least five years of follow up and were not myopic when they entered the study. There are 17 variables in the full dataset. However, we only used the followings as explanatory variables: spherical equivalent refraction (SPHEQ), axial length (AL), anterior chamber depth (ACD), lens thickness (LT), vitreous chamber depth (VCD) which are all continuous variables of same scale (mm). The dependent variable is whether a subject has myopia or not. Subjects having myopia coded as 1 and not having myopia coded as 0. The data matrix X is centered and standardized so that X 0 X is in the correlation form. The computed √ condition number κ = λmax /λmin , which is a measure of multicollinearity (λmax and λmin are the maximum and minimum eigenvalues of X 0 X respectively), is about 403.6198 that shows there is exists severe multicollinearity. We present the correlation matrix of the data in Table 3. According to Table 3, we compare the explanatory variables having the positive correlation and negative correlation between each other and we also see that there is a high correlation (0.94) between the variables AL and VCD. Therefore the restriction matrix H = (1, −1, −1, 1, 0) is used with h = 0. The logistic regression model is fitted using the IRLS algorithm. Estimated regression coefficients, standard errors and MSE values for different estimators are tabulated in Table 4. According to Table 4, all the methods have almost equal MSE values which are much more smaller than that of RMLE. MSE of RMLE is inflated and its standard errors are large compared to the other estimators. The plot of MSEs of

ACCEPTED MANUSCRIPT 7

ACCEPTED MANUSCRIPT RRE and RMLE as a function k is depicted in Figure 1. According to Figure 1, for all positive values k, RRE has less MSE value than that of RMLE.

5

Conclusion

We introduced a new restricted ridge estimator namely RRE for the coefficient of logistic regression model. Some methods to choose the biasing parameter have been proposed and a Monte Carlo experiment has been designed to evaluate the performance of the estimator RRE according to MSE criterion and demonstrate its superiority numerically. In addition, a real data application is also considered to illustrate benefits of using the new estimator in the context of restricted models. In conclusion, the use of RRE is recommended when multicollinearity is present in the restricted logistic regression model.

Acknowledgements This work was supported by the National Natural Science Foundation of China (No. 11501072), the Natural Science Foundation Project of CQ CSTC (No. cstc2015jcyjA00001), and the Scientific and Technological Research Program of Chongqing Municipal Education Commission (No. KJ1501114).

ACCEPTED MANUSCRIPT 8

ACCEPTED MANUSCRIPT References [1] Akdeniz, F. (2004). New biased estimators under the LINEX loss function. Statist.Papers, 45 175190. [2] Akdeniz, F. and Erol H. (2003), Mean squared error matrix comparisons of some biased estimators in linear regression. Comm. Statist. Theo. Meth., 32(12) 23892413. [3] Akdeniz, F., Erol, H., and Ozturk, F. (1999). MSE comparisons of some biased estimators in linear regression model. J.Appl. Statist. Sci., 9(1), 7385. [4] Akdeniz, F. and Tabakan, G. (2009). Restricted ridge estimators of the parameters in semiparametric regression model. Comm. Statist. Theo. Meth., 38(11) 1852-1869. [5] Asar, Y. and Genc, A. (2016). New Shrinkage Parameters for the Liu-type Logistic Estimators. Comm. Statist. Sim. Comp. 45:3, 1094-1103, DOI:10.1080/03610918.2014.995815. [6] Asar Y. (2015). Some New Methods to Solve Multicollinearity in Logistic Regression. Comm. Statist. Sim. Comp. DOI:10.1080/03610918.2015.1053925. [7] Baksalary, J. K. and Kala, R. (1983). Partial orderings between matrices one of which is of rank one. Bull. Polish Academy Sci.. Mathematics, 31:5-7. [8] Duffy, D. E. and Santner, T. J . (1989). On the small sample properties of norm-restricted maximum likelihood estimators for logistic regression models. Comm. Statist. Theo. Meth., 18:959-980. [9] Hoerl, A. E., and Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1) 55-67. [10] Hoerl, A. E., Kannard, R. W., and Baldwin, K. F. (1975). Ridge regression: some simulations. Comm. Statist. Theo. Meth., 4(2), 105-123.

ACCEPTED MANUSCRIPT 9

ACCEPTED MANUSCRIPT [11] Hosmer , D. W., Lemeshow, S., and Sturdivant, R. X. (2013). Applied logistic regression (Vol. 398): John Wiley and Sons. [12] Kibria, B . M . G. (2003). Performance of some new ridge regression estimators. Comm. Statist. Theo. Meth, 32 419-435. [13] Månsson, K., Kibria, B. G. and Shukur, G. (2012). On Liu estimators for the logit regression model. Economic Model., 29(4) 1483-1488. [14] Månsson, K., Golam Kibria, B. M., and Shukur, G. (2016). A restricted Liu estimator for binary regression models and its application to an applied demand system. J. Appl. Statist., 43(6) 1119-1127. [15] McDonald, G.C. and Galarneau D.I. (1975). A Monte Carlo evaluation of some ridge-type estimators. J. Am. Stat. Assoc., 70(350) 407416. [16] Muniz, G. and Kibria, B. M. G. (2009). On some ridge regression estimators: An Empirical Comparisons. Comm. Statist. Sim. Comp., 38 621-630. [17] Roozbeh, M., Arashi, M. and Kibria, B. M. Golam. (2015). Generalized ridge regression estimator in semiparametric regression models. J. Iranian Statist. Soc.(JIRSS), 14(1) 25-62. [18] Schaefer, R. L. (1986). Alternative regression collinear estimators in logistic when the data are collinear. J. Stat. Comput. Simul. 25:7591. [19] Schaefer, R . L ., Roi, L. D. and Wolfe, R . A. (1984). A ridge logistic estimator. Comm. Statist. Theo. Meth., 13 99-113. [20] Xu, J. W. and Yang, H, (2011a). More on the bias and variance comparisons of the restricted almost unbiased estimators. Comm. Statist. Theo. Meth., 40 4053-4064.

ACCEPTED MANUSCRIPT 10

ACCEPTED MANUSCRIPT [21] Xu, J. W. and Yang, H., (2011b). On the restricted almost unbiased estimators in linear regression. J. Appl. Statist., 38 605-617. [22] Zhong, Z., and Yang, H. (2007). Ridge estimation to the restricted linear model. Comm. Statist. Theo. Meth., 36(11) 2099-2115.

ACCEPTED MANUSCRIPT 11

ACCEPTED MANUSCRIPT

Table 1: Estimated MSEs when p = 4 ρ n k1 k2 k3 k4 k5 k6 RMLE

50

ρ = 0.9 100

200

50

ρ = 0.95 100

200

50

ρ = 0.99 100

200

50

ρ = 0.999 100

200

0.4391 0.6587 0.5898 0.3119 0.3617 0.4539 3.0606

0.2869 0.5556 0.4909 0.2285 0.2546 0.3706 2.4157

0.0613 0.2264 0.7039 0.3554 0.3346 0.5547 1.2102

0.5506 0.7250 0.8703 0.3962 0.4878 0.6262 8.4471

0.1945 0.4585 0.5649 0.2112 0.2473 0.3533 2.7895

0.0449 0.2140 0.5337 0.2863 0.3378 0.4063 1.4710

0.6766 0.8168 0.5537 0.5769 0.5983 0.5854 27.1401

0.4893 0.6907 0.3737 0.4328 0.4421 0.4390 15.7374

0.1234 0.3095 0.3004 0.2153 0.2470 0.2572 5.5328

0.8719 0.9234 0.7366 0.8126 0.8596 0.7371 357.1256

0.6754 0.7970 0.3555 0.6219 0.6507 0.5355 111.1632

0.4371 0.5994 0.2298 0.4144 0.4369 0.3585 48.0820

ACCEPTED MANUSCRIPT 12

ACCEPTED MANUSCRIPT

Table 2: Estimated MSEs when p = 8 ρ n k1 k2 k3 k4 k5 k6 RMLE

50

ρ = 0.9 100

200

50

ρ = 0.95 100

200

50

ρ = 0.99 100

200

50

ρ = 0.999 100

200

0.8237 0.9257 4.7795 0.6571 0.6967 2.1253 72.8478

0.3752 0.6608 3.7779 0.3011 0.2881 2.0900 7.9876

0.1378 0.4359 2.9368 0.3775 0.2671 1.8421 4.5065

0.8808 0.9565 5.8129 0.7114 0.7420 2.4238 141.1969

0.4579 0.7274 4.8727 0.2582 0.3101 2.2908 14.4209

0.2078 0.5512 4.3714 0.2351 0.1878 2.3863 8.7724

0.9591 0.9880 6.7835 0.8605 0.8704 2.5169 663.9299

0.8145 0.9304 6.0951 0.5832 0.6503 2.6621 89.6520

0.5183 0.7819 5.9501 0.2376 0.3150 2.4152 41.4891

0.9931 0.9982 13.9455 0.9640 0.9688 10.2667 12410.8706

0.9516 0.9855 7.9896 0.8298 0.8542 3.9311 747.2253

0.8922 0.9660 6.7975 0.6822 0.7257 2.3866 380.5898

ACCEPTED MANUSCRIPT 13

ACCEPTED MANUSCRIPT

Table 3: Correlation matrix of the data set SPHEQ AL ACD LT VCD

SPHEQ

AL

ACD

LT

VCD

1.0000 -0.3055 -0.2388 0.0727 -0.2471

-0.3055 1.0000 0.4563 -0.3289 0.9419

-0.2388 0.4563 1.0000 -0.3393 0.1994

0.0727 -0.3289 -0.3393 1.0000 -0.4516

-0.2471 0.9419 0.1994 -0.4516 1.0000

ACCEPTED MANUSCRIPT 14

ACCEPTED MANUSCRIPT

Table 4: Coefficients, standard errors, p-values and MSE values of estimators Coefficients k1

k2

Estimated k3

coefficients k4

k5

k6

RMLE

SPHEQ AL ACD LT VCD

-0.0016 -0.0008 -0.0002 0.0006 0.0000

-0.0004 -0.0002 0.0000 0.0002 0.0000

-0.1345 -0.0661 -0.0149 0.0534 0.0009

-0.0088 -0.0043 -0.0010 0.0035 0.0000

-0.0056 -0.0027 -0.0006 0.0022 0.0000

-0.0475 -0.0233 -0.0053 0.0189 0.0000

-15.9497 -21.2064 7.6405 2.3838 16.3298

SPHEQ AL ACD LT VCD

k1 0.0004 0.0004 0.0004 0.0004 0.0005

k2 0.0001 0.0001 0.0001 0.0001 0.0001

Standard k3 0.0328 0.0288 0.0299 0.0342 0.0446

Errors k4 0.0021 0.0019 0.0019 0.0022 0.0029

k5 0.0014 0.0012 0.0012 0.0014 0.0019

k6 0.0116 0.0102 0.0105 0.0121 0.0159

RMLE 2.6917 4.2689 3.0339 2.0791 3.7533

0.3565

0.3567

0.3427

0.3554

0.3559

0.3503

59.9861

MSE

ACCEPTED MANUSCRIPT 15

ACCEPTED MANUSCRIPT

Figure 1: Plot of MSE values of RRE and RMLE for different values of k

ACCEPTED MANUSCRIPT 16