New ALS Methods With Extrapolating Search Directions and ... - PolyU

5888

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 59, NO. 12, DECEMBER 2011

New ALS Methods With Extrapolating Search Directions and Optimal Step Size for Complex-Valued Tensor Decompositions Yannan Chen, Deren Han, and Liqun Qi

Abstract—In signal processing, data analysis and scientific computing, one often encounters the problem of decomposing a tensor into a sum of contributions. To solve such problems, both the search direction and the step size are two crucial elements in numerical algorithms, such as alternating least squares algorithm (ALS). Owing to the nonlinearity of the problem, the often used linear search direction is not always powerful enough. In this paper, we propose two higher-order search directions. The first one, geometric search direction, is constructed via a combination of two successive linear directions. The second one, algebraic search direction, is constructed via a quadratic approximation of three successive iterates. Then, in an enhanced line search along these directions, the optimal complex step size contains two arguments: modulus and phase. A current strategy is ELSCS that finds these two arguments alternately. So it may suffer from a local optimum. We broach a direct method, which determines these two arguments simultaneously, so as to obtain the global optimum. Finally, numerical comparisons on various search direction and step size schemes are reported in the context of blind separation-equalization of convolutive DS-CDMA mixtures. The results show that the new search directions have greatly improve the efficiency of ALS and the new step size strategy is competitive. Index Terms—Alternating least squares, block component model, CANDECOMP, CDMA, complex step size, PARAFAC, search direction, tensor decompositions.

I. INTRODUCTION

A

-order tensor is a quantity of which the elements are addressed by indices. The problem of decomposing a higher-order tensor into a sum of products of lower-order tensors finds more and more important applications in signal pro-

Manuscript received January 16, 2011; revised April 22, 2011 and July 28, 2011; accepted July 29, 2011. Date of publication August 15, 2011; date of current version November 16, 2011. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Hing Cheung So. The work of D. Han was supported by the NSFC Grants 11071122, 10871098 and by the Natural Science Foundation of Jiangsu Province Grant No. BK2009397. The work of L. Qi was supported by the Research Grant Council of Hong Kong. Y. Chen is with the School of Mathematical Sciences and Key Laboratory for NSLSCS of Jiangsu Province, Nanjing Normal University, Nanjing 210046. He is also with the College of Science, Nanjing Forestry University, Nanjing 210037, China (e-mail: [email protected]). D. Han is with the School of Mathematical Sciences and Key Laboratory for NSLSCS of Jiangsu Province, Nanjing Normal University, Nanjing 210046, China (e-mail: [email protected]). L. Qi is with the Department of Applied Mathematics, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong (e-mail: maqilq@polyu. edu.hk). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TSP.2011.2164911

cessing, data analysis, and scientific computing [3], [10], [20], [26], [28]. This decomposition is a generalization of singular value decomposition (SVD) that gives a low-rank approximation to a matrix (a second-order tensor), while a direct generalization of SVD is nontrivial. To describe the problem under consideration, we briefly introduce the CANDECOMP/PARAFAC (CP). Definition I.1: (CP): A CP of a third-order tensor is a decomposition of as a linear combination of a minimal number of rank-1 tensors,

where denotes the outer product, , and are the columns of three matrices , and , respectively. Higher-order extension of CP is straightforward. The CP can be dated back to the work of Hitchcock in 1927 [18], [19]. However it is out of the interests of researchers until the study of Carroll and Chang [4] and Harshman [17] in the fields of psychometries and phonetics in 1970, respectively. Recently, people find its applications in chemometrics [28] and independent component analysis [5], [11]. Moreover, Sidiropoulos et al. get its applications in wireless communication [27], where they propose a blind CP-based receiver for instantaneous CDMA mixtures impinging on an antenna array. In several applications, a tensor might result from contributions that are not rank-1 tensors. Definition I.2: (Mode-n Product): The mode-2 and mode-3 by the matrices products of a third-order tensor and , denoted by and , tensor and an tensor, result in an respectively, with elements defined by

Nonrank-one contributions lead to another decomposition, block component model (BCM), which is more general than CP and is defined as follows. Definition I.3: (BCM): A third-order tensor follows a BCM if it can be written as

1053-587X/$26.00 © 2011 IEEE

(1)

CHEN et al.: NEW ALS METHODS FOR TENSOR DECOMPOSITIONS

The vectors , , and of CP and two matrices are now replaced by a tensor and , respectively. This model is recently independently introduced by de Almeida et al. [8], [9] and by De Lathauwer [12], [13], [22], [25]. Computationally, ALS is a classical and unsophisticated method for CP [3], [17], [20], [28]. However, ALS needs a large number of iterations to converge due to the fact that the convergent rate of many iterations is almost null. To accelerate the rate of convergence of ALS, the line search along the incremental direction of an old iterate to a new one is proposed by Harshman [17]. For real-valued tensors, the optimal step size can be calculated because it is a solution of a polynomial equation with a single argument. This method is called “enhanced line search” (ELS) [6], [29], [30]. Since BCM is a generalization of CP, ALS for CP is generalized to compute BCM immediately [9], [14], [22], [25]. More recently, for complex-valued tensors that follow CP and BCM, Nion and De Lathauwer propose a line search scheme that is named as “enhanced line search with complex step” (ELSCS) contains [23], [24], where a complex step size factor two independent arguments: modulus and phase . In order to find the optimal step size, ELSCS proceeds in an alternating minimization manner, which consists of two steps: firstly, find for a fixed phase ; secondly, find the the optimal modulus optimal for a fixed . That is to say, ELSCS needs to solve two polynomial equations alternately to update and . After several iterations, a local optimal step size factor appears. ELS and ELSCS can enhance the performance of ALS, which is verified numerically in [23], [24], [29], and [30]. Nonetheless, the current search direction in ELS and ELSCS is the difference between two successive previous iterates, which sometimes, due to the nonlinearity of the cost function in ALS, is not suitable to generate search directions that accelerate the rate of convergence greatly. Comon et al. give an example in Section V-B of [6] to illustrate the poorness of linear direction, and they assert that: “if the directions provided by the algorithm are bad, ELS cannot improve much on the convergence.” From the viewpoint of numerical optimization, the higherorder information is necessary to construct efficient search directions. Some gradient based optimization methods, including nonlinear conjugate-gradient, limited-memory BFGS and truncated Newton’s method, are employed to exploit higher-order search directions for CP [1], [2]. However, if the gradient of the cost function is unknown or expensive to compute, such as the has a one considered in Section V where an argument matrix special Toeplitz structure, these optimization methods may lose their advantages. Based on ALS, we propose two extrapolating search directions containing higher-order information. The obtained search directions enjoy the advantages of preserving the structure of contributions and requiring slight additional costs of computation. At the same time the efficiency of the corresponding ALS methods can be enhanced greatly. The first direction is called “geometric search direction”, which is formed by adding the transformation of directions to the often used linear direction. The resulting direction is a linear

5889

combination of three previous iterates. When the combination parameter is set to zero, the geometric search direction reduces to a linear direction. The freedom in setting the combination parameter provides us the possibility of obtaining better search directions. The second direction is named “algebraic search direction”, where the higher-order information is collected by exploiting the quadratic extrapolation technique. The resulting direction is a convex combination of the quadratic extrapolating direction and the linear direction, and it is also a linear combination of three previous iterates. After setting down the search direction, we proceed to find the modulus and phase of the optimal complex step size factor. Instead of ELSCS that finds them alternately, we broach a direct method that solves a system of two polynomial equations with two arguments simultaneously. Using the results of algebraic geometry [7], such system of polynomials has solutions if and only if its resultant vanishes, where the resultant is a determinant of a single argument polynomial matrix. We give a novel method to compute all roots of the resultant. Then, all the solutions of the polynomial system can be found immediately. Finally, the optimal step size factor is the one that minimizes the cost function restricted in the search direction. This paper is organized as follows. The geometric and the algebraic search directions are presented in Section II. In Section III, we broach the technique of computing the optimal complex step size factor. The new ALS algorithm is described in Section IV. Some preliminary simulation results are reported in Section V. Finally, some concluding remarks are drawn in Section VI. II. SEARCH DIRECTIONS A. Preliminaries and the Linear Search Direction Given a tensor , a BCM of consists of the , and a estimation of two matrices for such that definition (1) tensor and the matrices holds. Denote and as the resulted from the concatenation of the matrices and , matrix whose elements respectively. Denote as the , which are stacked as follows are the elements of the tensors . Thus, a BCM of can be formulated by finding , and such that

where is the matrix representation of with elements , and denotes the partitiondefined as wise Kronecker product of and . This product results in a matrix defined by , and denotes the Kronecker product. of , Computationally, the task is to find an estimator of , , and which builds from the estimators , , and , such that the following cost function (2)

5890


is minimized. Here, is the Frobenius norm. When , a BCM reduces to a CP, and the partition-wise Krois replaced by the Khatri-Rao product that necker product is also called a column-wise Kronecker product. Owing to the multilinearity of BCM in three estimators, people consider one estimator at a time in minimizing the cost function, while fixing the other two. Obviously, this is a linear least squares problem that is easy to solve. Then, three estimators are found by alternately solving these linear least squares problems. In this way, ALS algorithm appears. Harshman observes via simulation that sometimes a large number of ALS iterates have almost null convergence rate and the estimators here increase gradually along fixed directions iteration, [17]. To accelerate the rate of convergence, at the he takes the increment in estimators as the linear search directions (3) (4) (5)

In the same way, we can get the geometric directions of estimators and

(7) (8)

C. Algebraic Search Direction Let us consider using the quadratic interpolation technique to extrapolate the new iterates and form the new algebraic extrapolating search directions. as a sample from We also take for example. Consider a continuous path in an unknown argument , and the selocate in such that quence of iterates . If we know the information about , , and , we can form a quadratic approximation to by interpolating the three pieces of information available to obtain

where , , and are the estimators of , , and , respectively, obtained from the ALS iteration , , and are the estimators obtained and ALS iteration. These linear directions are from the widely used. B. Geometric Search Direction Since the cost function (2) is obviously nonlinear, the search directions (3)–(5) based on linearization are not good enough to accelerate the rate of convergence of ALS. Now, we add some higher-order information to the linear search directions and get the new geometric extrapolating search directions. We take an estimator for example, and the geometric search and could be obdirections in the other two estimators , three pretained in a similar way. Suppose at iteration , and are known. At iterate vious iterates , is a forward linear search direction and is a backward linear search direction. From this point of view, the change from the backward direction to the forward direction is . When the rate of convergence is slow, is not . Therefore, we regard that the variation trend far from of the directions remains hold. So we add some variational information to the linear direction (3) and get the new geometric extrapolating search direction

For simplicity, we make an additional assumption that the prediction of the new iterate and three previous iterates , , have the identical distances, that is to say, . So the is not dependent on . Then, by some expression of calculation, we have . The displacement from to forms the quadratic approximate search direction

(9) The algebraic extrapolating search direction is the convex combination of the linear search direction (3) and the quadratic one (9)

(10)

(6) is a positive parameter. When tends to zero, the where geometric direction tends to the linear direction. Thus, a small value of is preferable. It is worthwhile to note that the geometric direction preserves the Toeplitz structure in an argument matrix .

. The Toeplitz structure in is also preserved. where In the same way, we get the algebraic extrapolating search directions of and ,

(11) (12)


5891

Note that, the geometric direction (6) is a linear combination of the linear direction (3) and the quadratic direction (9) with and , respectively. Therefore, both the coefficients geometric directions and the algebraic ones have some secondorder information. Moreover, the computational cost of them is ignorable in numerical algorithms.

matrices , , , and as (where and are omitted for convenience of

(17) Then the objective function (16) can be rewritten in a compact form as

III. THE OPTIMAL STEP SIZE At iteration , when the search directions , are determined, we perform a line search process

Denoting the the superscripts notation)

and (18) where (13)

(19)

(14)

matrix, Vec is the operator that writes a matrix in a vector format by concatenation of the columns , and such that the superscript denotes the conjugate transpose. . Then it can be represented Define a 4 4 matrix for by a componentwise form , where and . Let be a complex step size factor, where and are its modulus and phase, respectively. , then and . Define The task is now to find the optimal and that minimize the objective function (18). with respect to can be exThe partial derivative of pressed as

(15) where is a step size factor and the directions here can be the geometric search directions (6)–(8) or the algebraic search , , and directions (10)–(12). Then, started from , ALS iterates are performed to get the next estimators , and . In the line search framework, the step size scheme affects the rate of convergence of a numerical algorithm. Harshman choose a fixed value between 1.2 and 1.3 to be the step size factor [17]. For real-valued tensors, Rajih et al. introduce the ELS that calculate the optimal step size factor by rooting a polynomial [29], [30]. For the complex-valued tensors, Nion and De Lathauwer propose ELSCS that is an alternating polynomial rooting algorithm to approximate the optimal complex step size factor [23], [24]. The optimal complex step size factor is the minimum of the following objective function in a single complex argument

is an

(20) , and where is a 7 6 matrix (21), shown at the bottom of the page. On the with respect to can other hand, the partial derivative of be expressed as (22)

(16)

where is a 7 next page.

6 matrix, shown in (23) at the bottom of the

(21)

5892

The optimal condition for a complex optimal step size factor is that


to be the

From the partial derivatives (20) and (22), the above optimal is a solution of the following condition holds if and only if system of two polynomial equations:

roots of the resultant can be found via solving a determinant of the following 9 9 polynomial matrix

(26)

(24) with some Then, we can regard and as polynomials in polynomials in as its coefficients. That is to say, we rewrite (24) as

(25) where and are sixth-order column of an identity matrix. polynomials in , and is the By the Sylvester theorem [7], (25) has solutions if and only if its resultant vanishes. The resultant here is a determinant of the following 10 10 polynomial matrix:

is a 54th-order polynomial Obviously, the determinant in , due to the polynomials for and for are all of order six. The 54th-order polynomial here suffers from a trouble that its coefficients are complicated and unobtainable. We give a novel method to compute all the roots of the de. Since all the elements of the matrix are terminant sixth-order polynomials in except some zeros, we rewrite as

where the elements of are the coefficients of -order term of the corresponding elements of . That is to say

The useful roots of this resultant are solved as follows. First, notice that . Then

are solutions of (25) with objective function value . These two roots are useless, since their modului are zeros. The other

(27) . for Then, we have the following important theorem, which is an extension of the method in [15]. And it is the support theory of [21]. a Matlab function roots when

(23)


5893

Theorem III.1: Suppose for are matrices, and is nonsingular. Then all the roots of the th-order polynomial equation

are exactly the eigenvalues of the following

.. .

.. .

..

matrix

.. .

.

.. .

(28) where is a identity matrix. From this theorem, all the roots in of the determinant can be found via computing all the eigenvalues of the matrix and . Hence, it is stable. Then, we ig(28) with nore complex-valued roots since they are meaningless. For each real-valued root , we substitute it into the polynomials system (25) and solve the system to get the corresponding modului . After this, we find all the critical points of the objective function . Finally, the optimal step size factor is the one that gives the smallest objective function value. To state succinctly, we sum up an algorithm.

ELSCS. In the context of line search, we believe the phase is is also close to zero. close to zero, so the parameter should Omitting some higher order terms of polynomial not greatly affect the optimum. Hence, for the purpose of easily computing and resulting in a real-valued root, we truncate the polynomial to a linear function. In this case, we only need to . The constant compute the constant and linear terms of term is a determinant . The linear term is a sum of deterfor , where is a 9 9 matrix minants row of by the corobtained by replacing the elements of responding elements of . Hence, the computational cost is ten 9 9 determinants that needs flops. Whenever the parameter is determined, we compute the optimal modulus by ELS that is used in ELSCS. We argue that this heuristic scheme is as cheap as ELSCS at least. IV. THE NEW ALS ALGORITHM Based on the analysis above, we give the new ALS algorithm with extrapolating search direction and optimal step size schemes. New ALS Initialize: Choose three iterates , , , , and set

Given: iterates , and

,

S1. Compute

for

, and directions

and

is from (17) and (19).

in (21) and (23), respectively. in (27) and build (28).

S4. Compute all the eigenvalues of (28). S5. For each real , find the corresponding real by (25). Then, the optimal step size is the one with the minimal objective function value. The matrix is necessary for both ELSCS and the method for in (17) requires presented here. Computing flops. Then, computing requires flops because of symmetry. Therefore, the total number . of operations for is The computational cost after of the optimal complex step size method here comes mainly from the computation of all the flops eigenvalues of a 54 54 matrix (28), where about are required theoretically [16]. On the other hand, ELSCS requires a few alternating miniand to get a step size factor. Thus, its mizations between cost is computing the roots of some fifth- and sixth-order polyflops. It’s cheap. nomials, which needs We give a heuristic strategy to compute an approximate optimal step size factor, since computing all the eigenvalues of the 54 54 matrix (28) is time-consuming when compared with

,

,

,

) do

S1. Compute directions (6)–(8) or (10)–(12). .

S2. Find the optimal step size factor ,

S3.If , where

S2. Calculate S3. Form

and

.

, .

(e.g.,

while Subroutine for the optimal complex step size.

,

build

and

from (14) and (15);

else and

set

.

S4. Update

by

S5. Update

by

and

S6. Update

by

and

S7. Set

and

. . .

.

end The ALS iterations including S4-S6 are the same as the ones presented in [22] and [25]. In the remainder of this section, we go to the details and analyze their computational complexity. For the convenience of notation, the superscripts are omitted. are known. To preserve the First, update , when and Toeplitz structure, we update the generator vector of matrix . From (1), , where , , and . requires Computing flops. Then . Define and as the frontal slice of the . Rearrange correspond tensors, then to get such that , where

5894


is a sparse matrix with at most each column. Define

.. . and Then

.. .

with

nonzeros in

.. . .

where is a sparse matrix with at most nonzeros in each column. Solving the equation by sparse flops. matrix algorithm requires is Therefore, the total number of operations for updating . are known. Computing Second, update , where and requires flops. Then . Rearranging the tensors here as a matrix form, we get , where , and . Then Computing the pseudoinverse maand the matrix multiplication require trix and flops, respectively. Thus, there are operations. Finally, update , where and are known. Since , computing and its pseudoinverse matrix require and flops, respectively. Then computing requires flops. Totally, flops are required. V. SIMULATION RESULTS We are going to examine the performance of the new search directions and the new step size. For these purpose, we apply these techniques to solve the problem of blind separation-equalization of convolutive DS-CDMA mixtures received by an antenna array after multipath propagation, and compare them with the basic ALS and the ALS with ELSCS. We make the same assumptions as those in [22], [24], and [25], i.e., we assume that the signal of the th user is subject to inter-symbol-interference over consecutive symbols and that this signal arrives at antenna array via specular paths. For user , the frontal slice of collects samples of the convolved spreading waveform associated to the path, . The matrix holds the transmitted symbols and has a Toeplitz structure. The matrix collects the response of the antennas according to the angles of arrival of the paths. users, pseudorandom In our experiments, we consider , a short frame of QPSK spreading codes of length antennas, interfering symbols, and symbols, paths per user. The signal-to-noise ratio (SNR) at the input of the BCM receiver is defined by , where is the complex-valued noise-free tensor of observa-

Fig. 1. BER versus SNR.

tions and is the tensor of zero-mean white (in all dimensions) Gaussian noise. We run 1000 Monte Carlo experiments. For each one, the initialization of the iterate is performed by selecting the best one from ten different random starting points, which obey the standard Gaussian distribution. We stop the algorithm when the iteration number exceeds 500 or the stopping is satisfied, where . tolerance A. The First Result: Step Size In line search methods, the step size is important. Here, in ALS framework, we test five sorts of step size strategies with the same linear search directions. • “Basic ALS”, whose step size factor is always one. • “ELSCS” that seeks the complex step size factor in an alternating manner [24]. • “OSS-real” that finds the optimal real step size factor. • “OSS-complex” is the optimal complex step size. • And “OSS-simplify” is the heuristic implementation of the optimal complex step size scheme described in the last paragraph of Section III. Next, we give the results based on statistics. Which line search scheme gives the most accurate signal estimation? We show the bit error rate (BER) over all users in Fig. 1. First, OSS-complex works as good as ELSCS. Second, compared with them, OSS-real and OSS-simplify work a little bit worse. We recall that the imaginary part of the step size factor of OSS-real and OSS-simplify is zero and a heuristic guesswork, respectively. Thus, iterates are easy to be trapped around a false local optimum, sometimes. Which line search scheme is the fastest one? We illustrate in Table I the possibility of each step size strategy being the winner of these five strategies from a numerical viewpoint. The probability that OSS-simplify wins on a given problem in terms of iteration number is about 72.6%, and the corresponding probability in terms of CPU time is about 93.3%. We say two algorithms have the same CPU time if the difference of the corresponding CPU times is less than a tenth of a second. This rule is also valid in Table II.


5895

TABLE I THE POSSIBILITY THAT EACH STEP SIZE STRATEGY WINS (%)

TABLE II THE POSSIBILITY THAT EACH SEARCH DIRECTION WINS (%)

Fig. 4. Iterations versus values of and .

Fig. 2. Performance of mean number of iterations.

• Basic ALS that takes a fixed unit step size converges slowly. Hence an adaptive step size strategy is necessary. • Compared with OSS-real, the complex step size strategies could improve ALS by saving about five iterations. Hence, the imaginary part of the step size factor is valuable. Consequently, a real step size is not recommended in algorithms for complex-valued tensor decompositions. • When we look into the three complex step size strategies, OSS-complex takes exactly the optimal complex step size factor, ELSCS may take a local optimal one and OSSsimplify takes an approximate one. Compared with the OSS-complex, ELSCS saves about two ALS iterations, and OSS-simplify saves about seven ALS iterations. This phenomena confirms that in numerical applications, inexact line search methods perform better than the exact one. On the other hand, the comparative result of the CPU times for the three complex step size schemes are almost the same as that of iterations. From this observation we argue that the costs for computing a complex step size factor by these schemes are negligible when compared with the costs of ALS iterations (forming the search direction). • ELSCS performs quite well. The main reason is that the local minimum of the objective function is usually unique, which means that ELSCS always obtains the global optimal complex step size factor. B. The Second Result: Directions

Fig. 3. Performance of mean CPU times.

From Table I, although the OSS-complex do not work well, the OSS-simplify that is a heuristic implementation of OSScomplex outperforms all the other line search strategies. How much does OSS-simplify faster than the others? Figs. 2 and 3 illustrate the performance of mean value as well as standard deviation of iterations and CPU times, respectively. Form the two figures, we get the following conclusions.

First, we assign two values to the parameters and involved in the geometric and algebraic search directions, respectively. Since no ideal of solving this problem from a theoretical point of view currently, we run 500 Monte Carlo tests with . Fig. 4 shows the mean iterations versus some and are the values of and . Obviously, best choices for these two directions, respectively. Therefore, we keep these values for and in the following simulations. Next, we test four kinds of search directions in an ALS framework to observe their performances, where all the step size scheme are the fastest OSS-simplify. • “None”: basic ALS, no line search.

5896


Fig. 5. BER versus SNR.

Fig. 6. Performance of mean number of iterations.

• “Linear”: linear directions (3)–(5) that is often used. • “Geometric”: geometric directions (6)–(8). • “Algebraic”: algebraic directions (10)–(12). We now give some results based on statistics. Which search direction gives the most accurate signal estimation? The average bit error rate (BER) over all users is illustrated by Fig. 5. Compared to the often used linear directions, the new geometric and algebraic directions have more chance to point to the accurate decomposition of tensors. Which search direction is the fastest one? See Table II. Compared with the low percentage that the linear direction wins, the geometric and algebraic directions work much better. This result indicates the importance of search directions. Additionally, the algebraic directions works slightly better than the geometric ones. How much does the new directions faster than the others? Figs. 6 and 7 show the performance of mean value as well as standard deviation of iterations and CPU times, respectively. From them, we get some results as follows. • Compared with basic ALS, the algorithm using linear directions saves about 54% ALS iterations and 47% CPU

Fig. 7. Performance of mean CPU times.

times averagely. The result of this experiment is consistent with the fact that a line search method can accelerate ALS. • The algorithms using the geometric and algebraic directions save about 67% ALS iterations and 62% CPU times. This is better than the results of linear directions. When higher-order information is exploited in search directions, the iterations and CPU times of ALS are reduced significantly, although the memory is slightly enlarged. Since the vertical segments denoting standard deviation of the new geometric and algebraic directions are all lower than the average line of the often used linear directions, we say that the new directions are superior to the linear direction in a large possibility. This is concordant with the result in Table II. • The algebraic directions that formed by the convex combination of the linear and quadratic extrapolating directions are more suitable for noise corrupted data than the geometric ones. We believe that the Lagrangian extrapolating technique has a large potential for extracting higher-order information and forming better search directions. C. The Third Result: Nearly Collinear Case When the signals of the users are nearly collinear, which may happen in practice, the convergent rate of ALS is very slow. In the experiment of this subsection, the first user’s signal is random QPSK symbols, and the other users’ signals differ from it in just one bit. The condition number of factor is 10, and tensor is noise free. The typical curve of the cost function values versus iterations is shown by Fig. 8. Obviously, all the algorithms converge to the right decomposition. The new ALS algorithms with extrapolating search directions and optimal step size strategies outperform the existing algorithms. Since some second-order information is used, we observe a faster local convergence rate when iterations approach the solution. VI. CONCLUSION In this paper, we applied the extrapolating technique to construct two new search directions and proposed a novel tech-


Fig. 8. Cost function values versus iterations.

nique to compute the optimal complex step size factor. Simulation results showed that these techniques have successfully and remarkably improved the rate of convergence of ALS. All the techniques could be used to compute the decomposition of complex-valued tensors that follow CP and BCM. For the extraction of the second-order information, three iterates were used to form the search directions, so the new method can be viewed as a multistep method. How to explore the potentiality of multistep method and form effective search directions? This is one of our further works.

ACKNOWLEDGMENT The authors thank the reviewers for their comments and suggestions, which help us improve the presentation of the paper essentially.

REFERENCES [1] E. Acar, D. M. Dunlavy, and T. G. Kolda, “A scalable optimization approach for fitting canonical tensor decompositions,” J. Chemom., vol. 25, no. 2, pp. 67–86, 2011. [2] E. Acar, T. G. Kolda, and D. M. Dunlavy, “An Optimization Approach for Filtting Canonical Tensor Decompositions,” Sandia Report, 2009. [3] E. Acar and B. Yener, “Unsupervised multiway data analysis: A literature survey,” IEEE Trans. Knowl. Data Eng., vol. 21, no. 1, pp. 6–20, 2009. [4] J. D. Carroll and J. Chang, “Analysis of individual differences in multidimensional scaling via an N-way generalization of ‘Eckart-Young’ decomposition,” Psychometrika, vol. 35, no. 3, pp. 283–391, 1970. [5] P. Comon, J. McWhirter and I. Proudler, Eds., “Tensor decompositions,” in Proc. Math. Signal Process. V, Oxford, U.K., 2002, pp. 1–24. [6] P. Comon, X. Luciani, and A. L. F. de Almeida, “Tensor decompositions, alternating least squares and other tales,” J. Chemom., vol. 23, pp. 393–405, 2009. [7] D. Cox, J. Little, and D. O’Shea, Using Algebraic Geometry. New York: Springer-Verlag, 1998. [8] A. L. F. de Almeida, G. Favier, and J. C. M. Mota, “Generalized PARAFAC model for multidimensional wireless communications with application to blind multiuser equalization,” in Proc. 39th ASILOMAR Conf. Signal Syst. Comput., Pacific Grove, CA, 2005.

5897

[9] A. L. F. de Almeida, G. Favier, and J. C. M. Mota, “PARAFAC-based unified tensor modeling for wireless communication systems with application to blind multiuser equalization,” Signal Process., vol. 87, pp. 337–351, 2007. [10] A. L. F. de Almeida, G. Favier, and J. C. M. Mota, “Constrained tensor modeling approach to blind multiple-antenna CDMA schemes,” IEEE Trans. Signal Process., vol. 56, no. 6, pp. 2417–2428, 2008. [11] L. D. Lathauwer, B. D. Moor, and J. Vandewalle, “An introduction to independent component analysis,” J. Chemom., vol. 14, pp. 123–149, 2000. [12] L. D. Lathauwer, “Decompositions of a higher-order tensor in block terms. Part I: Lemmas for partitioned matrices,” SIAM J. Matrix Anal. Appl., vol. 30, no. 3, pp. 1022–1032, 2008. [13] L. D. Lathauwer, “Decompositions of a higher-order tensor in block terms. Part II: Definitions and uniqueness,” SIAM J. Matrix Anal. Appl., vol. 30, no. 3, pp. 1033–1066, 2008. [14] L. D. Lathauwer and D. Nion, “Decompositions of a higher-order tensor in block terms. Part III: Alternating least squares algorithms,” SIAM J. Matrix Anal. Appl., vol. 30, no. 3, pp. 1067–1083, 2008. [15] S. Goedecker, “Remark on algorithms to find roots of polynomials,” SIAM J. Sci. Comput., vol. 15, no. 5, pp. 1059–1063, 1994. [16] G. H. Golub and C. F. Van Loan, Matrix Computations, 3rd ed. Baltimore, MD: The Johns Hopkins Univ. Press, 1996. [17] R. A. Harshman, “Foundations of the PARAFAC procedure: Model and conditions for an ‘explanatory’ multi-mode factor analysis,” in Proc. UCLA Working Papers in Phonet., 1970, vol. 16, pp. 1–84. [18] F. L. Hitchcock, “The expression of a tensor or a polyadic as a sum of products,” J. Math. Phys., vol. 6, pp. 164–189, 1927. [19] F. L. Hitchcock, “Multiple invariants and generalized rank of a p-way matrix or tensor,” J. Math. Phys., vol. 7, pp. 39–79, 1927. [20] T. G. Kolda and B. W. Bader, “Tensor decompositions and applications,” SIAM Rev., vol. 51, no. 3, pp. 455–500, 2009. [21] MathWorks [Online]. Available: http://www.mathworks.com/help/ techdoc/ref/roots.html [22] D. Nion and L. D. Lathauwer, “A block factor analysis based receiver for blind multi-user access in wireless communications,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), Toulouse, France, May 2006, pp. 825–828. [23] D. Nion and L. D. Lathauwer, “Line search computation of the block factor model for blind multi-user access in wireless communications,” in Proc. IEEE Workshop on Signal Process. Adv. Wireless Commun. (SPAWC), Cannes, France, Jul. 2–5, 2006. [24] D. Nion and L. D. Lathauwer, “An enhanced line search scheme for complex-valued tensor decompositions. Application in DS-CDMA,” Signal Process., vol. 88, no. 3, pp. 749–755, 2008. [25] D. Nion and L. D. Lathauwer, “A block component model-based blind DS-CDMA receiver,” IEEE Trans. Signal Process., vol. 56, no. 11, pp. 5567–5579, 2008. [26] L. Qi, W. Sun, and Y. Wang, “Numerical multilinear algebra and its applications,” Front. Math. China, vol. 2, no. 4, pp. 501–526, 2007. [27] N. D. Sidiropoulos, G. B. Giannakis, and R. Bro, “Blind PARAFAC receivers for DS-CDMA systems,” IEEE Trans. Signal Process., vol. 48, pp. 810–823, 2000. [28] A. Smilde, R. Bro, and P. Geladi, Multi-Way Analysis, Applications in the Chemical Sciences. Chichester, U.K.: Wiley, 2004. [29] M. Rajih and P. Comon, “Enhanced line search a novel method to accelerate PARAFAC,” in Proc. Eusipco’05, Antalya, Turkey, 2005. [30] M. Rajih, P. Comon, and R. A. Harshman, “Enhanced line search: A novel method to accelerate PARAFAC,” SIAM J. Matrix Anal. Appl., vol. 30, no. 3, pp. 1148–1171, 2008.

Yannan Chen was born in China in 1981. He received the master degree from Nanjing Normal University, China, in 2007. He is currently pursuing the Ph.D. degree at the School of Mathematical Sciences, Nanjing Normal University. He is a Lecturer in Nanjing Forestry University. His research interests include multilinear algebra, tensor computation, nonlinear numerical optimization, and applications.

5898


Deren Han was born in China in 1974. He received the B.S. and Ph.D. degrees at Nanjing University, China, in 1997 and 2002, respectively. He is now an Associate Professor at School of Mathematical Sciences, Nanjing Normal University. His research interests include numerical optimization, variational inequalities, tensor decomposition, transportation research, and signal and image processing.

Liqun Qi received the B.S. degree at Tsinghua University, Beijing, China, in 1968 and the M.S. and Ph.D. degrees from the University of Wisconsin-Madison in 1981 and 1984, respectively. He has taught at Tsinghua University, the University of Wisconsin-Madison, the University of New South Wales, and the City University of Hong Kong. He is now the Chair Professor of applied mathematics and Head of the Department of Applied Mathematics, The Hong Kong Polytechnic University. He has published more than 150 research papers in international journals. His research interests include multilinear algebra, tensor computation, nonlinear numerical optimization and applications.