A Fixed Point Iterative Method for Low -Rank Tensor Pursuit

2952

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 61, NO. 11, JUNE 1, 2013

A Fixed Point Iterative Method for Low -Rank Tensor Pursuit Lei Yang, Zheng-Hai Huang, and Xianjun Shi

Abstract—The linearly constrained tensor -rank minimization problem is an extension of matrix rank minimization. It is applicable in many fields which use the multi-way data, such as data mining, machine learning and computer vision. In this paper, we adapt operator splitting technique and convex relaxation technique to transform the original problem into a convex, unconstrained optimization problem and propose a fixed point iterative method to solve it. We also prove the convergence of the method under some assumptions. By using a continuation technique, we propose a fast and robust algorithm for solving the tensor completion problem, which is called FP-LRTC (Fixed Point for Low -Rank Tensor Completion). Our numerical results on randomly generated and real tensor completion problems demonstrate that this algorithm is effective, especially for “easy” problems. Index Terms—Fixed point iterative method, low- -rank tensor, tensor completion.

I. INTRODUCTION

H

IGHER order tensors which emerged as a generalization of vectors and matrices make it possible to work on data that has intrinsically many dimensions. This can be seen in various fields such as chemometrics [1], psychometrics [2] and signal processing [3], [4] (see an in-depth survey by Kolda and Bader [5]). Considering the rank as a good depiction of sparsity, we find that many problems in these areas can be transformed into the mathematical task of finding a low rank tensor that can explain the original data well. This class of problems is named as the tensor rank minimization problem. In the context of matrices, matrix rank minimization problems have also been widely applied in various fields including signal processing, control and system. With the application of nuclear norm which is the tightest convex approach to the rank function, one can relax the non-convex NP-hard problem to a tractable, convex one [6]–[9]. Based on this foundation, various effective algorithms have been proposed to solve the matrix rank minimization problem, such as Fixed Point Continuation with Approximate Singular Value Decomposition (FPCA) by Manuscript received June 08, 2012; revised September 24, 2012, January 06, 2013, and March 01, 2013; accepted March 05, 2013. Date of publication March 22, 2013; date of current version May 10, 2013. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Bogdan Dumitrescu. This work was partially supported by the National Natural Science Foundation of China (Grant No. 11171252). L. Yang and X. Shi are with the Department of Mathematics, School of Science, Tianjin University, Tianjin 300072, China (e-mail: [email protected]; [email protected]). Z.-H. Huang is with the Center for Applied Mathematics and the Department of Mathematics, School of Science, Tianjin University, Tianjin 300072, China (e-mail: [email protected]). Color versions of one or more of the figures in this paper are available at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TSP.2013.2254477

Ma, Goldfarb and Chen [10], The Singular Value Thresholding Algorithm (SVT) by Cai, Candès and Shen [11], The Accelerated Proximal Gradient algorithm (APG) by Toh and Yun [12], and so on. Unlike for matrix rank minimization, there are only few studies on tensor rank minimization due to the complexity of the multi-way data analysis. Liu, Musialski, Wonka and Ye [13] opened the door of study on the tensor completion, which is an important special case of the tensor rank minimization problem. In [13], the authors laid the theoretical foundation of the low rank tensor completion. They first introduced the definition of the trace norm for tensors and gave a solution for the low rank tensor completion. Considering the dependency among multiple constraints, they employed a relaxation technique to separate the dependant relationships and adopted the block coordinate descent method to achieve a globally optimal solution. After this pioneering work, convex optimization begins to be used for tensor completion. Recently, Gandy, Recht and Yamada [14] used the -rank of a tensor as a sparsity measure and considered the low- -rank tensor recovery problem, i.e., the problem of finding the tensor of the lowest -rank that fulfills some linear constraints. The model discussed in [14] is more general than the one in [13]. They introduced a tractable convex relaxation of the -rank, proposed efficient algorithms to solve the low- -rank tensor recovery problem, and explicitly derived the convergence guarantees for proposed algorithms. Encouraging numerical results were reported in [14]. Some related work can also be found in Marco Signoretto et al. [15], [16], and Ryota Tomioka et al. [17], [18]. More recently, the exact recovery conditions for tensor -rank minimization via its convex relaxation have also been investigated by Zhang and Huang [19]. In this paper, we also use the -rank of a tensor as a sparsity measure and consider the tensor -rank minimization problem. Using the nuclear norm as an approximation of rank function yields the low -rank tensor pursuit problem. Motivated by the work in [10], we develop a fixed point iterative method for solving the low -rank tensor pursuit problem and show that the proposed method is convergent under some assumptions. Combining with a continuation technique, we also apply the proposed method to solve the relaxation model of the low -rank tensor completion problem, denoted by FP-LRTC. The computational results given in this paper show that FP-LRTC is effective, especially for “easy” problems which will be defined in Section VI.A. The rest of our paper is organized as follows. In Section 2, we briefly introduce some essential notations. Section 3 presents the tensor -rank minimization problem and its relaxation model. In Section 4, a fixed point iterative scheme is derived based on operator splitting technique. We will show the convergence of the

1053-587X/$31.00 © 2013 IEEE

YANG et al.: LOW -RANK TENSOR PURSUIT

2953

iterative scheme and propose the complete fixed point iterative method for low -rank tensor pursuit in Section 5. In Section 6, we introduce a continuation technique and present the results of some numerical tests and comparisons among different algorithms. Some final remarks are given in the last section.

where is the decision variable, and the linear map and vector are given. One of its A special cases is the matrix completion problem:

II. PRELIMINARIES ON TENSORS In this section, we briefly introduce some essential nomenclatures and notations used in this paper; and more details can be found in [5]. Scalars are denoted by lowercase letters, e.g., ; vectors by bold lowercase letters, e.g., ; . An N-way and matrices by uppercase letters, e.g., , whose elements are detensor is denoted as , where and . noted as Especially, a second-order tensor is a matrix and a first-order tensor is a vector. Let us denote the vector space of these ten. sors by , i.e., Matricization, also known as unfolding or flattening, is the process of reordering the elements of an -way array into a to denote the mode- unfolding of a tensor matrix. We use . Specially, the tensor element is mapped to the matrix element , where

where and are both matrices and is a subset of . index pairs The higher-order tensor rank minimization problem can be extended from the matrix (i.e., second-order tensor) case. But unlike matrix, the tensor rank is much more complex. One no[5], which is defined as the smallest number tion is as their sum. An -way of rank- tensors that generate rank- tensor is a tensor that is the outer product of vectors. Unfortunately, this kind of tensor rank is difficult to handle, as there is no straightforward algorithm to determine rank of a specific given tensor. In fact, this problem is NP-hard [20]. The other notion is -rank, which is easy to compute. Therefore, we pay our attention on the -rank in this work and consider the following minimization problems [14]: Problem 3.1: (Tensor -rank minimization) A

That is, the “unfold” operation on a tensor fold with opposite operation “fold” is defined as fold The -rank of an -way tensor of the ranks of the mode- unfoldings:

is defined as un. The . is the tuple

The inner product of two same-sized tensors is the sum of the products of their entries, i.e.,

The corresponding (Frobenius-) norm is . is just the When the tensor reduces to the matrix Frobenius norm of the matrix . denotes the vecFor any tensor ; For torization of , i.e., in Matlab notation, , we can ala linear transformation A , ways write its matrix representation as A where ; For any matrix denotes the (respectively, operator 2-norm of the matrix ; and ) denotes the smallest (respectively, the -th, the largest) eigenvalue of . We use to denote the identity operator and to denote the identity matrix. For any vector , to denote a diagonal matrix with its -th diagwe use Diag onal element being . III. LOW -RANK TENSOR PROBLEM In this section, we will consider the low -rank tensor problem. The derivation starts with the general version of the matrix rank minimization problem [6], [7]: A

(1)

is the decision variable, the linear map where A: with and vector are given. The corresponding tensor completion problem is Problem 3.2: (Low- -rank tensor completion) (2) where are -way tensors with identical size in each in the set are given while the mode, and the entries of remaining entries are missing. The tensor -rank minimization problem (1) (also its special case (2)) is a difficult non-convex problem due to the combi. Therefore, we will renation nature of the function by its convex envelope to get a convex and more place computationally tractable approximation to (1). For this purpose, we use the following notation: for any matrix denotes the nuclear norm of , i.e., the sum of the singular values of . Using it as an approximation to in (1) yields the low -rank tensor pursuit problem. Problem 3.3: (Low -rank tensor pursuit) A

(3)

In fact, the nuclear norm is the best convex approximation of the rank function over the unit ball of matrices with norm less than one [21]. In the following, we consider an unconstrained problem: A

(4)

is a penalty parameter. It is well known that an where optimal solution of (4) approaches an optimal solution of (3) as

2954


(or ) [22]. Thus, one can employ SUMT (Sequential Unconstrained Minimization Technique) to solve (3).

Then we relax the constraint problem:

and get the following

IV. A FIXED POINT ITERATIVE SCHEME In this section we will derive a fixed point iterative scheme. As we all know, the fixed point iterative technique has been proposed since a long time ago. Hale et al. [23] recently proposed a fixed point iterative method for the -regularized problem. Inspired by this work, Ma et al. [10] proposed the fixed point and Bregman iterative method for matrix rank minimization. The commonly fundamental idea of these methods is an operator splitting technique. Motivated by these works, we will develop a fixed point iterative scheme for low -rank tensor pursuit in the following. Firstly, we need to adopt the operator splitting technique. It is well known in convex analysis [24] that the optimal condition is , where of minimizing a convex function at . Thus, is an optimal is the subdifferential of solution to (4) if and only if

(10) where is also a penalty parameter and an optimal solution [22]. of (10) approaches an optimal solution of (9) as Additionally, from the discussion above, an optimal solution of (9) is also an optimal solution of (4) when ; and an optimal solution of (3) can be obtained by solving (4) as . Consequently, when , we can use the optimal solution of (10) to approach an optimal solution of and . Consequently, for convenience, we (3) as and consider the following problem: let

(5) where the map A A

is defined by . Note that (5) is equivalent to

(11) (6)

for any

. If we let

then (6) reduces to

i.e.,

is an optimal solution to

(12) (7)

It’s worth noting that under the definition of modefolding, the optimization in (7) can be written as:

Note that problem (11) is a convex, unconstrained optimization problem and an optimal solution of (11) can approach an and optimal solution of (3) when . Moreover, we can employ block coordinate descent (BCD) method for this problem, which was used in [13] for the tensor completion problem. The detailed implementation is as follows: , we will fixed all variables First, for any . Thus the resulting minimization problem is a matrix except nuclear norm minimization:

un(8)

The problem (8) is difficult to solve due to the interdependent nuclear norms. Therefore, we perform variable splitting and attribute a separate variable to each unfolding of . Let be new matrix variables, which represent the different mode- unfoldings of the tensor , i.e., introduce the new matrix variables such that for all . With these new ’s, we can rephrase (8) by variables

(9)

In the following, we will discuss how to solve (12). Before this, we need some concepts which can be found in [10]. Definition 4.1: (Nonnegative vector shrinkage operator) . For any , the nonnegative vector Suppose that is defined as shrinkage operator

Definition 4.2: (Matrix shrinkage operator) Suppose that and the Singular Value Decomposition (SVD) of is given by , where . For any , the matrix shrinkage operator is defined as

The theorem below provides a theoretical basis for designing our algorithm.


2955

Theorem 4.1: Given and . For any given is an optimal solution to problem (12) if and only if

Or, we can rewrite the above scheme (15) simply by

(16)

Proof: Since the objective function in (12) is convex, is an optimal solution to (12) if and only if

and A A . where In the next section, we will show the above fixed point iterative scheme can help us to get an optimal solution to (3). V. CONVERGENCE RESULTS

(13) is the subgradients of where see that (13) is equivalent to

Then it demonstrates that problem

at

. It is easy to

In this section, we first analyze the convergence properties of the fixed point iterative scheme (15) or (16). Before proving the main convergence results, we need to show the following three lemmas. The first lemma can be found in ([10], Lemma 1). Lemma 5.1: The shrinkage operator is non-expansive, i.e., and , for any

is an optimal solution to the Moreover,

Furthermore, by ([10], Theorem 3), we know that the matrix shrinkage operator applied to gives an optimal solution to the above problem. Hence, is an optimal solution to (12). This completes the proof. In addition, we can get the optimal with all other variables fixed by solving the following subproblem:

if and only if

. Lemma 5.2: For a linear transformation A , let A , where and Suppose that

and

. (17)

Then, the operator if and only if Proof: Since

is non-expansive, i.e., . Moreover, . , (17) implies . Thus, we have

(14) It is easy to check that the optimal solution to (14) is given by

It is worth noting that since the objective in (11) is convex and the first term is separable and the last two terms are differentiable, BCD is guaranteed to find the global optimal solution [25]. Therefore, we have and is an optimal Theorem 4.2: For any fixed solution to problem (11) if and only if

Then based on the discussions above, we can propose a simple fixed point iterative scheme as following:

(15)

where

is the -th eigenvalue of

. Hence,

(18) Obviously, when equalities in (18) become equalities. Thus, if and only if

, all in-

because has to be an eigenvector to the eigenfor the equations to hold with equality. value 1 of Then, we can draw that if . and only if Lemma 5.3: Let A and . Suppose that the operator is defined by (16). Then is non-expansive, i.e., for any ,

2956


Moreover,

if and only if

. Proof: By (16), we have that for any

where

,

, hence for any ,

where . Furthermore, by the definition of given . This in (16), it is easy to show that completes the proof of the second result. Then we show the main convergence result of the fixed point iterative scheme (15) or (16). , assume that the set X Theorem 5.1: For any fixed is nonempty. Let A and . Then, the sequence generated by (15) or (16) has at least an accumulation point and any of is an optimal solution of folaccumulation point lowing problem

Note that for every , by Lemmas 5.1 and 5.2, we have

(20) Proof: For any

, from Lemma 5.3,

X

which means that the sequence is monotonically non-increasing. Then we can easily obtain that

Consequently, the sequence is bounded and must have an accumulation point. Therefore we have

where can be any accumulation point of and , the image of , continuity of

where

is also an accumulation point of

. So, we obtain that

. By the

. Hence, we have

Then, by Lemma 5.3, we have

(19) Thus, we complete the first result of this lemma. Now, we consider the second result. The “ ” part is an immediate consequence. For the “ ” part, holds if and only if all the inequalities in (19) are equalities. Therefore, by using Lemmas 5.1 and 5.2, we obtain

and for

,

which implies since . It then is follows from Theorem 4.2 that the accumulation point an optimal solution to (20). This completes the proof. The following theorem comes from Theorem 9.2.2 in [22]. Theorem 5.2: Consider the following problem:

where and is a nonempty set in feasible solution, and let

are continuous functions on . Suppose that the problem has a


2957

which is a quadratic penalty function. Furthermore, suppose that to the problem to for each there exists a solution subject to , and that is minimize contained in a compact subset of . Then

where . Furthermore, the limit of any convergent subsequence of is an optimal solution to the original problem, and as . Now, based on Theorems 5.1 and 5.2, we can obtain the following important result. is an optimal Theorem 5.3: Suppose that for each solution of the following problem

(21)

Note that in (7) is defined as , then from Theorem 5.2 we can get that is also an . Furthermore, from the discusoptimal solution to (7) as sion about operator splitting, we know that an optimal solution is also to (7) is also an optimal solution to (4). Therefore, . Thus, is an optimal an optimal solution to (4) as solution to (3) by using Theorem 5.2 again. , any accumulation point of sequence Consequently, as is an optimal solution to problem (3). This completes the proof. It should be noted that the objective function of problem (21) comes from the twice penalty method, however, it can be seen from Theorem 5.3 that an optimal solution of problem (3) can . Moreover, be obtained by solving problem (21) as , an optimal solution from Theorem 5.1, for any fixed of problem (21) can be obtained by iterative scheme (15) or (16). Hence, based on the above discussions, we propose the complete fixed point iterative method for solving problem (3) as follows. Algorithm 5.1 A fixed point iterative method for low -rank tensor pursuit Input:

is contained in a compact set. Then as , any and is an optimal solution to accumulation point of sequence problem (3). . To simplify the Proof: Denote proof, we rewrite the objective function in (21) as follows:

Initialization: Set for

and

.

, do

while not converged, do A

A

for where end

end while end for Since the terms in are nonnegative, it can be regarded as is a penalty function. Besides, taking the assumption that contained in a compact set into account, we can use Theorem 5.2 , the limit of any convergent to conclude that when is an optimal solution to the following subsequence of problem:

(22)

It is easy to check that problem (22) is equivalent to the problem as follows:

Output: Remark: In fact, Algorithm 5.1 can be viewed as an exterior penalty function method [22]. We can get an optimal solution to (3) through solving a sequence of problems (21) defined . by a sequence of decreasing penalty parameters This technique is also referred to as SUMT (Sequential Unconstrained Minimization Technique). In addition, the assumption is contained in a compact set is not restrictive in that most practical cases, since the variables usually lie between finite lower and upper bounds. VI. NUMERICAL EXPERIMENTS In this section, we first introduce a continuation technique to accelerate the speed of convergence. The similar continuation technique was also used in [10], [12], [23]. It is different from the traditional SUMT. We will decrease the value of itdetermines the rate of reduction of eratively. The parameter

2958


TABLE I PARAMETERS IN ALGORITHM FP-LRTC

the consecutive , i.e., . And, we will stop the algorithm when is less than some specified value . On the iterations to solve other hand, for each fixed , we only use (21) to get an approximate solution for the current problem. And is to be solved, the when a new problem associated with is used as approximate solution for the problem with the starting point. This technique is mainly used to reduce the computational cost. Our numerical experience indicates that it is really effective for our algorithm. Then, Algorithm 5.1 can be written as below: Algorithm 6.1: A fixed point continuation method for low -rank tensor pursuit

the Lanczos algorithm, which is also used in [14], [17] to largest singular values compute only the predetermined and corresponding singular vectors to speed up the calculation [26], [27]. In our experiments, we use the procedure in [10] to . In the th-iteration, when computing the SVD of set

we set equal to the number of components in that are , where is a small positive number not less than is the largest component in the vector and used to form . Note that is non-increasing in this procedure. On the other hand, if is too small at some iteration, the shrinkage property (19) may be by 1. Our nuviolated. Thus, if (19) is violated, we increase merical experience indicates that this technique is very effective for our algorithm. A. Numerical Simulation

Input: Initialization: Set

and

for

, do repeat

.

iterations A

A

In this part, we randomly create a tensor with rank by the Tucker decomposition [5], with i.i.d. [28]: we first generate a core tensor . Then, we generate matrixes Gaussian entries , with whose entries are i.i.d. and set from

for

end end repeat end for Output: In the following, we apply Algorithm 6.1 to solve the low -rank tensor completion problem and denote it as FP-LRTC. We will evaluate the empirical performance of FP-LRTC both on simulated and real world data with the missing data. We also compare the results with the latest low -rank tensor completion algorithms, including TENSOR-HC (hard completion) [16], ADM-CON (ADMM for the “Constraint” approach) [17], DR-TR (the Douglas-Rachford splitting for tensor recovery) [14], ADM-TR(E) (alternative direction method algorithm for low-n-rank tensor recovery) [14] and LRTC (low rank tensor completion) [13]. All numerical experiments were run in Matlab 7.11.0 on a HP Z800 workstation with an Intel Xeon(R) 3.33 GHz CPU and 48 GB of RAM. Remark: Computing singular value decomposition is the , main computational cost in FP-LRTC. For a matrix if is much larger than , one can compute the SVD of and then get an approximate SVD of by simple operations to reduce the computational cost. This method was also used in the code of LRTC. But the singular values and corresponding singular vectors may be not accurate enough in real calculation. Consequently, we only use this method in FP-LRTC for “easy” problems (will be defined later). For “hard” problems, we use

With this construction, the -rank of equals alto denote the sampling ratio, i.e., a permost surely. We use of the entries to be known and choose the support centage of the known entries uniformly at random among all supports of . The values and the locations of the known size entries of are used as input for the algorithms. The relative error

is used to estimate the closeness of to , where is the “optimal” solution to (4) produced by the algorithms. The parameter settings used for FP-LRTC are summarized in Table I. by We define the parameter

where and . In fact, it can between the debe regarded as an extension of the ratio gree of freedom in a rank matrix to the number of samples in [11], in the matrix completion [10], [29] (or see in [12]). This ratio is used to show the hardness of the matrix completion problem. For the low -rank tensor completion, we may want to recover each mode- unfolding of the tensor . So we use the weighted average of all of to show the hardness of the low -rank tensor completion . Additionally, for , we say it problem denoted by


Fig. 1. Recovery results with different by FP-LRTC. Top: Relative Error; Bottom: CPU time in seconds. All results are average values of 10 independent trials.

is an “easy” problem because ’s are of very low rank compared to the size and the number of samples. Otherwise, it is a involved ’s that are not of very “hard” problem low rank and for which sampled a very limited number of enbecomes larger, it will be harder to solve tries. Hence when to guarantee that we can the problem. Then we need a larger get an approximate solution for the current problem with each fixed and the sequence generated by FP-LRTC can converge is significant for the to the real solution. Thus, the choice of speed and accuracy of FP-LRTC. In Fig. 1, we numerically compare the recovery results with by testing FP-LRTC on six problem setdifferent values of tings and identify a good implementation of setting for the remaining tests. From the results, we can see that the larger becomes, the more time it costs to recover a tensor. But if is too small, the tensor will be recovered unsuccessfully at some , it needs smaller times. On the other hand, for smaller to guarantee recovering a tensor successfully. For the problem

2959

Fig. 2. Recovery results of 60 60 60 tensors with different by FP-LRTC, TENSOR-HC, ADM-CON, ADM-TR(E) and DR-TR. Top: Relative Error; Bottom: CPU time in seconds. All results are average values of 10 independent trials.

setting: , we to get an optimal solution. However, for the only need problem setting: , we should need at least to recover a tensor. Therefore, in order to balance the speed and accuracy, we will : use the following strategy to set

The LRTC code is downloaded from http://peterwonka.net/ Publications/publications.html. In LRTC, we set the parameto 1 and set the maximum iteration number to 2000. ters The ADM-CON code is downloaded from http://www.ibis.t.utokyo.ac.jp/RyotaTomioka/Softwares/Tensor.html. The parameter is set to 0 in ADM-CON. In ADM-TR(E) and DR-TR,

2960


TABLE II COMPARISON OF DIFFERENT ALGORITHMS FOR TENSOR COMPLETION

the parameters were set to (see [14] for more detials). In TENSOR-HC, we set the regularizato 1 and to 10. tion parameters Different problem settings are used to test the algorithms. Four of these come from [14]. The order of the tensors varies from three to five, and we also vary the -rank and the sam. Table II presents these different settings and pling ratio the recovery performance for different algorithms. For every problem setting, we solve 10 randomly created tensor completo denote the number of tensors tion problems. We use that are recovered successfully, which means the relative error . and stands for the is less than average iterations, the average relative error and the average

time (seconds) for the examples that are successfully solved, respectively. Considering the situation of multiple circulations of FP-LRTC, we only report the total iterations. All these experiments can be considered to vary from small-sized problems to large-sized problems. From the results in Table II, we can easily see that it costs less time with lower rank and higher . The ’s in these situations are low, which sampling ratio can indicate that these are “easy” recovery problems. Additionally, with the same sampling rates and rank , the number of entries of a tensor positively correlates with the time it costs. By comparing the results of different methods, it is noticeable that LRTC and DR-TR are always poorer than other algorithms in both relative error and CPU time. FP-LRTC is robust and con-


2961

Fig. 3. (a) One slice of the original tensor from the KNIX data set [30], (b) input to the algorithm (50% known entries), (c) The recovered result by FP-LRTC .

Fig. 4. (a) One slice of the original tensor from the INCISIX data set [30], (b) input to the algorithm (60% known entries), (c) The recovered result by . FP-LRTC

verges faster than other algorithms except TENSOR-HC in most cases. And it also can be comparable with TENSOR-HC, which is the best recovery algorithm in our knowledge. Though for the , FP-LRTC performs poorer than “hard” problem TENSOR-HC and ADM-CON. For the problems , FP-LRTC results nearly the same computation time and relative errors as those of TENSOR-HC. Especially, for some , we can see that FP-LRTC can save much problems more time to recover a tensor. Then we fix the tensor size and compare the first five different algorithms (LRTC is poorer than other algorithms obviously by Table II) with different ranks . Fig. 2 depict the average results of 10 independent trials corresponding to different ranks for randomly created small-sized tensor completion problems. The sampling ratios is set to 0.6. From Fig. 2, we can see that FP-LRTC is usually faster and more robust than others and provides good solutions efficiently. In fact, for this set of problems, ranges from 0.05 to 0.42. Hence, it shows that FP-LRTC is more effective to solve “easy” problems. On the other hand, the relative error of FP-LRTC is more stable than that of others, esbecomes larger. pecially when B. Image Simulation In this part, we apply FP-LRTC on medical data. In these experiments, we use the same data as [14] from the OsiriX repository [30], which is available online. Fig. 3 shows a recovery experiment using the KNIX data set which contains 22 images of size 256 256 (50% of the entries are known). In Fig. 4, we use the INCISIX data set consisting of 166 images of size 256 256 (60% of the entries are known). Furthermore, to graphically illustrate the effectiveness of FP-LRTC, we applied it to image inpainting [31]. Color images can be expressed as third-order tensors. If the image is of low rank, or numerical low-rank, we can solve the image inpainting problem as a tensor completion problem (2). In our test, for each

Fig. 5. Original images (left column); Input to the algorithm (middle column); The recovered result by FP-LRTC (right column).

image we remove entries in all the channels simultaneously (first two rows of Fig. 5), or consider the case where entries are missing at random (last row of Fig. 5, 10% known entries). Fig. 5 reports the original pictures, the input data tensor and the outcome of our algorithm. VII. CONCLUSION In this paper we focused on tensor -rank minimization problem and adopted operator splitting technique and convex relaxation technique to transform it into a tractable optimization problem. A fixed point iterative method was proposed to tackle this problem. This method is easy to implement and very effective. Several application examples show the applicability of tensor completion in image processing. Recently, an acceleration method [12], [32], [33] was used in compressed sensing and complete problem, for which the authors obtained not only rate of convergence statement. convergence but also It is interesting to investigate the rate of convergence of the fixed point iterative method. In addition, we believe that the fixed point continuation framework should also yield robust and efficient methods for more general tensor optimization problems, which is also a topic of further research. ACKNOWLEDGMENT The authors are very grateful to the editor and the four anonymous referees for their valuable suggestions and comments, which have considerably improve the presentation of this paper. They also would like to thank Silvia Gandy for sending us the codes of ADM-TR(E) and DR-TR, and Marco Signoretto for sending them the code of TENSOR-HC.

2962

REFERENCES [1] A. Smilde, R. Bro, and P. Geladi, Multi-Way Analysis: Applications in the Chemical Sciences. New York, NY, USA: Wiley, 2004. [2] P. Kroonenberg, Three-Mode Principal Component Analysis: Theory and Applications. Leiden, The Netherlands: DSWO , 1983. [3] P. Comon, Tensor Decompositions: State of the Art and Applications. Mathematics in Signal Processing V, J. McWhirter and I. Proudler, Eds. Oxford , U.K.: Oxford Univ. Press, 2001, pp. 1–24. [4] L. D. Lathauwer and B. D. Moor, From Matrix to Tensor: Multilinear Algebra and Signal Process. Math. Signal Process. IV, J. McWhirter and I. Proudler, Eds. Oxford, U.K.: Clarendon, 1998, pp. 1–15. [5] T. G. Kolda and B. W. Bader, “Tensor decompositions and applications,” SIAM Rev., vol. 51, pp. 457–464, 2009. [6] M. Fazel, H. Hindi, and S. Boyd, “A rank minimization heuristic with application to minimum order system approximation,” in Proc. Amer. Control Conf., Arlington, VA, USA, Jun. 2001, vol. 6, pp. 4734–4739, 2001. [7] B. Recht, M. Fazel, and P. A. Parrilo, “Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization,” SIAM Rev., vol. 52, pp. 471–501, 2010. [8] E. J. Candès and B. Recht, “Exact matrix completion via convex optimization,” Found. Comput. Math., vol. 9, pp. 717–772, 2009. [9] B. Recht, “A simpler approach to matrix completion,” J. Mach. Learn. Res., vol. 12, pp. 3413–3430, 2011. [10] S. Q. Ma, D. Goldfarb, and L. F. Chen, “Fixed point and Bregman iterative methods for matrix rank minimization,” Math. Program., vol. 128, pp. 321–353, 2011. [11] J. F. Cai, E. J. Candès, and Z. W. Shen, “A singular value thresholding algorithm for matrix completion,” SIAM J. Optim., vol. 20, no. 4, pp. 1956–1982, 2010. [12] K. C. Toh and S. W. Yun, “An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems,” Pac. J. Optim., vol. 6, pp. 615–640, 2010. [13] J. Liu, P. Musialski, P. Wonka, and J. P. Ye, “Tensor completion for estimating missing values in visual data,” in Proc. IEEE Int. Conf. Comput. Vision (ICCV), Kyoto, Japan, 2009, pp. 2114–2121. [14] S. Gandy, B. Recht, and I. Yamada, “Tensor completion and low-nrank tensor recovery via convex optimization,” Inv. Probl., vol. 27, p. 025010, 2011. [15] M. Signoretto, L. De. Lathauwer, and J. A. K. Suykens, “Nuclear norms for tensors and their use for convex multilinear estimation,” K. U. Leuven, Leuven, Belgium, Internal Rep. 10-186, ESAT-SISTA, 2010, Lirias number: 270741. [16] M. Signoretto, Q. T. Dinh, L. De Lathauwer, and J. A. K. Suykens, “Learning with tensors: A framework based on convex optimization and spectral regularization,” K. U. Leuven, Leuven, Belgium, Internal Rep. 11-129, ESAT-SISTA, 2011. [17] R. Tomioka, K. Hayashi, and H. Kashima, “Estimation of low-rank tensors via convex optimization,” 2011 [Online]. Available: http://arxiv. org/abs/1010.0789, Arxiv preprint arXiv:1010.0789v2 [18] R. Tomioka, T. Suzuki, K. Hayashi, and H. Kashima, “Statistical performance of convex tensor decomposition,” presented at the Adv. Neural Inf. Process. Syst. (NIPS) 24, Granada, Spain, 2011. [19] M. Zhang and Z. H. Huang, “Exact recovery conditions for the low-nrank tensor recovery problem via its convex relaxation,” Dept. Math., School of Sci., Tianjin Univ., Tianjin, China, Tech. Rep., 2012. [20] J. Håstad, “Tensor rank is NP-complete,” J. Algorithms, vol. 11, pp. 644–654, 1990. [21] M. Fazel, “Matrix rank minimization with applications,” Ph.D. thesis, Stanford Univ., Stanford, CA, USA, 2002. [22] M. S. Bazaraa, H. D. Sherali, and C. M. Shetty, Nonlinear Programming, 3rd ed. New York, NY, USA: Wiley, 2006. [23] E. T. Hale, W. T. Yin, and Y. Zhang, “A fixed-point continuation method for -regularized minimization with applications to compressed sensing,” SIAM J. Optim., vol. 19, pp. 1107–1130, 2008. [24] R. T. Rockafellar, Convex Analysis. Princeton, NJ, USA: Princeton Univ. Press, 1970. [25] P. Tseng, “Convergence of block coordinate descent method for nondifferentiable minimization,” J. Optim. Theory Appl., vol. 109, pp. 475–494, 2001.


[26] M. W. Berry, “Large-scale sparse singular value decompositions,” Int. J. Supercomp. Appl., vol. 6, pp. 13–49, 1992. [27] R. M. Larsen [Online]. Available: http://soi.stanford.edu/~rmunk/ PROPACK/, software for large and sparse SVD calculations. [28] L. R. Tucker, “Some mathematical notes on three-mode factor analysis,” Psychomet., vol. 31, pp. 279–311, 1966. [29] Z. Wen, W. Yin, and Y. Zhang, “Solving a low-rank factorization model for matrix completion by a nonlinear successive over-relaxation algorithm,” Rice Univ., Houston, TX, USA, CAAM Tech. Rep. TR10-07, 2010. [30] OsiriX, DICOM sample image sets repository [Online]. Available: http://www.osirix-viewer.com/datasets/ [31] M. Bertalmío, G. Sapiro, V. Caselles, and C. Ballester, “Image inpainting,” presented at the SIGGRAPH 2000, New Orleans, LA, USA, 2000. [32] Y. Nesterov, “Gradient methods for minimizing composite objective function,” Univ. Cathol. Louvain, Center for Operations Research and Econometrics (CORE), Tech. Rep., 2007. [33] A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems,” SIAM J. Imaging Sci., vol. 2, pp. 183–202, 2009 [Online]. Available: http://epubs.siam.org/journal/ sjisbi

Lei Yang received the Bachelor degree in applied mathematics from Tianjin University and the Bachelor degree in finance from Nankai University in 2011. He is currently a graduate student with the Department of Mathematics, Tianjin University. His research interests include optimization, tensor spectral theory, compressed sensing, and matrix/tensor rank minimization. He is also interested in option pricing and portfolio.

Zheng-Hai Huang received the B.S. degree from Central China Normal University, China, in 1988 and the M.S. degree from Huazhong University of Science and Technology, China, in 1996. In 1999, he received the Ph.D. degree from Fudan University, China. From 2002 to 2004, he was a Research Fellow with the Department of Mathematics, National University of Singapore. In 2004, he joined Tianjin University as a Professor of Mathematics. He has published more than 80 research papers in the areas of optimization theory and methods. Recently, his research interests include compressed sensing, low-rank matrix minimization, and low-rank tensor minimization etc.

Xianjun Shi received the Bachelor degrees in applied mathematics from Tianjin University, P.R. China, in 2010. He is doing a Master programme of operational research and cybernetics in Tianjin University. His main research focus is on mathematical finance including option pricing, portfolio, and value at risk. He is also interested in tensor completion and unconstrained optimization problems.