The Singular Value Decomposition (SVD) in Tensors - CiteSeerX

Proceedings of the 6th WSEAS Int. Conf. on Systems Theory & Scientific Computation, Elounda, Greece, August 21-23, 2006 (pp7-13)

The Singular Value Decomposition (SVD) in Tensors (Multidimensional Arrays) as an Optimization Problem. Solution via Genetic Algorithms and method of Nelder-Mead. Nikos E. Mastorakis (*) WSEAS (Research and Development Department) http://www.worldses.org/research/index.html Agiou Ioannou Theologou 17-23 15773, Zografou, Athens, GREECE, www.wseas.org Abstract: - In this paper, an attempt to solve the Singular Value Decomposition problem in Tensors (multidimensional arrays) using a computational scheme of Evolutionary Computing (Genetic Algorithms) accompanied by the Nelder-Mead method is proposed. Key-Words: - Singular Value Decomposition, Genetic Algorithms, Evolutionary Computing, Nelder-Mead.

1 Introduction The Singular Value Decomposition (SVD) technique is well-known in applied mathematicians, engineers and scientists working on applied linear algebra, systems and control theory, signal processing, image processing (especially: face recognition), data mining and statistics, psychometrics and chemometrics. It can be seen as a factorization technique of a rectangular real or complex matrix and can be considered as a generalization of the spectral theorem, which says that normal matrices can be diagonalized using a basis of eigenvectors, to arbitrary, not necessarily square, matrices. The importance of the SVD is due on the great economy on data storage, data submission, image compression, data compression etc…The singular value decomposition was originally developed by differential geometers, who wished to determine whether a real bilinear forms could be made equal to another by independent orthogonal transformations of the two spaces it acts on. Eugenio Beltrami and Camille Jordan discovered independently, in 1873 and 1874 respectively, that the singular values of the bilinear forms, represented as a matrix, form a complete set of invariants for bilinear forms under orthogonal substitutions. James Joseph Sylvester also arrived at the singular value decomposition for real square matrices in 1889, apparently independent of both Beltrami and Jordan. Sylvester called the singular values the canonical multipliers of the matrix A. (*)

also with: Military Institutes of University Education (ASEI), Hellenic Naval Academy, Terma Hatzikyriakou 18539, Piraeus, GREECE

The fourth mathematician to discover the singular value decomposition independently is Autonne in 1915, who arrived at it via the polar decomposition. The first proof of the singular value decomposition for rectangular and complex matrices seems to be by Eckart and Young in 1936; they saw it as a generalization of the principal axis transformation for Hermitian matrices. In 1907, Erhard Schmidt defined an analog of singular values for integral operators (which are compact, under some weak technical assumptions); it seems he was unaware of the parallel work on singular values of finite matrices. This theory was further developed by Émile Picard in 1910, who is the first to call the numbers σk singular values (or rather, valeurs singulières) [21] The SVD of an m-by-n matrix is

B = σ 1u1v1T + σ 2 u 2 v 2T + ... + σ r u r v rT where

for

the

so

called

singular

(1) values

σ 1 , σ 2 ,..., σ r we assume σ 1 ≥ σ 2 ≥ ... ≥ σ r > 0 and σ i =

λi where λi are the eigenvalues of of

the B T B (or equivalently BB T , T denotes the transpose matrix). The vectors u1 , v1 , u 2 , v 2 ,..., u r , v r are certain orthonormal vectors and r = rank ( B ) ≤ min(m, n) A more detailed description of the method can be found in several standard books [1÷4]. In the standard tensor notation [5], Eq.(1) is written as:

Proceedings of the 6th WSEAS Int. Conf. on Systems Theory & Scientific Computation, Elounda, Greece, August 21-23, 2006 (pp7-13)

B = σ 1u1 ⊗ v1 + σ 2 u 2 ⊗ v 2 + ... + σ r u r ⊗ v r

(2)

( u1(1) , u 2(1) ,..., u r(1) ∈ ℜ n , 1

u1( 2) , u 2( 2) ,..., u r( 2) ∈ ℜ n2 , ….,

where ⊗ denotes the usual standard tensor product.

u1( K ) , u 2( K ) ,..., u r( K ) ∈ ℜ nK )

Some of the most important applications of the SVD is the data compression, the economy in the data storage, the noise reductions and cancellation, the approximate solution of many problems when they formulated in matrix notation, the least squares solution (pseudoinverse), the approximate multidimensional (multivariable) polynomial factorization [26] are based on the approximate writing of (2) as follows

then it has been shown, that the vectors: (1) (1) (1) (2) ( 2) ( 2) and u1 , u 2 ,..., u r u1 , u 2 ,..., u r (K ) (K ) (K ) do not necessarily form u1 , u 2 ,..., u r orthonormal sets. (We simply consider them as normal vectors i.e. with norm 1). Therefore, no orthogonality constraints on the vectors and u1(1) , u 2(1) ,..., u r(1) , u1( 2) , u 2( 2) ,..., u r( 2) (K ) (K ) (K ) u1 , u 2 ,..., u r are imposed in the decomposition (4). A decomposition of the form (4), was independently proposed in [8] and [9]. As discussed in [7], this decomposition is very useful in applications even without orthogonality of the vectors. In some applications, in fact, orthogonality is not even desirable. However, in other applications, the orthogonality is very important and must be examined. The maximum possible rank of a tensor is not given directly by the dimensions of the tensor, again in contrast to the matrix case. Indeed, the maximum possible rank of a matrix is the minimum of the dimensions,

B ≈ σ 1u1 ⊗ v1 + σ 2 u 2 ⊗ v 2 + ... + σ q u q ⊗ v q (3) Eq.(3) holds in the case that we decide to omit the q + 1, q + 2,..., r terms of the SVD, Eq.(2), in the case where

σ 1 ≥ σ 2 ≥ ... ≥ σ q >> σ q +1 The main advantage of Eq.(3) that leads to the aforementioned applications is that instead of n ⋅ m memory positions for the matrix B, we need only

q ⋅ (n − 1 + m − 1 + 1) = q ⋅ (n + m − 1) since every orthonormal n-vector requires n-1 positions. Unfortunately, in a tensor (m-D array) the method of the SVD cannot be applied directly as it has been pointed out [6]. Many scientific and engineering problems related with a variety of applications in real life demand analogous decomposition of a tensor. The author in [6] gave a solution by unifying dimensions into one and transforming the K-D tensor (K>2) into a 2-D array (matrix), applying SVD to this matrix and finally reconstructing the tensor by the decomposed matrix. The method of [6] has also found applications in many studies of other researchers, though up to now, we cannot have a direct SVD for a tensor in the case of K>2. So, for a tensor with K>2, with dimensions n1 , n 2 ,..., n K if we try to find a direct decomposition like

B = σ 1u1(1) ⊗ u1( 2) ⊗ ... ⊗ u1( K ) + .. + σ r u r(1) ⊗ u r( 2) ⊗ ... ⊗ u r( K ) (4)

r = rank ( B ) ≤ min(m, n) , but for a tensor n1 × n2 × ... × n K we do not have that r ≤ min(n1 , n2 ,..., n K ) In fact, the maximum possible r of an arbitrary tensor is unknown, although (usually loose) upper bounds, dependent on the dimensions of the tensor, do exist. In this paper, a decomposition of a tensor B is proposed using Genetic Algorithms and NelderMead.

2 Problem formulation and Solution Before proceeding in the solution of the problem, some background on GA (Genetic Algorithms) and Nelder-Mead is necessary. In [15], the author has also proposed a hybrid method that includes a) Genetic Algorithm for finding rather the neiborhood of the global minimum than the global minimu itself and b) Nelder-Mead algorithm to find the exact point of the global minimum itself. So, with this Hybrid method of Genetic Algorithm + Nelder-Mead we combine the advantages of both methods, that are a) the convergence to the global minimum (genetic

Proceedings of the 6th WSEAS Int. Conf. on Systems Theory & Scientific Computation, Elounda, Greece, August 21-23, 2006 (pp7-13)

algorithm) plus b) the high accuracy of the NelderMead method. If we use only a Genetic Algorithm then we have the problem of low accuracy. If we use only NelderMead, then we have the problem of the possible convergence to a local (not to the global) minimum. These disadvantages are removed in the case of our Hybrid method that combines Genetic Algorithm with Nelder-Mead method. See [5]. We recall the following definitions from the Genetic Algorithms literature: Fitness function is the objective function we want to minimize. Population size specifies how many individuals there are in each generation. We can use various Fitness Scaling Options (rank, proportional, top, shift linear, etc…[16]), as well as various Selection Options (like Stochastic uniform, Remainder, Uniform, Roulette, Tournament)[16]. Fitness Scaling Options: We can use scaling functions. A Scaling function specifies the function that performs the scaling. A scaling function converts raw fitness scores returned by the fitness function to values in a range that is suitable for the selection function. We have the following options: Rank Scaling Option: scales the raw scores based on the rank of each individual, rather than its score. The rank of an individual is its position in the sorted scores. The rank of the fittest individual is 1, the next fittest is 2 and so on. Rank fitness scaling removes the effect of the spread of the raw scores. Proportional Scaling Option: The Proportional Scaling makes the expectation proportional to the raw fitness score. This strategy has weaknesses when raw scores are not in a "good" range. Top Scaling Option: The Top Scaling scales the individuals with the highest fitness values equally. Shift linear Scaling Option: The shift linear scaling option scales the raw scores so that the expectation of the fittest individual is equal to a constant, which you can specify as Maximum survival rate, multiplied by the average score. We can have also option in our Reproduction in order to determine how the genetic algorithm creates children at each new generation. For example, Elite Counter specifies the number of individuals that are guaranteed to survive to the next generation. Crossover combines two individuals, or parents, to form a new individual, or child, for the next generation. Crossover fraction specifies the fraction of the next

generation, other than elite individuals, that are produced by crossover. Scattered Crossover: Scattered Crossover creates a random binary vector. It then selects the genes where the vector is a 1 from the first parent, and the genes where the vector is a 0 from the second parent, and combines the genes to form the child. Mutation: Mutation makes small random changes in the individuals in the population, which provide genetic diversity and enable the GA to search a broader space. Gaussian Mutation: We call that the Mutation is Gaussian if the Mutation adds a random number to each vector entry of an individual. This random number is taken from a Gaussian distribution centered on zero. The variance of this distribution can be controlled with two parameters. The Scale parameter determines the variance at the first generation. The Shrink parameter controls how variance shrinks as generations go by. If the Shrink parameter is 0, the variance is constant. If the Shrink parameter is 1, the variance shrinks to 0 linearly as the last generation is reached.

Migration is the movement of individuals between subpopulations (the best individuals from one subpopulation replace the worst individuals in another subpopulation). We can control how migration occurs by the following three parameters. Direction of Migration: Migration can take place in one direction or two. In the so-called “Forward migration” the nth subpopulation migrates into the (n+1)'th subpopulation. while in the so-called “Both directions Migration”, the nth subpopulation migrates into both the (n-1)th and the (n+1)th subpopulation. Migration wraps at the ends of the subpopulations. That is, the last subpopulation migrates into the first, and the first may migrate into the last. To prevent wrapping, specify a subpopulation of size zero.

Fraction of Migration is the number of the individuals that we move between the subpopulations. So, Fraction of Migration is the fraction of the smaller of the two subpopulations that moves. If individuals migrate from a subpopulation of 50 individuals into a population of 100 individuals and Fraction is 0.1, 5 individuals (0.1 * 50) migrate. Individuals that migrate from one subpopulation to another are copied. They are not removed from the source subpopulation. Interval of Migration counts how many

Proceedings of the 6th WSEAS Int. Conf. on Systems Theory & Scientific Computation, Elounda, Greece, August 21-23, 2006 (pp7-13)

generations

pass

between

migrations.

The Nelder-Mead simplex algorithm appeared in 1965 and is now one of the most widely used methods for nonlinear unconstrained optimization [22]÷[25]. The Nelder-Mead method attempts to minimize a scalar-valued nonlinear function of n real variables using only function values, without any derivative information (explicit or implicit). The Nelder-Mead method thus falls in the general class of direct search methods. The method is described as follows: Let f(x) be the function for minimization. x is a vector in n real variables. We select n+1 initial points for x and we follow the steps: Step 1. Order. Order the n+1 vertices to satisfy f(x1) ≤ f(x2) ≤ … ≤ f(xn+1), using the tie-breaking rules given below. Step 2. Reflect. Compute the reflection point xr from

xr = x + ρ ( x − x n+1 ) = (1 + ρ ) x − ρxn+1 ,

Step 5. Perform a shrink step. Evaluate f at the n points vi = x1 + σ (xi – x1), i = 2, … , n+1. The (unordered) vertices of the simplex at the next iteration consist of x1, v2, … , vn+1. After this preparation, we are ready to present the decomposition of a tensor B by reducing it to minimization problem as follows: For the tensor B with dimensions n1 , n 2 ,..., n K (K>2) we try to find a direct decomposition B = σ 1u1(1) ⊗ u1( 2) ⊗ ... ⊗ u1( K ) + .. + σ r u r(1) ⊗ u r( 2) ⊗ ... ⊗ u r( K ) following the next steps:

Step 1: Let B1 = B

We demand min

σ 1 ,u1(1) ,u1( 2 ) ,...,u1( K )

B1 − σ 1u1(1) ⊗ u1( 2 ) ⊗ ... ⊗ u1( K )

under the constraints: n

where x =

∑x /n i

is the centroid of the n best

u1(1) = 1 , u1( 2 ) = 1 , … , u1( K ) = 1

i =1

points (all vertices except for xn+1). Evaluate fr=f(xr). If f1 ≤ fr < fn , accept the reflected point xr and terminate the iteration. Step 3. Expand. If fr < f1 , calculate the expansion point xe, x e = x + χ ( x r − x ) = x + ρχ ( x − x n +1 ) = (1 + ρχ ) x − ρχx n +1

and evaluate fe=f(xe). If fe < fr, accept xe and terminate the iteration; otherwise (if fe ≥ fr), accept xr and terminate the iteration. Step 4. Contract. If fr ≥ fn, perform a contraction between x and the better of xn+1 and xr. a. Outside. If fn ≤ fr < fn+1 (i.e. xr is strictly better than xn+1), perform an outside contraction: calculate x c = x + γ ( x r − x ) = x + γρ ( x − x n +1 ) = (1 + ργ ) x − ργx n +1

and evaluate fc = f(xc). If fc ≤ fr, accept xc and terminate the iteration; otherwise, go to step 5 (perform a shrink). b. Inside. If fr ≥ fn+1, perform an inside contraction: calculate x cc = x − γ ( x − x n +1 ) = (1 − γ ) x + γx n +1 ,

and evaluate fcc = f(xcc). If fcc < fn+1, accept xcc and terminate the iteration; otherwise, go to step 5 (perform a shrink).

Step 2: Let B2 = B1 − u1(1) ⊗ u1( 2 ) ⊗ ... ⊗ x1( K ) .

We demand min ( K ) B2 − σ 2 u 2(1) ⊗ u 2( 2 ) ⊗ ... ⊗ u 2( K ) (1 ) ( 2 ) σ 2 ,u 2 ,u 2 ,...,u 2

under the constraints:

u2(1) = 1 , u2( 2) = 1 , … , u 2( K ) = 1

… Step q: Let Bq = Bq−1 − u q(1−)1 ⊗ uq( 2−1) ⊗ ... ⊗ uq( K−1) .

We demand min ( K ) Bq − σ q uq(1) ⊗ uq( 2) ⊗ ... ⊗ uq( K ) (1 ) ( 2 ) σ q ,uq ,uq ,...,uq

under the constraints:

uq(1) = 1 , uq( 2) = 1 , … , u q( K ) = 1

Termination Criterion

We terminate the algorithm in the step q+1 when σ q +1 ≈ 0 (for example σ q +1 < ε say

ε = 10 −4 )

or

if

q = n1 + n2 + ... + n K

Proceedings of the 6th WSEAS Int. Conf. on Systems Theory & Scientific Computation, Elounda, Greece, August 21-23, 2006 (pp7-13)

In every minimization step, the minimization is achieved by using Genetic Algorithms (GA) and the method of Nelder-Mead exactly as we described previously. We can use the MATLAB software package, [25]. Our GA has the following Parameters Population type: Double Vector Population size: 30 Creation function: Uniform Fitness scaling: Rank Selection function: roulette Reproduction: 6 – Crossover fraction 0.8 Mutation: Gaussian – Scale 1.0, Shrink 1.0 Crossover: Scattered Migration: Both – fraction 0.2, interval: 20 Stopping criteria: 100 generation We prove now the following Theorem: Theorem: All the singular values of the proposed tensor SVD are positive. That means: σ 1 ≥ 0 , σ 2 ≥ 0,...., σ q ≥ 0

6.6

-2

4.1

2.13

9

-3

6.5

2.9

11.6

-4

7.8

3.8

11.4

-3.7

8.2

4

19

-6.1

12.5

5.6

23

-8

15.5

8.2

Fig.1: Tensor of the Numerical Example

B1,1,1 = 6.6, B1,1, 2 = −2, B1,1,3 = 4.1, B1,1, 4 = 2.13

Proof: We have that B1 =

B1, 2,1 = 9, B1, 2, 2 = −3, B1, 2,3 = 6.5, B1, 2, 4 = 2.9

B1 − σ 1u1(1) ⊗ u1( 2 ) ⊗ ... ⊗ u1( K ) + σ 1u1(1) ⊗ u1( 2) ⊗ ... ⊗ u1( K ) ≤ B1 − σ 1u1(1) ⊗ u1( 2) ⊗ ... ⊗ u1( K ) + σ 1u1(1) ⊗ u1( 2 ) ⊗ ... ⊗ u1( K )

Hence:

B1,3,1 = 11.6, B1,3, 2 = −4, B1,3,3 = 7.8, B1,3, 4 = 3.8 B2,1,1 = 11.4 B2,1, 2 = −3.7, B2,1,3 = 8.2, B2,1, 4 = 4 B2, 2,1 = 19, B2, 2, 2 = −6.1, B2, 2,3 = 12.5, B2, 2, 4 = 5.6 B2,3,1 = 23, B2,3, 2 = −8, B2,3,3 = 15.5, B2,3, 4 = 8.2 Let B1 = B

B1 − B1 − σ 1u1(1) ⊗ u1( 2 ) ⊗ ... ⊗ u1( K ) ≤ σ 1 ⋅ 1 ⋅ 1...1

(5)

min

σ1 ,u1(1) ,u1( 2 ) ,u1( 3 )

B1 − σ 1u1(1) ⊗ u1( 2) ⊗ u1( 3)

under the constraints: (1) 1

(since u

( 2) 1

= 1, u

(K) 1

= 1, … , u

u1(1) = 1 , u1( 2 ) = 1 , u1( 3) = 1

= 1)

Because of the minimization, we have: B1 − σ 1u1(1) ⊗ u1( 2) ⊗ ... ⊗ u1( K ) ≤ B1

one finds u1(1) = [0.4500, 0.8930 ]T , (6)

Combining (5) and (6), we obtain: σ 1 ≥ 0 . Similarly σ 2 ≥ 0,...., σ q ≥ 0

3 Numerical Example Let us consider the same example as in [6]

u1( 2 ) = [0.3707, 0.5651, 0.7242]T ,

u1( 3) = [0.7716 , − 0.2587 , 0.5214, 2565 ]T and σ 1 = 46.3326 Therefore we can write: Let B1 = B

B ≈ σ 1u1(1) ⊗ u1( 2) ⊗ u1( 3) Moreover, we can find further additional singular values using the steps 2, 3, …. of the above algorithm.

Proceedings of the 6th WSEAS Int. Conf. on Systems Theory & Scientific Computation, Elounda, Greece, August 21-23, 2006 (pp7-13)

The results are in accordance with the result of [6]. The readers can compare.

4 Conclusion The author of the present paper introduced in 1996 a solution to the tensor SVD problem by unifying dimensions into one and transforming finally the KD tensor (K>2) into a 2-D array (matrix). After the paper of 1996, [6], many other researchers followed (See the Proceedings of the Workshop on Tensor Decompositions, held July 19–23, 2004, at the American Institute of Mathematics in Palo Alto, California, see also [11] and [27]). Now, in this paper, the Singular Value Decomposition (SVD) of a tensor (multidimensional array) K>2 with dimensions n1 , n 2 ,..., n K , is faced as an Optimization Problem using appropriate Genetic Algorithm and the method of NelderMead. The method offers satisfactory results and can be applied in a variety of problems in science and technology that use SVD of tensors. It can also be used in factorization of multivariable (multidimensional) polynomials [26]. Note also that the author has introduced the concept of the Positive Singular Value Decomposition in [14]. References:

[1] Horn, Roger A. and Johnson, Charles R , Matrix Analysis. Section 7.3. Cambridge University Press, 1985 [2] Golub, G. H. and Van Loan, C. F. , Matrix Computations. 3rd ed., Johns Hopkins University Press, Baltimore, 1996 [3] J.Ortega, Matrix Theory, Plenum Press, London, 1989. [4] G.Strang, Linear Algebra and its Applications, Hancourt Brace Jovanovich, San Diego, 1988 [5] R.M.Bowen and C.-C. Wang, Introduction to Vectors and Tensors. Linear and Multilinear Algebra, Vol.I and Vol. II, Plenum Press, New York and London, 1976. [6] Nikos E. Mastorakis, "Singular Value Decomposition in Multidimensional Arrays", International Journal of Systems Science, Vol.27, No.7, pp.647-650, July 1996. [7] Proceedings of the Workshop on Tensor Decompositions, held July 19–23, 2004, at the American Institute of Mathematics in Palo Alto, California. [8] J.D. Carroll and J. Chang, Analysis of individual differences in multidimensional scaling via an N-way generalization of “Eckart-

Young” decomposition, Psychometrika, 35 283–319, 1970 [9] R.A. Harshman, Foundations of the PARAFAC procedure: Model and conditions for an “explanatory” multi-mode factor analysis, UCLA Working Papers in Phonetics, 16 1–84., 1970 J.B. Kruskal, Rank, decomposition, and [10] uniqueness for 3-way and N-way arrays, in Multiway Data Analysis, R. Coppi and S. Bolasco (eds.), Elsevier, Amsterdam, 1989, 7– 18. [11] L. De Lathauwer, B. De Moor, and J. Vandewalle, A multilinear singular value decomposition, SIAM J. Matrix Anal. Appl., 21 1253–1278, 2000 Goldberg D.E. (1989), Genetic Algorithms [12] in Search, Optimization and Machine Learning, Addison-Wesley, Second Edition, 1989 Eberhart R., Simpson P. and Dobbins R. [13] (1996), Computational Intelligence PC Tools, AP Professionals. N.E.Mastorakis, "Positive Singular value [14] Decomposition", Recent Advances in Signal Processing and Communications (dedicated to the father of Fuzzy Logic, L.Zadeh), WSEASPress, pp.7-17, 1999, Nikos E. Mastorakis, “On the Solution of [15] Ill-Conditioned Systems of Linear and NonLinear Equations via Genetic Algorithms (GAs) and Nelder-Mead Simplex Searchs”, Proceedings of the 6th WSEAS International Conference on Evolutionary Computing, Lisbon, Portugal, June 16-18, 2005. [16] Nikos E. Mastorakis, “Solving Differential Equations via Genetic Algorithms”, Proceedings of the Circuits, Systems and Computers ’96 (CSC’96), Piraeus, Greece, July 15-17, 1996, 3rd Volume: Appendix, pp.733-737 E. Balagusuramy, Numerical Methods, Tata [17] McGraw Hill, New Delhi, 1999 [18] Gonos I.F., Mastorakis N.E., Swamy M.N.S.: “A Genetic Algorithm Approach to the Problem of Factorization of General Multidimensional Polynomials”, IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications, Part I, Vol. 50, No. 1, pp. 16-22, January 2003. Mastorakis N.E., Gonos I.F., Swamy [19] M.N.S.: “Design of 2-Dimensional Recursive Filters using Genetic Algorithms”, IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications, Part I, Vol. 50, No. 5, pp. 634-639, May 2003.

Proceedings of the 6th WSEAS Int. Conf. on Systems Theory & Scientific Computation, Elounda, Greece, August 21-23, 2006 (pp7-13)

[20] Mastorakis N.E., Gonos I.F., Swamy M.N.S.: “Stability of Multidimensional Systems using Genetic Algorithms”, IEEE Transactions on Circuits and Systems, Part I, Vol. 50, No. 7, pp. 962-965, July 2003. en.wikipedia.org/wiki/Singular_value_deco [21] mposition Lagarias, J.C., J. A. Reeds, M. H. Wright, [22] and P. E. Wright, "Convergence Properties of the Nelder-Mead Simplex Method in Low Dimensions," SIAM Journal of Optimization, Vol. 9 Number 1, pp. 112-147, 1998. [23] J. A. Nelder and R. Mead, “A simplex method for function minimization”, Computer Journal, 7 , 308-313, 1965 F. H. Walters, L. R. Parker, S. L. Morgan, [24] and S. N. Deming, Sequential Simplex Optimization, CRC Press, Boca Raton, FL, 1991 Matlab, Version 7.0.0, by Math Works, [25] Natick, MA, 1994 http://www.mathworks.com N.E.Mastorakis, "A Method of [26] Approximate Multidimensional Factorization via the Singular Value Decomposition", Found. of Comput. Desic. Sci. Vol.21, No.3, pp.137-144, 1996 [27] Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle. On the best rank-1 and rank(R1, R2, dots , RN) approximation of higherorder tensors. SIAM J. Matrix Anal. A., 21(4):1324–1342, 2000.