The OpenMP Xavier Clustering Method

Cluster Analysis Problem Specification The OpenMP Architecture The Xavier Clustering Method (XCM) The XCM-OpenMP Computational Results Conclusions

The OpenMP Xavier Clustering Method Sergio Barbosa Villas-Boas [email protected] Adilson Elias Xavier [email protected] Marcelo Lins de Souza [email protected] PESC/COPPE/UFRJ Universidade Federal do Rio de Janeiro

EURO Optimization 2013 - Rome


Outline 1

Cluster Analysis

2

Problem Specification

3

The OpenMP Architecture

4

The Xavier Clustering Method (XCM)

5

The XCM-OpenMP

6

Computational Results

7

Conclusions


Outline of Presentation 1

Cluster Analysis

2


3


4


5

The XCM-OpenMP

6


7

Conclusions


Applications of Cluster Analysis Clustering is an important problem that appears in a broad spectrum of applications, for example: Psychology Medicine Biology Economics Pattern Recognition Artificial Intelligence Data Mining many others


Cluster Analysis

Cluster Analysis The mathematical modelling of the clustering problem with minimal sum of squares leads to an NP-hard global optimization problem. This problem is not differentiable and non-convex, and has a large number of local minima.


Cluster Analysis Hyperbolic Smoothing Is possible to modify the mathematical modelling of the original problem, smoothing it using a technique known as Hyperbolic Smoothing, proposed by Adilson Xavier. This solving strategy converts the original problem to a differentiable unconstrained one. Solving Strategy The modified problem is then solved taking advantage of minimization methods unconstrained based on derivatives (we used LBFGS).


Simple example of clustering problem

Figure 1: Clusters and centroids observations in R2 .



Cluster Analysis

2


3


4


5

The XCM-OpenMP

6


7

Conclusions


Definition of the clustering problem Let be S= { S1 ,..., Sm } a set of m points (observations) in Rp and q a given number of clusters. Example below: m=6 points in R2 (p = 2), with q=3 clusters

Figure 2: Clustering problem in R2 .


Definition of the clustering problem The union of all clusters must be the original set, that is q [

Ci

i=1

There should exist no point belonging to more than one cluster Ci ∩ Ck = ∅ No cluster can be empty Ci 6= ∅

(1)


Cost function:minimum sum-of-squares

Among the many possible cost functions used in Cluster Analysis, the most natural, intuitive, and frequently adopted is the sum-of-squares of each observation to its centroid. It is called Minimum Sum-of-Squares Clustering (MSSC).


Problem Specification Problem Specification It is a Minimum Sum-of-Squares Minimum distance Clustering Problem (min-sum-min) This is a global optimization mathematical problem. It is a mathematical problem both non-smooth and non-convex, with a large number of local minima. It is in the class of NP-hard problems (Brucker, 1978). After the centroids are calculated, it is necessary to create a list of associations of each observation to its centroid (i.e. the nearest one).



Cluster Analysis

2


3


4


5

The XCM-OpenMP

6


7

Conclusions


Introduction

OpenMP OpenMP is an implementation of multithreading, a method of parallelizing whereby a master thread (a series of instructions executed consecutively) forks a specified number of slave threads and a task is divided among them. The threads then run concurrently, with the runtime environment allocating threads to different processors.


History

OpenMP standard specification started again in 1997 Version 3.1 of the OpenMP specification was released July 9, 2011. This is the version used in this work. The upcoming 4.0 release will be released soon Official web sight http://www.openmp.org/


History OpenMP is managed by the nonprofit technology consortium OpenMP Architecture Review Board (or OpenMP ARB), jointly defined by a group of major computer hardware and software vendors, including AMD IBM Intel Cray HP Fujitsu Nvidia Microsoft Oracle Corporation


Concepts

OpenMP OpenMP is an API that supports multi-platform shared memory multiprocessing programming in C, C++, and Fortran, on most processor architectures and operating systems. It consists of a set of compiler directives, library routines, and environment variables that influence run-time behavior


OpenMP execution model

OpenMP uses the fork-join model of parallel execution. All OpenMP programs begin with a single master thread. The master thread executes sequentially until a parallel region is encountered, when it creates a team of parallel threads (FORK). When the team threads complete the parallel region, they synchronize and terminate, leaving only the master thread that executes sequentially (JOIN).


OpenMP API Overview

The OpenMP API is comprised of three distinct components: Compiler Directives Runtime Library Routines Environment Variables


OpenMP API Overview

Compiler Directives A parallel region is a block of code that will be executed by multiple threads. This is the fundamental OpenMP parallel construct. For example # pragma omp parallel shared(a,b) is used to specify the variables in shared list are visible to all threads


OpenMP API Overview Runtime Library Routines The C++ implementation of the OpenMP standard includes the following environment variables. These environment variables are read at program startup and modifications to their values are ignored at runtime. For example: omp set num threads : Sets the number of threads in subsequent parallel regions. omp get thread num : Returns the thread number of the thread executing within its thread team. omp get num threads : Returns the number of threads in the parallel region.


OpenMP API Overview

Environment Variables A method to alter the execution features of OpenMP applications. Used to control loop iterations scheduling, default number of threads, etc. For example OMP NUM THREADS is used to specify number of threads for an application.


OpenMP example



Cluster Analysis

2


3


4


5

The XCM-OpenMP

6


7

Conclusions


Definitions

Let be S= { S1 ,..., Sm } a set of m points (observations) in Rp and q a given number of clusters. Let be xi , i = { 1, ... ,q } a set of centroids of the clusters, where each xi is called centroid and each xi ∈ Rp .


The XCM Method The set of centroids is represented by X ∈ Rpq . zj = min k sj − xi k xi ∈X

(2)

Object Function Specification Minimize

m X

f (zj )

j=1

where f (zj ) is an arbitrary monotonic increasing function of the distance zj

(3)


Monotonic increasing functions

Trivial Examples of monotonic increasing functions Pm

f (zj ) Pj=1 m j=1 f (zj )

= =

Pm

Pj=1 m

zj2

j=1 zj

Possible Distance Metrics L2 (Euclidean) L1 ( Manhattan) L∞ (Chebychev) Lp (Minkowshi)


The Centroid and observations

Consider a general observation point sj . Let be xi , i = 1, . . . , q, the centroids of clusters where each set of these centroid coordinates will be represented by X ∈ Rpq


The observation belongs to the nearest centroid

Observation Si belongs to centroid x1 because it is the one that produces the smallest zj (smallest distance). If centroid x1 moves, the Si may belong to some other centroid. The function to calculate zj (smallest distance to centroid) with respect to xi is strongly non differentiable.


Transforming the problem (P1)

The general clustering problem is equivalent to the following problem:

(P1)Minimize

m X

f (zj ) (4)

j=1

Subject to : zj = min k sj − xi k, j = 1, . . . , m i=1..q


Relaxing the problem (P2)

From the problem (P1), it is obtained the relaxed problem:


Auxiliary function ϕ(x) Let us use the auxiliary function ϕ(x) = max {0,x}.

From the inequalities zj - k sj - xi k ≤ 0, j=1,. . . ,m, i=1,. . . ,q . is follows that q X i=1

ϕ(zj − k sj − xi k) = 0, j = 1, . . . , m

(5)


Re-stating the problem (P3)

Using this set of equality constraints in place of the constraints in (P2), we obtain the following problem


Observing function ϕ


Adding eps and re-establishing the problem (P4) By performing a ε perturbation, it is obtained the problem:

(P4)Minimize

m X

f (zj )

j=1

Subject to :

q X

ϕ(zj − k sj − xi k) ≥ ε, j = 1, . . . , m, ε > 0

i=1

(6)


Seeing eps in graph


Smoothing the problem Let us define the auxiliary function φ(x, τ ) = for x ∈ R e τ > 0.

(x +

√

x 2 + τ 2) 2

(7)


Properties of PHI

Function φ has the following properties (a) φ(x, τ ) > ϕ(x), ∀τ > 0; (b) limτ →0 φ(x, τ ) = ϕ(x); (c) φ(x, τ ) is an increasing convex C ∞ function.


Smoothing the problem By using the Hyperbolic Smoothing approach for the problem (P4) it is obtained

(P5)Minimize

m X

f (zj )

j=1

Subject to :

q X i=1

(8) φ(zj − k sj − xi k, τ ) ≥ ε, j = 1, . . . , m


Graph view of smoothed function


Smoothing the calculation of distance

For the Euclidian metric, let us define the auxiliary function v u n uX θ(sj , xi , γ) = t (sjl − xil )2 + γ 2 l=1

for γ > 0.

(9)


Properties of theta function

(a) limγ→0 θ(sj , xi , γ) =k sj − xi k2 ; (b) θ is a C ∞ function.


Completely smoothed problem (P6)

Replacing the Euclidean distance k sj − xi k2 of the problem (P5) by θ function (P6)Minimize

m X

f (zj )

j=1

Subject to :

q X i=1

(10) φ(zj − θ(sj , xi , γ), τ ) ≥ ε, j = 1, . . . , m


Preparing to solve(P7) By considering the KKT (Karush-Kuhn-Tucker) conditions for the problem (P6), it is possible to conclude that equation below is true. (P7)Minimize F (x) =

st : hj (zj , x) =

m X

f (zj )

j=1 q X

φ(zj − θ(sj , xi , γ), τ ) − ε = 0, j = 1, ..., m

i=1

(11)


Remarks

The variable domain space has (pq + m) dimensions. All functions of this problem are C ∞ functions in relation to the variables (x,z) Important Each variable zj appears only in one equality constraint h(x, zj ) ∀ j = 1, . . . , m


Remarks

The partial derivative of h(x, zj ) in relation to zj is not equal to zero for all j = 1, ..., m; In the face of the previous remarks, it is possible to use the Implicit Function Theorem in order to calculate each component zj , j = 1, . . . , m, as a function of the centroids variables xi , i = 1, . . . , q.


Problem turns up to be an unconstrained optimization problem (P8)

(P8) Therefore, by using the Implicit Function Theorem, it is obtained the unconstrained problem unconstrained optimization problem, only variable is x (centroid positions) BFGS Minimize F (x) =

m X j=1

f (zj (x))

(12)


Problem turns up to be an unconstrained optimization problem (P8) where each zj (x) is obtained by the calculation of a zero of each equation applying Newton Raphson h(x, zj ) =

q X

φ(zj − θ(sj , xi , γ), τ ) − ε = 0, j = 1, ..., m

i=1

(13) for each xi calculates zj


Gradient of function Again, due the Implicit Function Theorem, the functions zj (x) have all derivatives in relation to the variables xi , i = 1, . . . , q. So, it is possible to calculate the gradient of the objective function of the problem (P8) ∇F (x) =

m X df (zj (x)) i=1

dzj

∇zj (x)

(14)

∂h(x, zj ) ∂zj

(15)

where ∇zj (x) = −∇h(x, zj )/


Remarks

It can be solved by making use of any method based in the first and second order derivatives information (conjugate gradient, quasi-Newton, Newton methods, etc). We used lbfgs. It is defined in a pq-dimensional space, therefore a smaller dimension problem in relation to the original problem (P7).


From NLP to Unconstrained


Heat and gradually cool simplified algorithm choose some initial point x. Start with large parameters, to have significant smoothing effect. Large parameters = high temperature. solve unconstrained problem. lower the parameters (cool the problem), use the previous solution as initial point. keep cooling and solving the problem until the parameters are sufficiently small.


Heat and gradually cool


Simplefied XCM


Remarks

The algorithm causes τ and γ approach zero, so the constraints of the sub problems it solves, given as in (P7), tend to those of (P4). The algorithm causes ε approaches to zero, so, in a simultaneous movement, the solved problem (P4) gradually approaches to problem (P3).



Cluster Analysis

2


3


4


5

The XCM-OpenMP

6


7

Conclusions


The XCM-OpenMP Method

XCM-OpenMP The transformation of problem P7 to problem P8, generate a large reduction in the number of variables of the problem allows the formulation like a function defined by a summation of parcels independent



We can see this idea in the function below, where each calculation f (zj ) depends only on x, in other words, the solution of the current iteration of the method

minimize F (x) =

m X

zj (x)2 = z1 (x)2 + z2 (x)2 + z3 (x)2 + . . . + zm (x)2

j=1

(16)



Main Proposition In the equation (13) there is need to find the zero of each of the m functions. The calculation of zi is independent one of each other (no internal dependency). With XCM-OpenMP we propose to solve each parcel of F (x) using multiple OpenMP threads.



Parallelization of gradient The same concept can be applied to the gradient function ∇F . The equation (14) defines the gradient. Function ∇F is a sum of independent parts, calculated in independent OpenMP threads.


The XCM-OpenMP Method The serial and parallel implementation are essentially the same mathematical XCM method. Main Differences Calculation of the objective function Gradient of the objective function

Once calculated the value of F (x) and ∇F (x), the values are passed to the function of unconstrained minimization, which was implemented using LBFGS (the same way both sequential and parallel version).


Entity Relationship Model to compare the algorithms


Some intricate details

The initialization step of the XCM-OpenMP differs from XCM class because before the vectors are allocated, it is necessary to know the number of OpenMP threads. Otherwise the initialization of both methods (XCM and XCM-OpenMP) are identical.


The XCM-OpenMP

The data concurrency control of XCM-OpenMP is elegantly simple, implemented only by consulting the thread id. Using a classic procedure of parallel computing, when all CPU threads finish the calculation of the parcels of F and ∇F , the values are reduced from a local (to the scope of the thread) variable in the non parallel scope.



Cluster Analysis

2


3


4


5

The XCM-OpenMP

6


7

Conclusions


Computational Results Implementation Details Implementation of XCM and XCM-OpenMP in Coyote Server with 24 cores Ubuntu version 13.04 (local) and Suse 12.1 (server) g++ 4.7 OpenMP 3.1 Netbeans 7.3 L-BFGS (Nocedal) Octave (to display results)


Problem TSPLIB-3038

Figure 3: TSPLIB-3038 (q = 5).


Problem TSPLIB-3038



Problem TSPLIB-3038



Problem TSPLIB-3038



Problem TSPLIB-3038



Problem TSPLIB-3038



Problem TSPLIB-3038 Compare Time and Speed-Up: Centroids Numbers 5 10 20 30 40 50

Seq. Time 3.204 8.861 31.701 82.946 122.795 242.395

Par. Time 0.302 0.442 1.112 3.367 5.272 9.004

Speed Up 10.609 20.049 28.508 24.634 23.291 26.920

Table 1: Time and Speed Up Results for Problem TSPLIB-3038


Problem TSPLIB-3038

Figure 9: Graphic results of problem TSPLIB-3038.


Problem 15112

Figure 10: Problem d15112 (q = 5).


Problem 15112



Problem 15112



Problem 15112



Problem 15112



Problem 15112



Problem 15112 Compare Time and Speed-Up: Centroids Numbers 5 10 15 20 25 30

Seq. Time 18.140 54.562 100.120 197.397 244.437 349.601

Par. Time 1.419 3.369 5.994 10.312 11.705 17.504

Speed Up 12.783 16.195 16.718 19.142 21.820 19.972

Table 2: Time and Speed Up Results for Problem 15112


Problem TSPLIB-15112

Figure 16: Graphic results of problem 15112.


Problem pla85900

Figure 17: Problem 85900 (q = 5).


Problem pla85900



Problem pla85900



Problem pla85900



Problem pla85900



Problem pla85900 Compare Time and Speed-Up: Centroids Numbers 5 10 15 20 25 30

Seq. Time 103.594 326.780 593.960 1059.451 1471.468 2253.773

Par. Time 9.137 20.863 33.285 55.963 70.779 117.727

Speed Up 11.337 15.663 17.844 19.023 20.790 19.144

Table 3: Time and Speed Up Results for Problem 85900


Problem pla85900

Figure 22: Graphic results of problem 85900.



Cluster Analysis

2


3


4


5

The XCM-OpenMP

6


7

Conclusions


Conclusions In this paper addressed the following topics Cluster Analysis, Unconstrained optimization, OpenMP Architecture XCM Method, Analysis and Implementation XCM-OpenMP Delevopment Conclusions The experimental results show great speed up the XCM-OpenMP version against the version XCM. In tests, the highest speed up was greater than 28 times.Emphasize that there was no loss in numerical accuracy

The OpenMP Xavier Clustering Method

The OpenMP Xavier Clustering Method

Suggest Documents

OpenMP

Ward's Hierarchical Clustering Method: Clustering Criterion and

Two-level clustering towards unsupervised discovery ... - Xavier Anguera

An OpenMP algorithm and implementation for clustering ... - WSU EECS

OpenMP Under The Hood

Cluster-enabled OpenMP: An OpenMP compiler for the ... - Hindawi

The Subsumption Method versus Hierarchical Clustering - Erasmus ...

Self-Organizing Map Clustering Method for the

The Method of Quantum Clustering - CiteSeerX

Geometric Clustering using the Information Bottleneck method

Method for Determining Appropriate Clustering

OpenMP 4.0

OpenMP 3.1

XAVIER SCHOOL

Xavier Serra

OpenMP 3.0

XAVIER SCHOOL

Grid-Clustering: A Fast Hierarchical Clustering Method for Very Large ...

Self-Tuning Clustering: An Adaptive Clustering Method for Transaction ...

Grid-Clustering: A Fast Hierarchical Clustering Method for ... - CiteSeerX

OpenMP as An Efficient Method to Parallelize Code

The OpenMP* Common Core - OpenMPCon

User co-scheduling for MPI + OpenMP applications using OpenMP

Loop Scheduling in OpenMP