Distributed and Stream Data Mining Algorithms for ... - Ca' Foscari

` Ca’ Foscari di Venezia Universita

Dipartimento di Informatica Dottorato di Ricerca in Informatica

Ph.D. Thesis: TD-2006-4

Distributed and Stream Data Mining Algorithms for Frequent Pattern Discovery Claudio Silvestri

Supervisor Prof. Salvatore Orlando

PhD Coordinator Prof. Simonetta Balsamo

Author’s Web Page: http://www.dsi.unive.it/∼claudio Author’s e-mail: [email protected] Author’s address: Dipartimento di Informatica Universit`a Ca’ Foscari di Venezia Via Torino, 155 30172 Venezia Mestre – Italia tel. +39 041 2348411 fax. +39 041 2348419 web: http://www.dsi.unive.it

To my wife

Abstract The use of distributed systems is continuously spreading in several applications domains. Extracting valuable knowledge from raw data produced by distributed parties, in order to produce a unified global model, may presents various challenges related to either the huge amount of managed data or their physical location and ownership. In case data are continuously produced (stream) and their analysis is required to be performed in real time, communication costs and resource usage are issues that require careful attention in order to run computation in the optimal location. In this thesis, we examine in details the problems related to the Frequent Pattern Mining (FPM) in distributed and stream data and present a general framework for adapting an exact FPM algorithm to a distributed or streaming context. The FPM problems we consider are Frequent Itemset Mining (FIM), and Frequent Sequences Mining (FSM). In the first case, the input data are sets of items and the frequent patterns are those included in a user-specified number of input set. The second one consists in finding frequent sequential patterns in a database of time-stamped events. Since the proposed framework uses (exact) frequent pattern mining algorithms as the building block of the approximate distributed/stream algorithms, we will also describe two efficient algorithms for FIM and FSM: DCI, introduced by Orlando et al., and CCSM, which is one of the original contributions of this thesis. The resulting algorithms for distributed and stream FIM have been tested with real world and synthetic datasets, and are able to find efficiently a good approximation of the exact results and scale gracefully. The framework for FSM is almost identical, but has not been tested yet. The few differences are highlighted in the conclusion chapter.

Sommario La diffusione dei sistemi distribuiti `e in continuo aumento in svariati campi applicativi e l’estrazione di correlazioni non evidenti nei dati grezzi prodotti pu`o essere strategica per le organizzazioni coinvolte. Questo tipo di operazione `e generalmente non banale e, quando i dati sono distribuiti, presenta ulteriori difficolt`a legate sia alla mole di dati coinvolti che alla loro propriet`a e locazione fisica. Nel caso i dati siano prodotti in flussi continui (stream) e sia necessario analizzarli in tempo reale, l’ottimizzazione dei costi di comunicazione e delle risorse necessarie sono aspetti che debbono essere presi attentamente in considerazione. In questa tesi sono analizzati in modo dettagliato i problemi legati alla ricerca di pattern frequenti (FPM) su dati distribuiti e stream di dati. In particolare `e presentato un metodo generale per ottenere, a partire da un qualunque algoritmo esatto per FPM, un algoritmo approssimato per il FPM su dati distribuiti e stream di dati. I tipi di pattern presi in considerazione sono gli itemset frequenti (FIM) e le sequenze frequenti (FSM). Nel primo caso i dati in ingresso sono insiemi di elementi (transazioni) ed i pattern frequenti sono a loro volta degli insiemi contenuti almeno in un numero di transazioni specificato dall’utente. Il secondo consiste invece nella ricerca di pattern sequenziali frequenti in una collezione di sequenze di eventi associati a precisi istanti di tempo. Poich´e il metodo proposto utilizza degli algoritmi esatti per l’estrazione di pattern frequenti come parti da riunire per ottenere degli algoritmi per dati distribuiti e stream di dati, verranno anche descritti due algoritmi efficienti per FIM e FSM: DCI, presentato da Orlando ed altri, e CCSM, che `e uno dei contributi originali di questa tesi. Gli algoritmi ottenuti applicando il metodo proposto sono stati utilizzati sia con dati reali sia con dati sintetici per valutarne l’efficacia. Gli algoritmi per FIM si sono dimostrati scalabili ed in grado di estrarre efficientemente una buona approssimazione della soluzione esatta. Il modello per FSM `e quasi identico, ma non `e ancora stato verificato sperimentalmente. Le poche differenze sono evidenziate nel capitolo finale.

Acknowledgments I would like to thank Prof. Salvatore Orlando for his guidance and support during my Ph.D. studies. I am also grateful to him for the opportunity to collaborate with the ISTI-CNR High Performance Computing Lab. In this context, I would like to thank Raffaele Perego, Fabrizio Silvestri, and Claudio Lucchese who co-authored some of the papers I published and, in several ways, helped me in improving the quality of my work. I thank my referees, Prof. Hillol Kargupta and Prof. Rosa Meo, for their attention in reading this thesis and their valuable comments. Most part of this work has been carried out at the Dipartimento di Informatica, Universit`a Ca’ Foscari di Venezia. I would like to thank all the faculty and personnel for their support and for making the department a friendly place for doing research. Special thanks to Moreno and Matteo, for the long discussions on free software and other interesting subjects, and to all the others (ex-) Ph.D. students for the pleasant time spent together: Chiara, Claudio, Damiano, Fabrizio, Francesco, Giulio, Marco, Matteo, Massimiliano, Ombretta, Paolo, Silvia, and Valentino. In this last year I have been a guest at the Dipartimento di Informatica e Comunicazione, Universit`a degli Studi di Milano. I am grateful to Maria Luisa Damiani, for the opportunity of collaboration, and to the people working at the DB&SEC Lab, for the friendly working environment. This work was partially supported by the PRIN’04 Research Project entitled ”GeoPKDD - Geographic Privacy-aware Knowledge Discovery and Delivery”. Finally, I would like to thank my extended family, who has never lost faith in this long-term project, and all of my friends.

Contents 1 Introduction 1.1 Data distribution . . . . . . . . . 1.2 Data evolution . . . . . . . . . . 1.3 Applications . . . . . . . . . . . . 1.4 Association Rules Mining . . . . . 1.4.1 Frequent Itemsets Mining 1.4.2 Frequent Sequence Mining 1.4.3 Taxonomy of Algorithms . 1.5 Contributions . . . . . . . . . . . 1.6 Thesis overview . . . . . . . . . .

I

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

First Part

2 Frequent Itemset Mining 2.1 The problem . . . . . . . . . 2.1.1 Related works . . . . 2.2 DCI . . . . . . . . . . . . . . 2.2.1 Candidate generation 2.2.2 Counting phase . . . 2.2.3 Intersection phase . . 2.3 Conclusions . . . . . . . . .

1 2 3 6 7 9 10 11 13 14

17 . . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

3 Frequent Sequence Mining 3.1 Introduction . . . . . . . . . . . . . . . . 3.2 Sequential patterns mining . . . . . . . . 3.2.1 Problem statement . . . . . . . . 3.2.2 Apriori property and constraints . 3.2.3 Contiguous sequences . . . . . . . 3.2.4 Constraints enforcement . . . . . 3.3 GSP . . . . . . . . . . . . . . . . . . . . 3.3.1 Candidate generation . . . . . . . 3.3.2 Counting . . . . . . . . . . . . . . 3.4 SPADE . . . . . . . . . . . . . . . . . . . 3.4.1 Candidate generation . . . . . . . 3.4.2 Candidate support check . . . . . 3.4.3 cSPADE: managing constraints . .

. . . . . . .

. . . . . . . . . . . . .

. . . . . . .

. . . . . . . . . . . . .

. . . . . . .

. . . . . . . . . . . . .

. . . . . . .

. . . . . . . . . . . . .

. . . . . . .

. . . . . . . . . . . . .

. . . . . . .

. . . . . . . . . . . . .

. . . . . . .

. . . . . . . . . . . . .

. . . . . . .

. . . . . . . . . . . . .

. . . . . . .

. . . . . . . . . . . . .

. . . . . . .

. . . . . . . . . . . . .

. . . . . . .

. . . . . . . . . . . . .

. . . . . . .

. . . . . . . . . . . . .

. . . . . . .

. . . . . . . . . . . . .

. . . . . . .

. . . . . . . . . . . . .

. . . . . . .

. . . . . . . . . . . . .

. . . . . . .

19 20 20 21 22 22 24 25

. . . . . . . . . . . . .

27 28 30 30 32 32 33 34 34 35 35 35 36 37

ii

Contents

3.5

3.6 3.7

II

CCSM . . . . . . . . . . . . . . 3.5.1 Overview . . . . . . . . 3.5.2 The CCSM algorithm . . 3.5.3 Experimental evaluation Related works . . . . . . . . . . Conclusions . . . . . . . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

Second Part

37 38 38 43 45 47

49

4 Distributed datasets 4.1 Introduction . . . . . . . . . . . . . . . . . 4.1.1 Frequent itemset mining . . . . . . 4.2 Approximated distributed frequent itemset 4.2.1 Overview . . . . . . . . . . . . . . 4.2.2 The Distributed Partition algorithm 4.2.3 The APRed algorithm . . . . . . . . 4.2.4 The APInterp algorithm . . . . . . . 4.2.5 Experimental evaluation . . . . . . 4.3 Conclusions . . . . . . . . . . . . . . . . .

. . . . . . . . mining . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

51 51 53 53 54 55 57 59 62 68

5 Streaming data 5.1 Streaming data . . . . . . . . . 5.1.1 Issues . . . . . . . . . . 5.2 Frequent items . . . . . . . . . 5.2.1 Problem . . . . . . . . . 5.2.2 Count-based algorithms 5.2.3 Sketch-based algorithms 5.3 Frequent itemsets . . . . . . . . 5.3.1 Related work . . . . . . 5.3.2 The APStream algorithm 5.4 Conclusions . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

73 73 74 74 75 75 81 82 82 84 95

III Conclusions A Approximation assessment Bibliography

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

97 99 103 107

List of Figures 1.1 1.2 1.3 1.4 1.5 1.6

Incremental data mining . . . . . . . . . . . Data stream mining . . . . . . . . . . . . . . Transaction dataset . . . . . . . . . . . . . . Sequence dataset . . . . . . . . . . . . . . . Effect of maxGap constraint . . . . . . . . . Taxonomy of algorithms for frequent pattern

. . . . . . . . . . . . . . . . . . . . . . . . . mining .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

2.1 2.2

Set of itemsets compressed data structure . . . . . . . . . . . . . . . 23 Example of cache usage . . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8

GSP candidate generation . . . . . . . . . . . . . . . . . . . . . CCSM candidate generation . . . . . . . . . . . . . . . . . . . . Example of cache usage . . . . . . . . . . . . . . . . . . . . . . . CCSM idlist reuse . . . . . . . . . . . . . . . . . . . . . . . . . . Number of intersection for different intersection methods . . . . Number of frequent sequences in datasets CS11 and CS21 . . . . Execution times of CCSM and cSPADE- variable maxGap value . Execution times of CCSM and cSPADE- fixed maxGap value . .

. . . . . . . .

. . . . . . . .

34 40 41 42 43 44 45 45

4.1 4.2 4.3 4.4 4.5

Similarity of APRed approximate results . . . . . . . . . . . . . . . . Number of spurious patterns as a function of the reduction factor r fpSim of the APInterp results . . . . . . . . . . . . . . . . . . . . . . Comparison of Distributed One-pass Partition vs. APInterp . . . . . . Speedup for two of the experimental datasets . . . . . . . . . . . . .

. . . . .

66 67 68 69 70

. . . . . . . .

. . . . . .

. 5 . 6 . 9 . 10 . 12 . 13

5.1 Similarity and ASR as a func. of memory/transactions/hash entries . 95 5.2 Similarity and ASR as a function of stream length . . . . . . . . . . . 96 C.3 Distributed stream mining framework . . . . . . . . . . . . . . . . . . 101

iv

List of Figures

List of Tables 1.1

Taxonomy of data mining environments . . . . . . . . . . . . . . . . .

4.1 Datasets used in APRed experimental evaluation . 4.2 Datasets used in APInterp experimental evaluation 4.3 Test results for APRed . . . . . . . . . . . . . . . . 4.4 Accuracy indicators for APInterp results . . . . . . 5.1 5.2

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

4 63 64 65 71

Sample supports and reduction ratios . . . . . . . . . . . . . . . . . . 88 Datasets used in experimental evaluation . . . . . . . . . . . . . . . . 93

vi

List of Tables

1 Introduction Data mining is, informally, the extraction of knowledge hidden in huge amounts of data. However, if we are interested in a more detailed definition, several different ones do exist [23]. Depending on the application domain (and on the author), Data mining could just mean the extraction of a particular aggregate information from somehow preprocessed data, or the whole process beginning with data cleaning and integration, and ending with result visual representation. From now on, we will reserve the term Data mining to the first meaning, using the more general KDD (Knowledge Discover in Databases) for the whole workflow that is needed in order to apply Data mining algorithms to real world problems. The kind of knowledge we are interested in, together with the organization of input data and the criteria used to discriminate among useful and useless information, contributes to characterize a specific data mining problem and its possible algorithmic solutions. Common data mining tasks are the classification of new objects according to a scheme learned from examples, the partitioning of a set of objects into homogeneous subsets, the extraction of association rules and numerical rules from a database. In several interesting application frameworks, such as wireless network analysis and fraud detection, data are naturally distributed among several entities and/or evolve continuously. In all of the above-indicated data mining tasks, dealing with either of these peculiarities provides additional challenges. In this thesis we will focus on the distribution and evolution issues related to the extraction of Association Rules from transactional databases (ARM), one of the most important and common data mining task, both for the immediate applicability of the knowledge extracted by this kind of analysis, and for the wide range of application fields where it can be used. Association Rules are rules relating the occurrence of distinct subset of items in the same set, i.e. ”65 % of market basket containing steak and salad will also contains wine”, or in the same collection of set ”50 % of customer that buy a CD player will, later, buy CDs”. In particular, we will concentrate our attention on the most computationally expensive phase of ARM, the mining of frequent patterns from distributed and stream data. These patterns can be either frequent itemsets (FIM) or frequent sequences (FSM), i.e., respectively subsets contained in at least a user indicated number of input set and subsequence of at least a user specified number

2

1. Introduction

input sequences. Since we will use frequent pattern mining algorithms for static and non-evolving datasets as the building block for our approximate algorithms, to be exploited on distributed and stream data, we will also describe efficient algorithms for FIM and FSM: DCI, introduced by Orlando et al. in [44], and CCSM, which is one of the original contributions of this thesis. This chapter introduces, without focusing on any particular data mining task, the general issues concerning the evolution of data, and their distribution/partitioning among several entities. Then it quickly introduces ARM and its core FIM/FSM phase in centralized and non-evolving datasets. Both will be discussed more in detail both the first part of the thesis, since they constitute the foundation for the distributed and streaming FIM/FSM problems. We also deal with taxonomy of both FIM and FSM algorithms, which will be useful in understanding the reasons that lead us to the choice of DCI and CCSM algorithms as the building blocks for our distributed and stream algorithms. The chapter concludes with a summary of the achievements of our research, and a description of the structure of the rest of the thesis.

1.1

Data distribution

Reasons leading to data distribution. In many real systems, data are naturally distributed, usually due to a plural ownership or to a geographical distribution of the processes that produce the data. The logistic organization of entities involved in the data collection process, performance and storage constraints, as well as privacy and company interest, may lead to the choice of using separate databases, instead of a centralized one accessed by several remote locations. The sales point of a large chain are a typical example: there is no need of a central database for performing ordinary sale activities, and using it would make the operations of the shop dependent on the reliability and bandwidth of the communication infrastructure. Gathering all data to a single site, after they have been produced, would be subject to the same ownership/privacy issues as using a centralized database. In other cases, data are produced locally in large volumes and immediately moved to other storage and analysis locations, due to the impossibility to store or process them with the resources available at a single site, as in the case of satellite image analysis or high-energy physics experiments. In all of these cases, performing a data mining task means to coordinate the sites in a mix of partial movement of data and exchange of local knowledge, in order to get the required global model. Homogeneous vs. heterogeneous data. Problems that are seemingly similar may need sensibly different solutions, if considered in different communication and data localization settings. Data can be either homogeneous or heterogeneous. If data are represented by tuples, in the first case all data presents the same dimensions,

1.2. Data evolution

3

while in the second one data each node has its own schema. Let us consider two examples: the sales data of a chain of shops and the personal data collected about us by different department of public administration. Sales data contain a representation of the sale transactions and are maintained by the shop where the items were bought. In this case, data are homogeneous: data collected at different shop are similar, but related to different transactions. Personal data are also maintained at different site: the register office manages birth data, the tax register own tax data, another register collect data about our cars. In this case, data are heterogeneous, since for each individual each register maintains different kind of data. Data localization is a key factor in characterizing data mining problems. Most classical data mining algorithms expect all data to be grouped in a unique data repository. Each data mining task presents different issues when it is considered in a distributed environment, instead of a centralized one. However it is possible to identify a general requirement common to most distributed data mining system architectures and tasks: careful attention should be paid to both communication and computation resources, in order to use them in a nearly optimum way. Data distribution issues and algorithms will be discussed in more details in chapter 4, with a focus on frequent pattern mining algorithms. A good survey on distributed data mining algorithms and applications is [48].

1.2

Data evolution

In different application context, data mining can be applied either to past data, as a one time task, or repeatedly to evolving datasets. The classical data mining algorithms refer to the first case: the full dataset is available and there will be no data modification during the computation or between two consecutive computations. This is enough to understand a phenomenon and make plans for the future in most cases. In several applications, like wireless network analysis, intrusion detection, stock market analysis, sensor network data analysis, and, in general, any setting in which every information available should be used to make an immediate decision, the approach based on finite statically stored data sets could be not satisfactory. These cases demands for new classes of algorithms, able to cope with evolutions of data. In particular, two issues need to be addressed: the complexity of recomputing everything from scratch and the potential infiniteness of data. In case only the first issue is present, the setting is referred to as Incremental/Evolving Data Mining; otherwise, it is indicated as Stream Data Mining. The presence and kind of data evolution is another key factor in characterizing data mining problems.

4

1. Introduction

Data localization Centralized

A single entity can access every data

Distributed

Each node can access just a part of the data and ...

homogeneous

... data related to the same entity (e.g.: people) are owned by just one node

heterogeneous

... data related to the same entity (e.g.: people) may be spread among several nodes

Data evolution Statical

Data are definitively stored and invariable (e.g.: related to some past and concluded event)

Incremental

New data are inserted and access to past data is possible (e.g.: related to an ongoing event)

Evolving

The dataset is modified with either updates, insertions or deletions, and access to past data is possible.

Streaming

Data arrives continuously and for an indefinite time. Access to past data is restricted to a limited to part of them or summaries. Table 1.1: Taxonomy of data mining environments.

Incremental and Evolving Data Mining. In incremental data mining, new data are repeatedly inserted into the dataset. Some algorithm also take care of deletions or modifications of previous data, this case is indicated as evolving data mining. In a typical situation, we have a dataset D and the results of the required data mining task on D. Then D is modified and the system is asked for a new result set. Obviously, a way to obtain the new result is to recompute everything from scratch, and it is possible since all past data are accessible. However, this implies a computation time that in some case may clash with near real time system response requirements, whereas in other cases is just a waste of resources, especially when the dataset get bigger. Incremental/Evolving data mining algorithms, instead, are able to update the solution according to dataset updates, modifying just the part of the result set that is interested by the modifications of the dataset. A fitting example could concern the sales data of a supermarket: at the end of each day, the daily update is performed. The overall amount of data is still reasonable for an ordinary computation; however, there is no point in reprocessing some year of past sales data. A better approach would be considering the past result and the new data, and querying the past data only when a modification of the result

1.2. Data evolution

5

is expected. Figure 1.1 summarize the simultaneous evolution of data and results

Figure 1.1: Incremental data mining: previous result availability allows for a reduction of necessary computation.

after each mining step in incremental data mining. Stream Data Mining. An increasing number of applications require support for processing data that arrive continuously and in huge volumes. This setting is referred as Stream Data Mining. The main difference with Incremental/Evolving Data Mining is the large and potentially infinite amount of data, but also the continuity aspect deserves some attention. The first consequence is that Stream Data Mining algorithms, since they are dealing with infinite data, cannot access to every data received in the past, but just to a limited subset of them. In case of sustained arrival rate, this means that each received data can be read only a few times, often just once. An algorithm dealing with data streams should require an amount of memory that is not related to the (infinite) amount of data analyzed. At the same time, it should be able to cope with the data arrival rate, returning, if necessary, an approximate solution in order to keep up with the stream. Building a model based on every received data until the user makes the query can be simply impossible in most cases, either for response time or resource contraints. Even the apparently trivial task of exactly counting the number of items received so far, potentially requires infinite memory, since after N items received we will need log2 (N ) bits in order to represent the counters. A solution that requires O(log(N )) memory is, however, considered suitable for a streaming context, since for real data stream infinite actually means really long. However,

6

1. Introduction

Figure 1.2: Data stream mining: data are potentially infinite and accessible just on arrival. Results can be referred to the whole stream or a limited part.

if we slightly extend the problem, asking for the number of ”distinct” items, an exact answer is impossible without using a O(N ) memory. For this reason, in data stream mining, approximate algorithms are quite common. Another way to reduce the resource requirement is to restrict the problem to a user specified temporal window, e.g. the last week. This approach is called Window Model, whereas the previously introduced one is the Landmark Model. Figure 1.2 summarize these two different approaches.

1.3

Applications

The issues encountered when mining data originated by distributed sources may be related to the quality of received data, the high data arrival rate, the kind of communication infrastructure available between data sources and data sinks, or the need to avoid privacy breach. Let us see three practical cases of distributed systems and practical motivations that may lead to the use of distributed data mining algorithms for the analysis of data, instead of collecting and processing everything in a central repository. Geographically distributed Web-farm. Popular web sites generate a lot of traffic from the server to the clients. A solution viable to ensure high availability and throughput is to use several geographically distributed replicas and let each client connect to the closer one (e.g. www.tucows.com). This approach, even if really practical for system availability network and response time, makes the analysis of data for user behavior and intrusion detection more complex. In fact, while using a single web-farm all access log are available in the same site, in this case they are partitioned among several farm, sometimes connected by high latency link. A na¨ıve

1.4. Association Rules Mining

7

solution is to collect all data in a centralized database, either periodically or in real time, and it is in most case the best solution, at least if the data arrival rate is low or we are not interested in recent data. However, this is not satisfying when log data are huge, and real time analysis is required, as for intrusion detection. Sensor network. The same kind of problems may arise, even worse, when the sources of data streams are sensors connected by a network. Quite often communication link with sensors have a reduced bandwidth, for example in case of seismic sensors placed in inhabited places, far from computation infrastructures. Financial network. Furthermore data centralization may be unfeasible when confidential information are handled and must not be shared with unauthorized subjects in order to protect privacy rights or company interests. A classical example concerns the credit card fraud detection. Let us suppose that a group of banks is interested in automatically detecting possible frauds; each participating entity is interested in making the resulting model accurate, and based on as much data as possible, but banks cannot communicate the transactions of their customers to other banks. In all these cases, even if for different reasons, collecting all raw data to a repository before analyzing them is unfeasible and distributed techniques are needed in order to elaborate, at least partially, the data in place.

1.4

Association Rules Mining

As we have seen in the previous section, dealing with evolving and distributed data presents several issues, independently of the particular targeted data mining task. However, each data mining task has its peculiarities, and the issues in different cases are not really the same, but just similar and related to the same aspect. In order to analyze more thoroughly the issues and possible solutions, we have to focus on a particular task or group of tasks. We have decided, in this thesis, to concentrate our attention on Association Rules Mining, and more precisely on its most computationally expensive phase, the mining of frequent patterns in distributed dataset and data stream, where these patterns can be either itemsets (FIM) or sequences (FSM). In this section we will quickly introduce the Association Rule Mining (ARM), one of the most popular DM task [4, 18, 19, 54], both for the immediate applicability of the knowledge extracted by this kind of analysis and for the wide range of application fields where it can be applied, from medical symptoms developed in a patient to objects sold in a commercial transaction. Here our goal is just to quickly introduce this topic, and its computational challenging Frequent Pattern Mining sub problem, by limiting our attention to the centralized case. A more detailed description of the problem will be found in chapters 2 and 3 for the centralized sub problems, in chapter 4 for the distributed one and in chapter 2 for the stream case.

8

1. Introduction

The essence of associative rules is the analysis of the co-occurrence of facts in a collection of set of facts. If, for instance, the data represent the objects bought in the same shopping chart by the customers of a supermarket, then the goal will be finding rules relating the fact that a market basket contains an item with the fact that another item has been bought at the same time. One of these rules could be ”people who buy item A also buy item C in conf % cases”, but also the more complex ”people who buy item A and item B also buy item C in conf % cases” where conf % is the confidence of the rule, i.e. a measure of how much that rule can be trusted. Another interestingness measure, frequently used in conjunction with confidence, is the support of a rule, which is defined as the number of records in the database that confirm the rule. Generally, the user specifies minimum thresholds for both, so an interesting rule should have both a high support and a high confidence, i.e. it should be based on a significant number of cases to be useful, and at the same time, there should be few cases in which it is not valid. The combined use of support and confidence is the measure of interestingness most commonly adopted in literature, but in some case can be misleading if the user does not look carefully at the big picture. Consider the following example: both A and B appear in 80% of input data, and in 60% of cases, they appear in the same = 75%, transaction. The rule ”A implies B” has support 60% and confidence 60 80 thus apparently this is a good rule. However, if we analyze the full context, we can see that the confidence is lower than the support of B, hence the actual meaning of this rule is that A negatively influences B. The usage of other interestingness measures has been widely discussed. However, there is no clear winner, and the choice depends on the specific application field. Sequential rules or (temporal association rules) are an extension of association rules, which also considers sequential relationships. In this case, the input data are sequences of set of facts and the rules have to deals with both co-occurrences and ”followed by” relationships. Continuing with the previous example about market basket analysis (MBA), this means considering each transaction as related to a customer, identified by a fidelity card or something similar. So each input sequence is the shopping history of a customer and a rule could be ”people who buy item A and item B at the same time will also buy item C later in conf % cases” or ”people who buy item A followed by item B within one week will also buy item C later in conf % cases”. The extraction of both association rules and sequential rules from a database is typically composed of two phases. First, it is necessary to find the so-called frequent patterns, i.e. patterns that occur in a significant number of records. Once such patterns are determined, the actual association rules can be derived in the form of logical implications: X ⇒ Y , which reads whenever X occurs in a transaction (sequence), most likely also Y will occur (later). The computationally intensive part is the determination of frequent patterns, more precisely of frequent itemsets for association rules and frequent sequences for sequential rules.

1.4. Association Rules Mining

1.4.1

9

Frequent Itemsets Mining

The Frequent Itemsets Mining (FIM ) problem consists in the discovery of subsets that are common at least to a user-defined number of input set. Figure 1.3 shows a small dataset related to the previous MBA example. There are eight transactions, each containing a variable number of distinct items. If the user chosen minimum support is three, then the pair ”scanner and speaker” is a frequent pattern, whereas ”scanner and telephone” is not a frequent one. Obviously, any larger pattern containing both a scanner and a telephone cannot be frequent. This fact is known as apriori principle and, expressed in a more formal way, states that a pattern can be frequent only if all its subsets are frequent too.

Figure 1.3: Transaction dataset.

The computational complexity of the FIM problem derives from the exponential size of its search space P(M ), i.e. the power set of M , where M is the set of items contained in the various transactions of 12/02/2002 a dataset D. In the example in Figure 1.3, 10/01/2002 23/12/2002 there are 8 distinct items and the larger transaction contains four items, this lead to
P4 8 k=1 k = 162 possible patterns to examine, considering all transaction of maximal length, and 48 considering the actual transaction lengths. However, the number of distinct patterns is 29 and the number of frequent pattern is even smaller, e.g., there are just 7 items and 4 pairs occurring more than once, but only 4 items contained in more than two transactions. 20/04/2002 10/11/2002 Clearly, the na¨ıve approach consisting in generating all subset for every transaction and updating a set of counters would be extremely inefficient. A way to prune the search space is to consider only those patterns whose subsets are all frequent. The correctness of this approach derives from the apriori principle, which grants that it is impossible for discarded pattern to be frequent. The Apriori algorithm [6] and other derived algorithms16/05/2002 [2, 9, 11, 25, 49, 50, 44, 55, 68] exactly exploits this 10/06/2002

10

1. Introduction

pruning technique.

1.4.2

Frequent Sequence Mining

Sequential pattern mining (FSM) [7] represents an evolution of Frequent Itemsets Mining, allowing also for the discovery of before-after relationships between subsets of input data. The patterns we are looking for are sequences of sets, indicating that the elements of a set occurred at the same time and before the elements contained in the following sets. The ”occurs after” relationship is indicated with an arrow, e.g. {A, B} → {B} indicates an occurrence of both item A and B followed by an occurrence of item B. Clearly, the inclusion relationship is more complex than in case of subsets, so it needs to be defined. Here we informally introduce this concept, which we will define formally in chapter 3. For now, we consider that a sequence pattern Z is supported by an input sequence IS, if Z can be obtained by removing items and sets from IS. As an example the input sequence {A, B} → {C} → {A} supports the sequential patterns {A, B}, {A} → {C}, {A} → {A}, but not the pattern {A, C}, because the occurrence of A and C in the input sequence are not simultaneous. We highlight that the ”occurs after” relationship is satisfied by {A} → {A}, since anything between the two items can be removed. Figure 1.4 shows a small dataset containing just three input sequences, each 10/01/2002

12/02/2002

20/04/2002

16/05/2002

23/12/2002

10/11/2002

10/06/2002

Figure 1.4: Sequence dataset.

associated with a customer according to the above example. For each transaction,

1.4. Association Rules Mining

11

the date is printed, but for the moment, we consider the time just a key for sorting transactions. If we set the minimum support to 50%, we can see that the pattern ”computer and camera followed by a speaker” is frequent and supported by the behavior of two customers. We observe that the apriori principle still holds for sequence patterns. If we define the containment relationship between patterns, analogously to the one defined between patterns and input sequence, we can state that every subsequence of a frequent sequence is frequent. So we are sure that ”computer followed by a speaker” is a frequent pattern without looking at the dataset, because the above-mentioned pattern is frequent, and, at the same time, we know that every pattern containing a ”lamp” is not frequent. The computational complexity of FSM is higher than that of FIM, due to the possible repetitions of items within each pattern. Thus, having a small number of distinct items often does not help, unless the length of input sequences is small too. However, since the apriori principle is still valid, several efficient algorithms for FSM exist, based on the generation of frequent patterns from smaller frequent ones. In several application context it is interesting to exploit the presence of a time dimension in order to obtain a more precise knowledge, and, in some case, to also transform an intractable problem into a tractable one by restricting our attention only to a the cases we are looking for. For example, if data represents the failure in a network infrastructure, when looking for congestion we are interested in short time periods, and the failure of an equipment a day after another one may be not as significant as the same sequence of failures within a few seconds. In this case, an expert of this domain can enforce a constraint on the maximum gap between occurrences of events, thus obtaining a better focus on actually important patterns and a strong reduction in the complexity. In the example in Figure 1.4, if we decide to limit our research to occurrences having a maximum gap smaller than seven month the pattern ”computer followed by a speaker” will be supported just by one customer shopping sequence, since in the first one the gap between the occurrence of the computer and the occurrence of the speaker is too large. Figure 1.5 shows the effect of maximum gap constraint on the support of some of the patterns of the above example: the deleted ones simply disappear, because their occurrences have inadequate gaps. This behavior poses serious problems to some of the most efficient algorithms, as we will explain in chapter 3, since some of their super-pattern may be frequent anyway. It is the case of the pattern ”camera followed by scanner followed by speaker” which has one occurrence with maximum gap equal to seven month even if ”camera followed by speaker” has no occurrence at all.

1.4.3

Taxonomy of Algorithms

The apriori principle states that no superset of an infrequent set can be frequent. This determines a powerful pruning strategy that suggests a level-wise approach to solve both FIM [4] and FSM [7] problems. Apriori is the first of a family of algorithms

12

1. Introduction

Figure 1.5: Effect of maxGap constraint.

based on this method. First, every frequent item is found, and then the focus is on pairs composed of frequent items, and so on. Exploring the search space level-wise grants that every time a new candidate is considered, the support of all its sub patterns is known. An alternative approach is the depth first discovery of frequent patterns: by enforcing the apriori constraint just on some of the sub patterns, the search space is explored deeply. This is usually done in an attempt to preserve locality, examining consecutively similar patterns [25, 24, 52]. In both cases, the support of patterns is computed by updating a counter each time an occurrence is found. Moreover, when all the data fit in main memory, a more efficient approach based on intersection can be devised. Each item x is associated with all the IDs of all transactions where x appears, and the support is equal to the size of the intersection of the two sets. This set of IDs can be obtained using either bitmap [44] or tidlists [68]. In FSM, the technique is similar, but needs a longer description; an exhaustive explanation can be found in chapter 3. The use of intersection in depth-first algorithms is highly efficient, thanks to the availability of partial intersection results related to shorter patterns. For example, if we examine every pattern with a given prefix before moving to a different one, then the list of occurrences associated with that prefix can be reused, with little waste of memory, in the computations related to its descendant. However, the results obtained are unsorted and this can be a problem in case the results are to be merged with other ones, as in case of mining on distributed and streaming data, since we are forced to wait the end of the computation before being able to merge the results. On the other hand, level-wise algorithms pose a strong obstacle to the efficient reuse of partial intersection results due to the limited locality in the search space traversal. When the search space is not partitioned as in the depthfirst algorithm, it is impossible to exploit the partial intersection results computed

1.5. Contributions

13

Figure 1.6: Taxonomy of algorithms for frequent pattern mining.

at level k − 1 in order to compute the intersections at level k, as partial result can become quickly too large to be maintained in main memory. To the best of our knowledge the only two level-wise algorithms that solved this issue, using a result cache and an efficient partial result reuse, are DCI for FIM and CCSM for FSM. DCI was introduced in [44] and extended in [43] with an efficient support inference optimization, whereas CCSM was introduced in [47]. Since these algorithms grant some ordering on the results, they have been chosen as the basic building block of our distributed and streaming algorithms in the second part of this thesis (in the future work chapter for the part concerning FSM), since they make an heavy use of result merging. Figure 1.6 summarizes this taxonomy of FIM and FSM algorithms.

1.5

Contributions

In this thesis, we present original contributions in three related area: frequent sequence mining with gap constraints, approximate mining of frequent patterns on distributed datasets and approximate mining of frequent pattern on streaming data. The original contribution in the sequence mining field is CCSM, a novel algorithm for the discovery of frequent sequence patterns in collection of list of temporally annotated sets, with constraints on the maximum gap between the occurrences of two part of the sequence (maxGap). The proposed method consists in choosing an ordering that improve the locality and reduces the number of test on pattern support when the maxGap constraint is enforced, combined with an effective caching policy of intermediate results. This work has been published in [46, 47]. Another original contribution, this one on approximate distributed frequent itemset mining, deals with homogeneous distributed datasets: several entities cooperate, and each one has its own dataset with exclusive access. The two proposed algo-

14

1. Introduction

rithms [59, 61], allow for obtaining a good approximate solution, and need just one synchronization in one case and none in the other. In APRed , the algorithm proposed in [59], each node begins the computation with a reduced support threshold. After a first phase, needed to understand the peculiarities of the dataset, the minimum support is increased again to an intermediate value chosen according to the behavior of pattern computed during the first phase. Thereafter each node can continue independently and send, at the end of the computation, the result to the master, which reconstructs an approximation of the set of all global frequent patterns. The goal of the support reduction is to force infrequent patterns to be revealed in partitions where they have nearly frequent support. The results obtained by this method are close to the exact ones for several real-world datasets originated by shopping chart and web navigation. To the best of our knowledge, this is the first algorithm for approximate distributed FIM based on an adaptive support reduction scheme. Similar accuracy in the results, but with higher performance thanks to the asynchronous behavior and no support reduction, are achieved by the APInterp algorithm that we have introduced in [61]. It is based on the interpolation of unknown pattern supports, based on the knowledge acquired from the other partitions. The absence of synchronizations, and of any two-way communication between the master and the worker nodes, makes APInterp suitable for streaming data, considering each new incoming block of data as a partition, and the rest of data as another one. In this way the merge and interpolate task can be applied repeatedly. This is the basic idea of APStream , the algorithm we presented in [60]. In our tests on real world datasets, the results are similar to the exact ones, and the algorithm processes the stream in linear time. The described interpolation framework can be easily extended to distributed stream and to the FSM problem, using the CCSM algorithm locally. A more challenging extension, due to the subsumption-related result merging issues, concerns the approximate distributed computation of Frequent Closed Itemsets (FCI), described in our preliminary work [32]. Furthermore, the heuristic used in interpolation can be easily substituted with another one, better fitted to the particular targeted application. However even the very simple and generic one used in our tests gives good results. To the best of our knowledge, the AP method is the first distributed approach that requires just one way communications (i.e., with global pruning optimization disabled, the worker nodes use only local information), tries to interpolate the missing supports by exploiting the available knowledge and is suitable to both distributed and stream settings.

1.6

Thesis overview

This thesis is divided into self-contained chapters. Each chapter begins with as a short overview containing an informal introduction to the subject and a description

1.6. Thesis overview

15

of the scope of the chapter. The first section in most chapters is usually a more formal introduction to the problem, with definitions and references to related works. When other algorithms are used, either to describe the proposal contained in the core of the chapter or its improvements in relation to the state of the art, these algorithms are described immediately after the introduction. The core part of the chapter contains an in depth description of the proposed method, followed by a discussion on its pro and cons, and the descriptions of the experimental setup and results. For the sake of readability, since parts of the citations are common to more chapters, the references are listed at the end of the thesis. For the same reason the measures used for evaluating the approximation of the solutions are described in and appendix. The first part of the thesis is made of two chapters that deal with algorithms that we will use in the following chapters about distributed and streaming data mining, as previously explained in the section about FIM and FSM algorithm taxonomy. The first chapter introduces the frequent itemset mining problem and describes DCI [44], a state of the art algorithm for frequent itemset mining that we will use extensively in the rest of the thesis. The second chapter describes CCSM, a new algorithm for gap constrained sequence mining that we presented in [47]. In the second part of the thesis, the third chapter deals with approximate frequent itemset mining in homogeneous distributed datasets, and describes our two novel approximate algorithms APRed and APInterp , based on support reduction and interpolation. The fourth chapter extends the support interpolation method, introduced in the previous chapter, to streaming data [60]. Finally, the last chapter describes some future works and draws some conclusion. In particular, we describe how to extend the proposed interpolation framework in order to deal with frequent sequences, using CCSM in local computation. Moreover, we discern how to combine the APInterp and APStream in an algorithm for the discovery of frequent itemset on distributed data streams.

16

1. Introduction

I First Part

2 Frequent Itemset Mining Each data mining task has its peculiarities and issues when dealing with evolving and distributed data, as we have briefly outlined in the introduction. A more detailed analysis requires focusing on a particular task. In this thesis, we have decided to analyze in detail this problem by discussing Association Rules Mining (ARM) and Sequential Association Rules Mining (SARM), two of the most popular DM task. The crucial steps in ARM, and by far the most computationally challenging, is the extraction of frequent subsets from an input database of sets of distinct items, also known as Frequent Itemset Mining (FIM). In case the datasets is referred to the activities of a shop, and data are sale transactions composed of several items, the goal of FIM is to find the sets of items that are bought together, at least, in a user specified number of transactions. The challenges in FIM derive from the large size of its search space, which, in the worst case, corresponds to the power set of the set of items, and thus is exponential in the number of distinct items. Restricting as much as possible this space and efficiently performing computations on the remaining part are key issues for FIM algorithms. This chapter formally introduces the itemset mining problem and describes DCI (Direct Count and Intersect), a hybrid level-wise algorithm, which dynamically adapts its search strategy to the characteristics of the dataset and to the evolution of the computation. This algorithm was introduced in [44], and extended in [43] with an efficient, key pattern based, support inference method. With respect to the Frequent Pattern Mining algorithms taxonomy, presented in the introduction, DCI is a level-wise algorithm, able to ensure an ordering of the results, and use an efficient hybrid counting strategy, switching to an in-core intersection based support computation as soon as there is enough memory available for distributed and stream settings. DCI has been chosen as the building block for our approximate algorithms due to its efficiency, and the results ordering, which is particularly important when merging different result sets. Moreover, DCI exactly knows the exact amount of memory needed for the whole intersection phase before starting it, and this has been exploited in APStream , our stream algorithm, for dynamically choosing the size of the block of transactions to process at once.

20

2.1

2. Frequent Itemset Mining

The problem

A dataset D is a collection of subsets of a set of items I = it1 , . . . , itm . Each element of D is called a transaction. A pattern x is frequent in dataset D, with respect to a minimum support minsup, if its support is greater than σmin = minsup · |D|, i.e., the pattern occurs in at least σmin transactions, where |D| is the number of transactions in D. A k-pattern isSa pattern composed of k items, Fk is the set of all frequent k-patterns, and F = i Fi is the set of all frequent patterns. F1 is also called the set of frequent items. The computational complexity of the FIM problem derives from the exponential size of its search space P(M ), i.e., the power set of M , where M is the set of items contained in the various transactions of D.

2.1.1

Related works

A way to prune the search space P(M ), first introduced in the Apriori [6] algorithm, is to restrict the search to itemsets whose subsets are all frequent. Apriori is a levelwise algorithm, since it examines the k-patterns only when all the frequent patterns of length k − 1 have been discovered. At each iteration k, a set of potentially frequent patterns, having all of their subset frequent, are generated starting from the previous level results. Then the dataset is read sequentially, and the counters associated with each candidate are updated according to the occurrences founds. After the database scan, only the candidates having a support greater than the threshold are inserted in the result set and used for generating the next iteration candidates. Several other algorithms based on the apriori principle have been proposed. Some use the same level wise approach, but introduce efficient optimizations, like a hybrid count/intersection support computation [44] or the reduction of the number of candidates using a hash based technique [49]. Others use a depth-first approach, either class based [68] or projection based [2, 25]. Others again, use completely different approaches, based on multiple independent computations on smaller part of the dataset, like [55, 50]. Related research topics are the discovery of maximal and closed frequent itemsets. The first ones are those frequent itemsets that are not included in any other larger frequent itemset. As an example, consider the FIM result set F = {{A} : 4, {B} : 4, {C} : 3, {A, B} : 4, {A, C} : 3, {B, C} : 3, {A, B, C} : 3}, where the notation set : count indicates frequent itemsets along with their supports. In this case there is only one maximal frequent pattern Fmax = {{A, B, C} : 3}, since the other itemsets are included in it. Clearly the algorithms that are able to mine directly the set of maximal pattern, like [9, 3, 11], are faster and produce a more compact output than FIM algorithms. Unfortunately, the information contained in the result set are not the same: in the above example, there is no way to deduce the support of pattern A from Fmax . Frequent closed itemsets are those frequent itemsets that are set-included in any larger frequent itemset having the same support. The group of patterns subsumed by the same itemset appears exactly in the

2.2. DCI

21

same set of transactions, and forms a class of equivalence, whose representative element is the largest. Considering again the previous example, the patterns {A} and {B} are subsumed by the pattern {A, B} whereas the patterns {C}, {A, C} and {B, C} are subsumed by {A, B, C}. Thus, the set of frequent closed itemsets is Fclosed = {{A, B} : 4, {A, B, C} : 3}. Note that in this case the support of any frequent itemset can be deduced as the support of its smaller superset contained in the result, thus the {A, C} pattern support equal to 3, i.e., it has the same support than the pattern {A, B, C}.

2.2

DCI

The approximate algorithms that we will propose in the second part of the thesis for distributed and stream data are build on traditional FPM algorithms, used for local computations. The partial ordering of results, the foreseeable resource usage, and the ability to recompute quickly a pattern support using the in-core vertical bitmap, made DCI our algorithm of choice. DCI is a multi-strategy algorithm that runs in two phases, both level-wise. During its initial count-based phase, DCI exploits an out-of-core horizontal database, with variable-length records. At the beginning of each iteration k, a set Ck of k-candidates is generated, based on the frequent patterns contained in Fk−1 , then their number of occurrences is verified during a database scan. At the end of the scan, the itemsets in Ck having a support greater than the threshold σmin are inserted into Fk . As execution progress, the dataset size is reduced by removing transactions and items no longer needed for computation using a technique inspired by DHP [49]. As soon as the pruned dataset becomes small enough to fit in memory, DCI adaptively changes its behavior. It builds a vertical layout database in-core, and starts adopting an intersection-based approach to determine frequent sets. During this second phase DCI uses intersections to check the support of kcandidates, generated on the fly by composing all the pairs of (k − 1)-itemsets that are included in Fk−1 and share a common (k − 2)-prefix. When a candidate is found to be frequent, it is inserted into Fk . In order to ensure high spatial and temporal locality, each Fi is maintained lexicographically ordered. This grants that (k −1)-patterns sharing a common prefix are stored contiguously in Fk−1 and, at the same time, the candidates are considered in lexicographical order, thus granting the ordering of the result. Furthermore, this allows accessing previous iteration results from disk in a nearly sequential way and storing immediately each pattern as soon as it is discovered to be frequent. DCI uses several optimization techniques, such as support counting inference based on key patterns [43] and heuristics to dynamically adapt to both dense and sparse datasets. Here, however we will put our attention only on candidate generation and the counting/intersection phases. Also in the pseudo-code, contained in algorithm 1, the code part related to optimizations has been removed.

22

2.2.1

2. Frequent Itemset Mining

Candidate generation

Candidates are generated in both phases, even if at different times. In the countbased phase, all the candidates are generated at the beginning of each iteration and then their supports are verified, whereas in the intersection-based one, the candidates are generated and their supports are checked on the fly. Another important difference concerns the memory usage: during the first phase the candidates and the results are maintained in memory and the dataset is on disk, whereas during the second phase the candidates are generated on the fly, the result are immediately offloaded to disk, and the dataset is kept in main memory. The generation of candidates of length k is based on the composition of patterns of k − 1 items sharing the same k − 2 long prefix. For example, if F2 = {A, B}, {A, C}, {A, D}, {B, C} is the set of frequent 2-patterns, then the set of candidates for the 3rd iteration will be C3 = {A, B, C}, {A, B, D}. DCI organizes itemsets of length k in a compressed data structure, optimized for spatial locality and fast access to groups of candidates sharing the same prefix, taking advantage of lexicographical ordering. A first array contains the k − 1 prefix and a second one contains an index to the contiguous block of item suffixes contained in the third array. Figure 2.1 shows the usage of these arrays. The patterns {A, B, D, M } and {A, B, D, I} are represented by the second prefix followed by the suffixes in positions from 7 to 8, i.e., from the index position to the position before the one associated to the next prefix. Generating the candidates using this data structure is straightforward, and simply consists of the generation of all the pairs for each block of suffixes. E.g. for the block corresponding to the prefix {A, B, C}, {A, B, D, G} is inserted in candidate prefixes, with suffixes H,I and L, followed by {A, B, D, H} with suffixes I and L, followed {A, B, D, I} with suffix L. Not every generated candidate obeys the apriori principle, so we can observe that the candidate pattern {A, B, D}, in the first example, cannot be frequent, since its subpattern {B, D} is not frequent. When the candidates are stored in memory, during the counting-based phase, the apriori principle is enforced before inserting candidates into candidate set. On the other hand, checking the presence of every subset has a cost, which increases as the patterns get longer. If we also consider that the relevant subpatterns are not in any particular order, this disrupts both spatial and temporal locality in the access to the previous iteration results (Fk−1 ). For this reason, and the low cost and high locality of intersection-based support checking, the authors has decided to limit the candidate pruning step to the count-based phase.

2.2.2

Counting phase

In the first iteration, similarly to all FSC algorithms, DCI exploits a vector of counters. In subsequent iterations, it uses a Direct Count technique, introduced by the same authors in [42]. The goal of this technique is to make the access to the counters associated with candidates as fast as possible. So, instead of using a hash tree,

2.2. DCI

23

prefix

index

a b c a b d b d f

3 7 9

a a a a a a a a b b

b b b b b b b b d d

c c c c d d d d f f

d e f g h i l m i n

suffix d e f g h i l m i n

0 1 2 3 4 5 6 7 8 9

Compressed Memory = 9 + 3 + 10 = 21

Non−Compressed Memory = 4 x 10 = 40

Figure 2.1: Compressed data structure used for itemset collection can also improve candidate generation. This figure originally appeared in [43]. or others complex data structures, it extends the approach used for items. When k = 2, each pair of (frequent) items is associated with a counter in an array through an order preserving perfect hash function. Since the order in pairs of items is not significant, and
the elements of a pair are distinct, the number of counters needed is m(m−1) = m2 , where m is the number of frequent items. 2 When k > 2, using direct access to counters would require a large amount of memory. In this case, the direct access prefix table contains a pointer to a contiguous block of ordered candidates sharing
the same
2-prefix. Note that the number of mk m locations in the prefix table is 2 6 2 , where mk is the number of distinct items in the dataset during iteration k, which is less than or equal to m, thanks to pruning. Indeed, during the k th count-based iteration, DCI removes from each generic transaction t every item that is not contained in at least k − 1 frequent itemsets of Fk−1 and k candidate itemsets of Ck . Clearly, as the execution progress, the size of the dataset actually used in computations decrease and, thanks to pruning, the whole dataset will rapidly shrink enough to fit in main memory, for the final intersection-based phase. Even with large datasets and limited memory, this often happen after 3 or 4 iterations, thus limiting the drawbacks of the count-based phase, which becomes less efficient as k increases.

24

2. Frequent Itemset Mining

Algorithm 1: DCI input : D, minsup 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

// find the frequent itemsets; F1 ← f irst scan(D, minsup); //second and following scans on a temporary db D0 ; F2 ← second scan(D0 , minsup); k ← 2; while D0 .vertical size() > memory available() do k ← k + 1; Fk ← DCI count(D0 , minsup, k); end k ← k + 1; // count-based iteration and create vertical database VD ; Fk ← DCI count(D0 , VD, minsup, k); while Fk 6= ∅ do k ← k + 1; Fk ← DCI intersect(VD, minsup, k) ; end

2.2.3

Intersection phase

The intersection-based phase uses a vertical database, in which each item α is paired with a set of transactions tids(α) containing α, different from the horizontal one used before, in which a set of items is associated with each transaction. Since a transaction t supports pattern x iff x ⊆ t, the set of transactions supporting x can be obtained by intersecting the sets of transactions (tidlist) associated with each items in x. Thus the support σ(x) of a pattern x will be \ σ(x) = tids(α) α∈x In DCI the sets of transactions are represented as bit-vectors, where the ith bit is equal to 1 when the ith transactions contains the item and is equal to 0 otherwise. This representation allows for efficient intersections-based on the bitwise and operator. The memory necessary to contain this bitmap-based vertical representation is mk ·nk bits, where mk and nk are respectively the numbers of items and transactions in the pruned database used at iteration k. As soon as this amount is less than the available memory, the vertical dataset representation can be built on the fly in main memory in order to begin the intersection based phase of DCI. During this phase, the candidates are generated on the fly in lexicographical order, and their supports are checked using tidlist intersections. The above-described method for support computation is indicated as k-way intersection. The k bit-vectors

2.3. Conclusions

25

associated with items contained in a k-pattern are and-intersected and the support is obtained as the number of 1’s present in the resulting bit-vector. If this value is greater than the support threshold σmin , then the candidate is inserted into Fk . Since the candidates are generated on the fly, the set of candidates needs no longer to be maintained. Moreover, both Fk−1 and Fk can be kept on disk. Indeed, Fk−1 is lexicographically ordered and can be loaded in block having the same (k −2)prefix, and, thanks to the order of candidate generation, appending frequent patterns at the end of Fk preserves the lexicographic order. The set intersection is a commutative and associative operation, thus the operands can be intersected in any order and grouped in any way. A possible method is intersecting the tidlist of items pair wise, starting from the beginning, i.e., the first with the second, the result with the third, the result with the fourth and so on. Since the candidates are lexicographically ordered, consecutive candidates are likely to share a prefix of some length. Hence, the intersections related to this prefix are pointlessly repeated for each candidate. In order to exploit this locality, DCI uses an effective cache containing the intermediate results of intersections. When the support of a candidate c is checked immediately after the candidate c0 , the tidlist associated with their common prefix can be obtained directly from the cache. 1 2 3 4

Cached Pattern {A} {A, B} {A, B, C} {A, B, C, D}

Cached tidList tids(A) tids(A) ∩ tids(B) (tids(A) ∩ tids(B)) ∩ tids(C) ((tids(A) ∩ tids(B)) ∩ tids(C)) ∩ tids(D)

Figure 2.2: Example of cache usage.

For example, after the computation of the support of the itemset {A, B, C, D}, the tidlists associated with all of its prefix are present in cache, as showed in Figure 2.2. Note that each cache position is obtained from the previous one by intersection with the tidlist of a single item. Hence, if the next candidate pattern is {A, B, C, G}, only the last position of the cache need to be replaced, and this implies just one tidlist intersection, since the tids intersections of {A, B, C} can be retrieved from the third entry of the cache.

2.3

Conclusions

In this chapter, we have described the frequent itemset mining (FIM) problem, the state of the art of FIM algorithms, and DCI, an efficient FIM algorithm, introduced by Orlando et al. in [44]. We will use DCI in the second part of the thesis as a building block for our approximate algorithm for distributed and stream data. DCI

26

2. Frequent Itemset Mining

has been chosen among the other FIM algorithms thanks to its efficiency and the result ordering, which is particularly important when merging different result sets. Moreover, we can exactly predict the exact amount of memory needed by DCI for the whole intersection phase before starting it, and this has been exploited in APStream , our stream algorithm, for dynamically choosing the size of the block of transactions to process at the same time.

3 Frequent Sequence Mining The previous chapter has introduced the Frequent Itemset Mining (FIM) Problem, the most computationally challenging part of Association Rules Mining. This chapter deals with Sequential Association Rules Mining (SARM) and in particular with its Frequent Sequence Mining (FSM) phase. In this thesis work we have decided to focus on this two popular data mining tasks, with particular regard to the issues related to distributed and stream settings, and the usage of approximate algorithms in order to overcome these problems. The algorithm proposed in this chapter, can be used as a building block for the Frequent Sequence version of our approximate distributed and stream algorithms described in the second part of this thesis, thanks to its efficiency, and results ordering, which is particularly important when merging different result sets. The frequent sequence mining (FSM) problem consists in finding frequent sequential patterns in a database of time-stamped events. Going on with the supermarket example, market baskets are linked to a time-line and no longer anonymous. An important extension to the base FSM problem is the introduction of time constraints. For example, several application domains require limiting the maximum temporal gap between events occurring in the input sequences. However pushing down this constraint is critical for most sequence mining algorithms. This chapter formally introduces the sequence mining problem and proposes CCSM (Cache-based Constrained Sequence Miner), a new level-wise algorithm that overcomes the troubles usually related to this kind of constraint. CCSM adopts an innovative approach based on k-way intersections of idlists to compute the support of candidate sequences. Our k-way intersection method is enhanced by the use of an effective cache that stores intermediate idlists for future reuse inspired by DCI [44] (see previous chapter). The reuse of intermediate results entails a surprising reduction in the actual number of join operations performed on idlists. CCSM has been experimentally compared with cSPADE [69], a state of the art algorithm, on several synthetically generated datasets, obtaining better or similar results in most cases. Since some concept introduced in GSP [62] and SPADE [70] algorithm are used to explain the CCSM algorithm, a quick description of these two follows the problem description. Other related works are referred at the end of the chapter.

28

3.1

3. Frequent Sequence Mining

Introduction

The problem of mining frequent sequential patterns was introduced by Agarwal and Srikant in [7]. In a subsequent work, the same authors discussed the introduction of constraints on the mined sequences, and proposed GSP [62], a new algorithm dealing with them. In the last years, many innovative algorithms were presented for solving the same problem, also under different user-provided constraints [69, 70, 53, 20, 52, 8]. We can think of the problem of mining Frequent Sequence Mining (FSM) as a generalization of Frequent Itemset Mining (FIM) to temporal databases. FIM algorithms aims to find patterns (itemsets) occurring with a given minimum support within a transactional database D, whose transactions correspond to collections of items. A pattern is frequent if its support is greater than (or equal to) a given threshold s%, i.e. if it is set-included in at least s%·|D| input transactions, where |D| is the total number of transactions in D. An input database D for the FSM problem is instead composed of a collection of sequences. Each sequence corresponds to a temporally ordered list of events, where each event is a collection of items (itemset) occurring simultaneously. The temporal ordering among the events is induced from the absolute timestamps associated with the events. A sequential pattern is frequent if its support is greater than (or equal to) a given threshold s%, i.e. if it is ”contained” in (or it is a subsequence of) at least s% · |D| input sequences, where |D| is the number of sequences included in D. To make more intuitive both problem formulations, we may consider them within the application context of the market basket analysis (MBA). In this context, each transaction (itemset) occurring in a database D of the FIM problem corresponds to the collection of items purchased by a customer during a single visit to the market. The FIM problem for MBA consists in finding frequent associations among the items purchased by customers. In the general case, we are thus not interested in the timestamp of each purchased basket, or in the identity of its customer, so the input database does not need to store such information. Conversely, FSM problem for MBA consists in predicting customer behaviors on the basis of their past purchases. Thus, D has also to include information about timestamp and customer identity of each basket. The sequences of events included in D correspond to sequences of ”baskets” (transactions) purchased by the same customer during distinct visits to the market, and the items of a sequential pattern can span a set of subsequent transactions belonging to the same customer. Thus, while the FIM problem is interested in finding intra-transaction patterns, the FSM problem determines intertransaction sequential patterns. Due to the similarities between the FIM and FSM problems, several FIM algorithms have been adapted for mining frequent sequential patterns as well. Like FIM algorithms, also FSM ones can adopt either a count-based or intersection-based approach for determining the support of frequent patterns. The GSP algorithm, which is derived from Apriori [7], adopts a count-based approach, together with a

3.1. Introduction

29

level-wise visit (Breadth-First) of the search space. At each iteration k, a set of candidate k-sequences (sequences of length k) is generated, and the dataset, stored in horizontal form, is scanned to count how many times each candidate is contained within each input sequences. The other approach, i.e. the intersection-based one, relies on a vertical-layout database, where for each item X appearing in the various input sequences we store an idlist L(X). The idlist contains information about the identifiers of the input sequences (sid ) that include X, and the timestamps (eid ) associated with each occurrence of X. Idlists are thus composed of pairs (sid, eid ), and are considerably more complex than the lists of transaction identifiers (tidlists) exploited by intersection-based FIM algorithms. Using an intersectionbased method, the support of a candidate is determined by joining lists. In the FIM case, tidlist joining is done by means of simple set-intersection operations. Conversely, idlist joining in FSM intersection-based algorithms exploits a more complex temporal join operation. Zaki’s SPADE algorithm [70] is the best representative of such intersection-based FSM algorithms. Several real applications of FSM enforce specific constraints on the type of sequences extracted [62, 53]. For example, we might be interested in finding frequent sequences of purchase events which contain a given subsequence (super pattern constraint), or where the average price of items purchased is over a given threshold (aggregate constraint), or where the temporal intervals between each pair of consecutive purchases is below a given threshold (maxGap constraint). Obviously, we could solve this problem with a post-processing phase: first, we extract from the database all the frequent sequences, and then we filter them on the basis of the posed constraints. Unfortunately, when the constraint is not on the sequence itself but on its occurrences (as in the case of the maxGap constraint), sequence filtering requires an additional scan of the database to verify whether a given frequent pattern has still a minimum support under the constraint. In general, FSM algorithms that directly deal with user-provided constraints during the mining process are much more efficient, since constraints may involve an effective prune of candidates, thus resulting in a strong reduction of the computational cost. Unfortunately, the inclusion of some support-related constraints may require large modifications in the code of an unconstrained FSM algorithm. For example, the introduction of the maxGap constraint in the SPADE algorithm, gave rise to cSPADE, a very different algorithm [69]. All the FSM algorithms rely on the anti-monotonic property of sequence frequency: every subsequence of a frequent sequence is frequent as well. More precisely most algorithms rely on a weaker property, restricted to a well-characterized part of subsets. This property is used to generate candidate k-sequence from frequent (k − 1)-sequences. When an intersection-based approach is adopted, we can determine the support of any k-sequence by means of join operations performed [55] on the idlist associated with its subsequences. As a limit case, we could compute the support of a sequence by joining the atomic idlists associated with the single items included in the sequence, i.e., through a k-way join operation [44]. More efficiently,

30

3. Frequent Sequence Mining

we could compute the support of a sequence by joining the idlists associated with two generating (k − 1)-subsequences, i.e., through a 2-way join operation. SPADE [70] just adopts this 2-way intersection method, and computes the support of a ksequence by joining two of its (k − 1)-subsequences that share a common suffix. Unfortunately, the adoption of 2-way intersections requires maintaining the idlists of all the (k − 1)-subsequences computed during the previous iteration. To limit memory requirement, SPADE subdivides the search space into small, manageable chunks. This is accomplished by exploiting suffix-based equivalence classes: two k-sequences are in the same class only if they share a common (k − 1)-suffix. Since all the generating subsequences of a given sequence belong to the same equivalence class, equivalence classes are used to partition the search space in a way that allow each class to be processed independently in memory. Unfortunately, the efficient method used by SPADE to generate candidates and join their idlists, cannot be exploited when a maximum gap constraint is considered. Therefore, cSPADE is forced to adopt a different and much more expensive way to generate sequences and join idlists, also maintaining in memory F2 , the set of frequent 2-sequences. This chapter discuss CCSM (Cache-based Constrained Sequence Miner), a new level-wise intersection-based FSM algorithm, dealing with the challenging maximum gap constraint. The main innovation of CCSM is the adoption of k-way intersections to compute the support of candidate sequences. Our k-way intersection method is enhanced by the use of an effective cache, which store intermediate idlists. The idlist reuse allowed by our cache entails a surprising reduction in the actual number of join operations performed, so that the number of joins performed by CCSM approaches the number of joins performed when a pure 2-way intersection method is adopted, but require much less memory. In this context, it becomes interesting to compare the performances of CCSM with the ones achieved by cSPADE when a maximum gap constraint is enforced. The rest of the chapter is organized as follows. Section 3.2 formally defines the FSM problem, while Section 3.5.2 describes the CCSM algorithm. In Section 3.5.3, there are some experimental results and a discussion about them. Finally, Section 5.4 presents some concluding remarks.

3.2 3.2.1

Sequential patterns mining Problem statement

Definition 1. (Sequence of events) Let I = {i1 , ..., im } be a set of m distinct items. An event (itemset) is a non-empty subset of I. A sequence is a temporally ordered list of events. We denote an event as (j1 , . . . , jm ) and a sequence as (α1 → . . . → αk ), where each ji is an item and each αi is an event (ji ∈ I and αi ⊆ I). The symbol → denotes a happens-after relationship. The items that appear together in an event happen simultaneously. The length |x| of a sequence x is the number

3.2. Sequential patterns mining

of items contained in the sequence (|x| = k-sequence.

31

P

|αi |). A sequence of length k is called a

Even if an event represents a set of items occurring simultaneously, it is convenient to assume that there exists an ordering relationship R among them. Such order makes unequivocal the way in which a sequence is written, e.g., we cannot write BA → DBF since the correct way is AB → BDF . This allows us to say, without ambiguity, that the sequence A → BD is a prefix of A → BDF → A, while DF → A is a suffix. A prefix/suffix of a given sequence α are particular subsequences of α (see Def. below). Definition 2. (Subsequence) A sequence α = (α1 → . . . →αk ) is contained in a sequence β = (β1 →...→βm ) (denoted as αβ), if there exist integers 1≤i1 minsup · |D|} ; Send R0 to all the slaves ; Compute r00 = max{r ∈ [0.5, 1]|Sim(R0 , R00 (r)) >Pγ} where γ is a user provided similarity threshold, R00 (r) = {x ∈ R0 | i σir (x) > minsup · |D|}, σir (x) = [[σi (x) > r · minsup · |Di |]] · σi (x) ; Send minsup00 = r00 · minsup to all the slaves ; Receive local results R00 i from S P all the slaves ; 0 00 Return R ∪ {x ∈ i R i | i σi (x) > minsup · |D|} ;

It is worth noting that the master discards already computed local results. In particular, the presence of patterns in R0 i (see point 2) and R00 i (see point 7) that do not result globally frequent, causes a waste of resources. This is a negative side effect and, in the experimental section, we will use this quantity as a measure of the efficiency of the proposed algorithm, in order to asses the impact on performance of lowering the minimum support threshold. We will see, however, that by exploiting

4.2. Approximated distributed frequent itemset mining

59

the approximate knowledge R0 of F1 ∪ F2 for candidate pruning we can effectively reduce this drawback.

4.2.4

The APInterp algorithm

APInterp , the second distributed algorithm we propose in this chapter, tries to overcome some of the problems encountered by APRed and Distributed One-pass Partition when the data skew between the data partitions is high. The more evident is that several false positives could be generated, increasing the resource utilization and the execution time of both Distributed Partition and Distributed One-pass Partition. As APRed , also APInterp addresses this issue by means of global pruning based on partial knowledge of F2 : each locally frequent pattern that contains a globally non-frequent 2-pattern will be locally removed from the set of frequents patterns before sending it to the master and performing next candidate generation. Moreover this skew might cause a globally frequent pattern x to result infrequent on a given partition Di only. In other words, since σi (x) < minsup · |Di |, x will not be returned as a frequent pattern by the ith slave. As a consequence, the master of Distributed One-pass Partition cannot count on the knowledge of σi (x), and thus cannot exactly compute the global support of x. Unfortunately, in Distributed One-passP Partition the master might also deduce that x is not globally frequent, because j,j6=i σj (x) < minsup · |D|. As explained in the previous section, APRed uses support reduction in order to limit this issue. Unfortunately, this method exposes APRed to the combinatorial explosion of the intermediate results, in case the reduced minsup is too small for the processed dataset. APInterp , instead, allows the master to infer an approximate value for this unknown σi (x) by exploiting an interpolation method. The master bases its interpolation reasoning on the knowledge of: • the exact support of each single item on all the partitions, and • the average reduction of the support count of pattern x on all the partitions where x resulted actually frequent (and thus returned to the master by the slave), with respect to the support of the least frequent item contained in x: σj (x) j∈f part(x) ( minitem∈x (σj (item)) )

P avg reduct(x) =

|f part(x)|

where fpart(x) corresponds to the set of data partitions Dj where x actually resulted frequent, i.e. where σj (x) ≥ minsup · |Dj |. The master can thus deduce the unknown support σi (x) on the basis of avg reduct(x) as follows: σi (x)interp = min (σi (item) ∗ avg reduct(x)) item∈x

60

4. Distributed datasets

It is worth remarking that this method works if the support of larger itemsets decrease similarly in all the dataset partitions, so that an average reduction factor (different for each pattern) can be used to interpolate unknown values. Finally note that, as regards the interpolated value above, we expect that the following inequalities hold: σi (x)interp < minsup · |Di | (4.1) So, if we obtain that σi (x)interp ≥ minsup · |Di |, this interpolated result cannot be accepted. If it was true, the exact value σi (x) should have already been returned by the ith slave. Hence, in those few cases where the inequality (4.1) does not hold, the interpolated value returned will be: σi (x)interp = (minsup · |Di |) − 1 The proposed interpolation schema yields a better approximation of exact results than Distributed One-pass Partition. The support values computed by the latter algorithm are, in fact, always equal to the lower bounds of the intervals containing the exact support of any particular pattern. Hence any kind of interpolation producing an approximate result set, whose supports are between the interval bounds, should be, generally, more accurate than peeking always its lower bound. Obviously several other way of computing a support interpolation could be devised. Some are really simple as the average of the bounds while others are complex as counting inference, used in a different context in [43]. We chose this particular kind of interpolation because it is simple to calculate, since it is based on data that we already maintain for other purposes, and it is aware of the data partitioning enough to allow for accurate handling of datasets characterized by heavy data-skew on item distributions. We can finally introduce the pseudo-code of APInterp (algorithms 4 and 5). As in Distributed Partition, we have a master and several slaves, each in charge of a horizontal partition Di of the original dataset. The slaves send information to the master about the counts of single items and locally frequent 2-itemsets. Upon reception of all local results (synchronization), the master communicates to the slaves an approximate global knowledge on F 0 2 , used by the slaves to prune candidates for the rest of the mining process. Finally, once received information about all locally frequent patterns, the master exploits the interpolation method sketched above for inferring unknown support counts. Note that when a pattern is locally frequent in all the partitions, the master is able to determine exactly its support. Otherwise, an approximate inferred support value is produced, along with an upper bound and a lower bound for that support. In the pseudo-code Fki denotes the set of frequent k-patterns in partition i (or globally when i is not present), F 0 k indicate an approximation of Fk and Single Countsi1 is the support of all 1-patterns in partition i. For the sake of simplicity, some detail of the algorithm has been altered in the pseudo-code. In particular, points 4 and 5 of the slave pseudo-code are an over-

4.2. Approximated distributed frequent itemset mining

Algorithm 4: APInterp - Slave i 1 2 3 4 5

Compute local Single Countsi1 and F2i . ; Send local partial results to the master ; Receive the global approximation F 0 2 of F2 ; Continue computation, by using F 0 2 for pruning candidates ; Send local results to the master. If computation is over, send an empty set ;

Algorithm 5: APInterp - Master 1 2 3

4

5 6

7 8 9

Receive local partial results Single Countsi1 and F2i from all the slaves; Compute the exact F1 , on the basis of the local counts of single items; Compute the approximate P S F 0 2 = {x ∈ i F2i | i counti (x) > minsup · |D|} i where if x ∈ F2 then counti (x) is equal to σi (x), or is equal to σi (x)interp otherwise ; Send F 0 2 to all the slaves ; Receive local results from all the slaves (empty for slaves terminated before the third iteration) ; Compute and return, S for each P k, the approximate F 0 k = {x ∈ i Fki | i counti (x) > minsup · |D|} i where if x ∈ Fk then counti (x) is equal to σi (x), or is equal to σi (x)interp otherwise;

61

62

4. Distributed datasets

simplification of the actual code: pattern are sent, asynchronously, as soon as they are available in order to optimize communication. Each slave terminates when, at iteration k, less than k + 1 pattern are frequent; this is equivalent to checking emptiness of F 0 ik+1 , but more efficient. On the other side, the master continuously collects results from still active slaves and processes them as soon as all expected result sets of the same length arrive.

4.2.5

Experimental evaluation

In the following part of the section, we describes the behavior exhibited by our distributed approximate algorithms in our experiments. We have run the APRed and APInterp algorithms on several datasets using different parameters. The goal of these tests is to understand, how similarities of the results vary as the minimum support and number of partitions change and the scalability. Similarity and Average Support Range. The method we are proposing yields approximate results. In particular APInterp computes pattern supports which may be slightly different from the exact ones, thus the result set may miss some frequent patterns (false negatives) or include some infrequent patterns (false positives). In order to evaluate the accuracy of the results we use a widely used measure of similarity between two pattern sets introduced in [50], and based on support difference. At the same time, we have introduced a novel similarity measure, derived from the previous one and used along with it in order to assess the quality of the algorithm output. To the same end, we use the Average support Range (ASR), an intrinsic measure of the correctness of the approximation introduced in [61]. An extensive description of this measures and a discussion on their use can be found in the appendix A. Experimental environment The experiments were performed on a cluster of seven high-end computers, each equipped with an Intel Xeon 2 GHz, 1 GB of RAM memory and local storage. In all our tests, we mapped a single process (either master or slave) to each node. This system offers communications with good latency (a dedicated Fast Ethernet). However, since APInterp requires just one synchronization, and all communication are pipelined, its communication pattern should be suitable even for a distributed system characterized by a high latency network. Experimental data We performed several tests using datasets from the FIMI’03 contest [1]. We randomly partitioned each dataset and used the resulting partitions as input data for different slaves.

4.2. Approximated distributed frequent itemset mining

63

During the test for APRed , we used two different partitioning, briefly indicated with the suffix P1 and P2 in plot and tables. In doing so, we tried to cover different number of possible cases with respect to partition size and number of partitions. Table 4.2.5 show a list of these datasets along with their cardinality, the number of partitions used in tests, and the minimum and maximum sizes of the partitions. Each dataset is also identified by a short reference code. Table 4.1: Datasets used in APRed experimental evaluation. P1 and P2 in the dataset name refers to different partitioning of the same dataset. Dataset (reference) accidents-P1 (A1) accidents-P2 (A2) kosarak-P1 (K1) kosarak-P2 (K2) mushroom-P1 (M1) mushroom-P2 (M2) retail-P1 (R1) retail-P2 (R2) T10I4D100K-P1(T10-1) T10I4D100K-P2(T10-2) T40I10D100K-P1(T40-1) T40I10D100K-P2(T40-2)

#Trans. /1000 340 340 990 990 8 8 88 88 100 100 100 100

# Part 10 10 20 20 4 10 4 4 10 10 10 10

Part. size /1000 13..56 15..55 11..79 21..78 1..3 0.5..1 14..31 10..31 2..17 8..16 3..19 5..13

In APInterp tests, each dataset was divided in to a number of partitions ranging from 1 to 6, both in partition of similar and significantly different size. The first ones, the balanced partitioned datasets, were used in order to assess speedup for the tests on our parallel test bed. Table 4.2 shows a list of these datasets along with their cardinality and the minimum and maximum sizes of the partitions (for the largest number of partition). Each dataset is also identified by a short code, starting with U in case the sizes of partitions differ significantly. The number of partitions is not reported in this table, since it depends on the number of slaves involved in the specific distributed test. For each dataset, we computed the reference solution using DCI [44], an efficient sequential algorithm for frequent itemset mining (FIM). APRed experimental results First we present the results obtained using APRed , for which we only used the most strict Absolute Similarity measure (α = 1, see appendix A) for accuracy testing. Table 4.3 shows a summary of computation results for all datasets, obtained by using a self-similarity threshold γ = 0.98 to determine minsup0 = r · minsup, where r ∈ [0.5, 1]. We have reported the absolute similarity of approximate results to

64

4. Distributed datasets

Table 4.2: Datasets used in APInterp experimental evaluation. When a datasets is referenced by a keyword prefixed by U (see Reference column), this means that it was partitioned in an unbalanced way, with partitions of significantly different sizes. Dataset accidents-bal accidents-unbal kosarak-bal kosarak-unbal mushroom-bal mushroom-unbal retail-bal retail-unbal pumbs-bal pumbs-unbal pumbs-star-bal pumbs-star-unbal connect-bal

Reference A UA K UK M UM R UR P UP PS UPS C

#Trans. 340183 340183 990002 990002 8124 8124 88162 88162 49046 49046 49046 49046 67557

Part. size 55778..57789 3004..84011 163593..166107 112479..237866 1337..1385 328..1802 14307..14888 6365..23745 8044..8289 1207..12138 8034..8291 3156..12089 11086..11439

exact results, the number of globally frequent patterns and the number of distinct discarded patterns, i.e. the patterns that are locally frequent but are discarded in points 2 and 7 of master pseudo-code because they are not globally frequent. Figure 4.1 shows several plots comparing the self-similarity used during computation, i.e. based on similarity between R0 and R00 , with the exact similarity between the global approximate results and the exact ones for r ∈ [0.5, 1]. If we pick a particular value of r in the plot, corresponding to a value of selfsimilarity γ, we can graphically find the similarity of the whole approximate solution to the exact one when r is used for the second part of the computation. We have found that in sparse datasets the similarity is usually nearly equal to (greater than) self-similarity, so the proposed empirical determination of r should yield good results. Even when the selection is slightly mislead by an excessively good partial result on R00 . This is the case of the Accident P2 dataset. Table 4.3 shows that APRed for this dataset chooses a support reduction factor of 0.95, and the similarity of the final result is 95%, which is a remarkably good result. Nevertheless, in the bottom left plot in Figure 4.1, we can see that by using a slightly smaller reduction factor (0.75), it was possible to boost the similarity of the final result close to 100%. Figure 4.2 shows the number of discarded patterns (points 2 and 7 of master pseudo-code) as a function of r. In order to put in evidence the effectiveness of the pruning based on F 0 1 and F 0 2 , we report curves relative to different types of pruning. Pruning local patterns using an approximate knowledge of F1 and F2 is enough to obtain a good reduction in the number of discarded pattern in most of the sparse datasets.

4.2. Approximated distributed frequent itemset mining

65

Table 4.3: Test results for APRed , obtained using the empirically computed local minimum support (minsup00 = r00 · minsup) for patterns with more than 2 items (for self-similarity threshold γ = 0.98). Dataset A1 A2 K1 K2 M1 M2 R1 R2 T10-1 T10-2 T40-1 T40-2

Min. supp. 40 % 40 % 0.6 % 0.3 % 40 % 40 % 0.2 % 0.2 % 0.2 % 0.2 % 2% 2%

Simil. r00 0.95 0.95 0.85 0.80 0.50 0.50 0.55 0.55 0.60 0.65 0.80 0.85

0.95 0.96 0.97 0.99 0.44 0.07 0.92 0.91 0.93 0.94 0.92 0.96

# Freq 29646 28675 1132 4997 413 399 2675 2682 13205 13173 2293 2293

# Discarded 678289 633908 1968 12379 366 288 7492 7786 31353 17444 19220 18186

The APRed algorithm performed worse on dense datasets, such as Accidents, where too many locally frequent patterns are discarded, and Mushroom, where similarity of approximate results to exact results was really low. Large data skews seem to be a big issues for APRed , since in these cases several frequent patterns are not returned at all (lots of false negatives, and thus small values for both Recall and Similarity). APInterp experimental Results The experiments were run for several minimum support values and for different partitioning on each dataset. In particular, except when showing the effects of varying the minimum support and the number of partitions, we reported results corresponding to three and six partitions and to the two smallest minimum support thresholds used, usually characterized by a difference of about one order of magnitude in execution time. Table 4.4 shows a summary of computation results for all datasets, obtained for three and six partitions using two different minimum support values. The first four columns contain the code of the dataset and the parameters of the test. The next two columns contain the number of frequent patterns contained in the approximate solution and the execution time. The average support range column contains the average distance between the upper and lower bounds for the support of the various patterns, expressed as a percentage of the number of transactions in the dataset (see Definition 14). The following columns show the precision and recall metrics and the number of false positives/negatives. As expected, there are really few false negatives and consequently the value of Recall is close to 100%, but the Precision

66

4. Distributed datasets

Dataset: T10I4D100K P2 Min. supp = 0.2%

Dataset: Kosarak P2 Min. supp = 0.3%

100

100

95 95 90 90 %

%

85 80

85

75 80 70 Self-similarity F1+F2 Similarity

65 0.5

0.55

0.6

0.65

0.7

Self-similarity F1+F2 Similarity

75 0.75 r

0.8

0.85

0.9

0.95

1

0.5

0.55

0.6

0.65

100

99

90

98

80

97

70

96

50

94

40 30

Self-similarity F1+F2 Similarity

92 0.5

0.55

0.6

0.65

0.7

0.8

0.85

0.9

0.95

1

0.85

0.9

0.95

1

Self-similarity F1+F2 Similarity

20 0.75 r

0.8

60

95

93

0.75 r

Dataset: Mushroom P1 Min. supp = 40%

100

%

%

Dataset: Accidents P2 Min. supp = 40%

0.7

0.85

0.9

0.95

1

0.5

0.55

0.6

0.65

0.7

0.75 r

0.8

Figure 4.1: Similarity between the approximate distributed result and the exact one for APRed . The most strict value (α = 1) was used for support difference weight. This means that patterns with different supports are considered as not matching. Selfsimilarity is a measure used for similarity estimation during distributed elaboration, when true results are not available.

is slightly smaller. Unfortunately, since these metrics do not take into account the support, a false positive having true support really close to the threshold has the same weight than one having a very small support. The last columns contain the similarity measure for the approximate results introduced in Definitions 12 and 13. The very high value of the f pSim proves that false positives have a support close to the exact one (but smaller than the exact one, so that they are actually not frequent). This behavior, i.e. a lot of false positives with a value of f pSim close to 100%, is particularly evident for datasets K and UK. Figure 4.3 shows a plot of the fpSim measure obtained for different datasets partitioned among a variable number of slaves. As expected, the similarity is higher when the dataset is partitioned in few partitions. Anyway, in most case there is no significant decrease. We have also compared the similarity of the approximate result obtained using support interpolation to the Distributed One-pass Partition one. The results are shown in Figure 4.4. The proposed heuristic for support interpolation does improve similarity, in particular for small minimum support values. Since no false positives are produced by Distributed One-pass Partition, in this case f pSim would be identical

4.2. Approximated distributed frequent itemset mining

Dataset: T10I4D100K P2 Min. supp = 0.2%

Dataset: Kosarak P2 Min. supp = 0.3% 90

no pruning F1 pruning F1+F2 pruning

4

Discarded local pattern / Frequent pattern

Discarded local pattern / Frequent pattern

4.5

3.5 3 2.5 2 1.5 1

no pruning F1 pruning F1+F2 pruning

80 70 60 50 40 30 20 10 0

0.5

0.55

0.6

0.65

0.7

0.75 r

0.8

0.85

0.9

0.95

1

0.5

0.55

0.6

0.65

Dataset: Accidents P2 Min. supp = 40% 200

no pruning F1 pruning F1+F2 pruning

180

0.7

0.75 r

0.8

0.85

0.9

0.95

1

0.9

0.95

1

Dataset: Mushroom P1 Min. supp = 40% 3500 Discarded local pattern / Frequent pattern

Discarded local pattern / Frequent pattern

67

160 140 120 100 80 60 40 20 0

no pruning F1 pruning F1+F2 pruning

3000 2500 2000 1500 1000 500 0

0.5

0.55

0.6

0.65

0.7

0.75 r

0.8

0.85

0.9

0.95

1

0.5

0.55

0.6

0.65

0.7

0.75 r

0.8

0.85

Figure 4.2: Relative number of distinct locally frequent patterns that are not globally frequent as a function of r for different pruning strategies for APRed . They are discarded at point 2 and 7 of the master pseudo-code. This is a measure of the waste of resources due to both data-skewness and minimum support lowering. Accidents, a dense dataset, causes a lot of trashed locally frequent patterns.

to Sim, thus this measure is plotted just for the APInterp algorithm. Finally, we have verified the speedup of the APInterp algorithm, using only uniformly sized partitions. Figure 4.5 shows the measured speedup when an increasing number of slaves is exploited. Note that when more slaves are used, the dataset has to be partitioned accordingly. The APInterp algorithm performed worse on dense datasets, such as Connect, where too many locally frequent patterns are discarded when we add slaves. On the other hand, in some cases we obtained also superlinear speedups. This could be due to the approximate nature of our algorithm: the support of several patterns could be computed even if some slaves does not participate in the elaboration. Acknowledgment The datasets used during the experimental evaluation are some of those used for the FIMI’03 (Frequent Itemset Mining Implementations) contest [1]. Thanks to the owners of these data and people who made them available in current format. In particular Karolien Geurts [21] for Accidents, Ferenc Bodon for Kosarak, Tom

68

4. Distributed datasets

fpSimilarity(%) 100 99.5 99

%

98.5 98

A (minsupp= 20%) C (minsupp= 20%) K (minsupp= 0.1%) M (minsupp= 5%) P (minsupp= 70%) PS (minsupp= 25%) R (minsupp= 0.05%) UA (minsupp= 20%) UK (minsupp= 0.1%) UPS (minsupp= 25%) UR (minsupp= 0.05%)

97.5 97 96.5 96 95.5 1

2

3

4

5

6

# partitions

Figure 4.3: fpSim of the APInterp results relative to datasets partitioned in different ways.

Brijs [10] for Retail and Roberto Bayardo for the conversion of UCI datasets. Other datasets were generated using the publicly available synthetic data generator code from the IBM Almaden Quest data mining project [6].

4.3

Conclusions

In this chapter, we have discussed APRed and APInterp , two new distributed algorithms for approximate frequent itemset mining. The key idea of APRed is that by using a reduced minimum support (r · minsup, r ∈ [0.5, 1]) for distributed local elaboration on dataset partitions, without modifying the support threshold for global evaluation of fetched results, we can be confident that the final approximate results obtained will result quite correct. Moreover, even if we lower the support threshold, APRed still results efficient, and the amount of data sent to the master by the local slave is relatively small. This is due to a strong pruning activity: locally frequent candidate patterns are in fact pruned by using an approximate knowledge of F2 (often discarding more than 90% of globally infrequent candidate patterns). In our test, APRed performs particularly well on sparse datasets: in several cases

4.3. Conclusions

69

Similarity Dataset Kosarak - 6 unbalanced partitions 100

%

90

80

70

60 Distr. One-pass Part. Sim() AP Sim() AP fpSim()

50 0.1

0.2

0.3

0.4 0.5 Minimum support (%)

0.6

0.7

0.8

Figure 4.4: Comparison of Distributed One-pass Partition vs APInterp .

an 80% reduction of minsup is enough to achieve a similarity close to 100%. On the other end on most dense dataset the number of missing and spurious patterns is definitely too high. APInterp , instead, exploits a novel interpolation method to infer unknown counts of some patterns, which are locally frequent in some dataset partitions. Since no support reduction is involved, APInterp is able to mine dense dataset for values of minsup that are too small to be used with APRed . For the same reason, also the issue related to bad choice of the support reduction factor (see the Accident dataset case in the APRed results), are avoided. For dataset partitioning characterized by high data skew, the APInterp approach is able to strongly improve the accuracy of the approximate results. Our tests prove that this method is particularly suitable for several (mainly sparse) datasets: it yields a good accuracy and scale nicely. The best approximate results obtained for the various datasets were characterized by a similarity above 99%. Even if some false positives are found, the high similarity value computed on the whole result set proves that the exact supports of these false positives are actually close to the support threshold, and thus of some interest to the analyst. The accuracy of the results is better than in Distributed One-pass Partition case. The main reason for this is that the Distributed One-pass Partition algorithm yields,

70

4. Distributed datasets

Speed up 10

A minsupp=20% K minsupp=0.1%

9 8

Speedup(n)

7 6 5 4 3 2 1 1

2

3

4

5

6

# partitions

Figure 4.5: Speedup for two of the experimental datasets, Kosarak(K) and Accidents (A), with balanced partitioning.

for any patterns, a support value that is the lower bound of the interval in which the exact support is included. Hence, the count estimated by our algorithm, which falls between the lower and upper bounds, is generally closer to the exact count than the lower bound. Furthermore, the proposed interpolation schema does not increase significantly the overall space/time complexity and is resilient to heavy skew in the distribution of items. Finally, both in APInterp and APRed , synchronization occurs just once as in a na¨ıve distributed Partition, and, differently from Partition, slaves do not have to be polled for specific pattern counts, thus limiting potential privacy breaches related to low support patterns.

4.3. Conclusions

71

Table 4.4: Accuracy indicators for APInterp results obtained using the maximum number of partitions and the lowest support.

Dataset A A A A C C C C K K K K M M M M P P P P PS PS PS PS R R R R R R UA UA UA UA UK UK UK UK UP UP UP UP UPS UPS UPS UPS UR UR UR UR UR UR

# slaves 3 3 6 6 3 3 6 6 3 3 6 6 3 3 6 6 3 3 6 6 3 3 6 6 3 3 3 6 6 6 3 3 6 6 3 3 6 6 3 3 6 6 3 3 6 6 3 3 3 6 6 6

Minsup % 20.00 30.00 20.00 30.00 70.00 80.00 70.00 80.00 0.10 0.20 0.10 0.20 5.00 8.00 5.00 8.00 70.00 80.00 70.00 80.00 25.00 30.00 25.00 30.00 0.05 0.10 0.20 0.05 0.10 0.20 20.00 30.00 20.00 30.00 0.10 0.20 0.10 0.20 70.00 80.00 70.00 80.00 25.00 30.00 25.00 30.00 0.05 0.10 0.20 0.05 0.10 0.20

Minsup (count) 68036 102054 68036 102054 47289 54045 47289 54045 990 1980 990 1980 406 649 406 649 34332 39236 34332 39236 12261 14713 12261 14713 44 88 176 44 88 176 68036 102054 68036 102054 990 1980 990 1980 34332 39236 34332 39236 12261 14713 12261 14713 44 88 176 44 88 176

# freq 899740 151065 912519 152873 4239440 546795 4335664 560499 852636 42963 947486 59601 3773538 864245 3888898 926827 2858126 145435 2921763 152855 2177124 441472 2227435 444542 17766 6105 1902 18372 6190 1967 901687 151268 916744 152942 818017 52212 922792 49420 2800681 149253 2879809 152124 2207340 455973 2162459 453334 17654 6185 1896 17901 6390 1968

Time (s) 51.92 7.45 27.72 4.57 56.09 6.79 93.50 10.73 68.77 8.14 31.94 5.15 41.14 8.78 67.61 15.49 39.07 2.14 58.23 2.62 29.31 5.80 45.9 9.06 0.86 0.53 0.34 0.69 0.41 0.30 66.72 10.07 35.19 5.55 121.46 11.65 45.30 5.54 38.14 2.25 56.82 2.69 29.79 6.26 44.49 9.02 0.96 0.57 0.36 0.80 0.43 0.29

Avg.Sup. Range(%) 0.289 0.378 0.574 0.768 2.401 2.894 4.093 5.191 0.013 0.033 0.024 0.077 0.542 0.862 0.899 1.182 3.766 3.170 6.068 7.020 1.672 1.238 2.526 2.261 0.005 0.009 0.018 0.006 0.010 0.024 0.309 0.440 0.639 0.782 0.011 0.062 0.020 0.050 3.217 5.101 5.216 6.777 2.102 1.980 1.976 2.359 0.005 0.010 0.019 0.005 0.012 0.025

Precision % 98.87 98.95 97.51 97.80 97.37 97.67 95.24 95.24 88.81 89.53 80.56 65.93 99.50 76.13 96.57 71.00 94.42 97.57 92.36 92.99 94.80 97.82 92.72 96.98 91.07 93.39 94.59 88.63 92.47 92.63 98.68 98.81 97.06 97.75 92.62 74.21 82.76 79.27 96.30 95.17 93.71 93.46 93.53 94.90 95.51 95.46 91.56 92.48 94.75 90.56 90.33 92.56

Recall % 99.96 99.97 99.99 100.00 99.93 99.98 99.97 99.93 99.11 98.05 99.89 99.80 99.97 99.98 100.00 100.00 99.99 99.81 99.99 99.97 99.93 99.78 99.99 99.61 99.89 99.84 99.82 99.92 99.88 99.96 99.98 99.96 99.99 99.98 99.17 98.54 99.95 99.69 99.94 99.91 99.99 100.00 99.96 99.98 99.99 99.99 99.91 99.83 99.78 99.95 99.91 99.93

False pos % 1.13 1.05 2.49 2.20 2.63 2.33 4.76 4.76 11.19 10.47 19.44 34.07 0.50 23.87 3.43 29.00 5.58 2.43 7.64 7.01 5.20 2.18 7.28 3.02 8.93 6.61 5.41 11.37 7.53 7.37 1.32 1.19 2.94 2.25 7.38 25.79 17.24 20.73 3.70 4.83 6.29 6.54 6.47 5.10 4.49 4.54 8.44 7.52 5.24 9.44 9.67 7.44

False neg % 0.04 0.03 0.01 0.00 0.07 0.02 0.03 0.07 0.89 1.95 0.11 0.20 0.03 0.02 0.00 0.00 0.01 0.19 0.01 0.03 0.07 0.22 0.01 0.39 0.11 0.16 0.18 0.08 0.12 0.04 0.02 0.04 0.01 0.02 0.83 1.46 0.05 0.31 0.06 0.09 0.01 0.00 0.04 0.02 0.01 0.01 0.09 0.17 0.22 0.05 0.09 0.07

Sim % 98.83 98.92 97.51 97.80 97.30 97.65 95.20 95.17 88.10 87.97 80.49 65.84 99.45 76.12 96.52 71.00 94.40 97.39 92.34 92.96 94.73 97.61 92.69 96.59 90.97 93.25 94.42 88.57 92.37 92.60 98.66 98.77 97.05 97.73 91.91 73.40 82.72 79.08 96.24 95.07 93.69 93.44 93.48 94.88 95.49 95.43 91.49 92.34 94.55 90.52 90.25 92.50

fpSim % 99.81 99.76 99.58 99.44 98.70 98.73 97.17 96.74 99.20 98.24 99.90 99.81 99.94 98.62 99.81 97.98 97.37 98.52 95.50 95.28 99.05 99.34 98.45 98.85 99.90 99.85 99.82 99.88 99.89 99.95 99.84 99.77 99.41 99.31 99.23 98.89 99.94 99.72 98.02 97.04 96.66 96.05 99.11 99.19 98.92 98.70 99.92 99.84 99.78 99.94 99.91 99.92

72

4. Distributed datasets

5 Streaming data Many critical applications require a nearly immediate result based on a continuous and infinite stream of data. In our case, we are interested in mining all frequent patterns and their supports from an infinite stream of transactions. We begin this chapter by describing the peculiarities of streaming data, then we will introduce the problem of finding the most frequent items and itemset in a stream, along with some state of the art algorithms for solving them. Finally, we will describe our contribution: a streaming algorithm for approximate mining of frequent patterns.

5.1

Streaming data

Before introducing the notation used in this chapter, we briefly summarize the notation previously used for frequent itemset and frequent items. A dataset D is a collection of subsets of items I = it1 , . . . , itm . Each element of D is called a transaction. A pattern x is frequent in dataset D with respect to a minimum support minsup, if its support is greater than σmin = minsup · |D|, i.e. the pattern occurs in at least σmin transactions, where |D| is the number of transactions in D. A kpattern isSa pattern composed of k items, Fk is the set of all frequent k-patterns, and F = i Fi is the set of all frequent patterns. If D contains just transactions of one item, then all of the frequent patterns are 1-patterns. These patterns are named frequent items. Since the stream is infinite, new data arrive continuously and results change continuously as well. Hence, we need a notation for indicating that a particular dataset or result is referred to a particular time interval. To this end, we write the interval as a subscript after the entity. Thus D[t0 ,t1 ) indicates the part of the stream received since t0 and before t1 . For the sake of simplicity we will write just D instead of D[1,t] , when referring to all data received until current time t, if this notation is not ambiguous. As usual, a square bracket indicates that the bound is part of the interval, whereas a parenthesis indicates that it is excluded. A pattern x is frequent at time t in the stream D[1,t] , with respect to a minimum support minsup, if its support is greater than σmin[1,t] = minsup · |D[1,t] |, i.e. the pattern occurs in at least σmin[1,t] transactions, where |D[1,t] | is the number of transactions in the stream D until time t. A k-pattern is a pattern composed of k

74

5. Streaming data

items, Fk[1,t] is the set of all frequent k-patterns, and F[1,t] is the set of all frequent patterns.

5.1.1

Issues

The infinite nature of these data sources is a serious obstacle to the use of most of the traditional methods since available computing resources are limited. One of the first effects is the need to process data as they arrive. The amount of previously happened events is usually overwhelming, so they can be either dropped after processing or archived separately in secondary storage. In the first case access to past data is obviously impossible whereas in the second case the cost for data retrieval is likely to be acceptable only for some ”ad hoc” queries, especially when several scan of past data are needed to obtain just one single result. Other important differences with respect to having all data available for the mining processed at the same time regard the obtained results. As previously explained, both the data and the results evolve continuously. Hence a result is referred to a part of the stream and, in our case, to the whole part of the stream preceding a given time t. Obviously, an algorithm suitable for streaming data should be able to compute the ’next step’ solution on-line, starting from the previously known D[1,t−1) and the current data D[t−1,t) , if necessary with some additional information stored along with the current solution. In our case, this information is the count of a significant part of frequent single items, and a transaction hash table used for improving deterministic bounds on supports returned by the algorithm, as we will explain later in this chapter.

5.2

Frequent items

Even the apparently simple discovery of frequent items in a stream is challenging, since its exact solution requires to store a counter for each distinct item received. Some items may appear initially in a sporadic way and then become frequent, thus the only way to exactly compute its support is to maintain a counter since its first appearance. This could be acceptable when the number of distinct items is reasonably bounded. If the stream contains a large and potentially unbounded number of spurious items, as in case of data with probabilities of occurrence that follows Zipf’s law like internet traffic data, this approach may lead to a huge waste of memory. Furthermore, the number of distinct items is potentially proportional to the length of the stream. The Top Frequent items problem is closely related to the frequent items one, except that the user does not directly decide the support threshold: the result set contains only a given number of items having the highest supports. In this case too the resource usage is unbounded. This issue has been addressed by several approximate algorithms, which sacrifice the exactness of the result in order to limit the space complexity. In this section, we will formally introduce the problem, and

5.2. Frequent items

75

then we will describe some representative approximate algorithms for finding the set of most frequent items.

5.2.1

Problem

Let D[1,n] = s1 , s2 , . . . , sn be a data stream, where each position in the stream si contains an element of the items I = it1 , . . . , itm . Let item iti occur σ[1,n] (iti ) times in D[1,n] . The k items having the highest frequencies are named the top-k items whereas items whose frequencies are greater than σmin = minsup · |D| are named frequent items. As explained before and in [12] the exact solution of this problem is a highly memory intensive problem. Two relaxed versions of this problem have been introduced in [12]: FindCandidateTop(S, k, l) and FindApproxTop(S, k, ). The first one is exact and consists in finding a list of l items containing the k most frequent, whereas the second one is approximate. Its goal is to find a list of items having a frequency greater than (1 − ) · σ[1,n] (itk ) where itk is the k th most frequent item. FindCandidateTop can be very hard to solve for some input distributions, in particular when the frequencies of the k th and the (l + 1)th are similar. In such cases, the approximate problem is more practical to solve. Several variation of the top-k frequent items problem have been proposed. The Hot Items problem described in [40] for large datasets and, several year after, adapted to data streams ([16, 30]) is essentially the top-k frequent items problem formalized in a slightly different way. The techniques used for solving this family of problems can be classified into two large categories: count-based techniques and sketch-based techniques. The first ones monitor a limited set of potentially ”interesting” items, using a counter for each one of them. In this case, an error arise when an item is erroneously kept out of the set or inserted too late. The second family provides a frequency estimation for every item by using a hash-indexed vector of counters. In this case, the risk of completely missing the occurrences of an item is avoided, at the cost of looser guarantees on the computed frequencies.

5.2.2

Count-based algorithms

Count-based algorithms maintains a set of counters, each one associated with a specific item. When the number of distinct items is supposed to be high, there could be not enough memory for allocating all the counters. In this case, it is necessary to limit our attention to a set of items compatible with the available memory. Only the items in the monitored set have an associated counter, which is incremented upon their arrival. Other items have just an opportunity to replace one of the monitored items. In fact, in most methods, the set of monitored items varies during the computation. Each algorithm of this family is characterized by the data structure for the efficient maintenance of counters and the policy for replacing ”old” counters it uses.

76

5. Streaming data

The Frequent algorithm This method has been originally proposed in [40] for large datasets and is inspired by an algorithm for finding a majority element. More recently two unrelated works ([16, 30]) have described new versions adapted to streams. A well-known algorithm for discovering the most frequent item in a set, containing repetitions of two distinct items, consists in removing pairs of distinct items from the set while this is possible. The elements left are all identical and their identity is the solution. In the case there are more than two distinct items this method will still work, provided that the majority exists, i.e., the most frequent element has more than n2 occurrences, where n is the stream length. Algorithm 6 shows the most efficient implementation of Majority. It requires just two variables: one contains the value currently supposed to be the majority and the other is a counter, indicating a lower bound for the advantage of the current candidate against any other opponent. At the end of the scan of the data, the only possible candidate is known. In case we are not dealing with streams, a second scan over the data will give the definitive answer. Algorithm 6: Majority input : data [1]...data [n] output: majority element if any 1 2 3 4 5 6 7 8 9 10 11 12 13

candidate ← data[1]; C ← 1; for i ← 2 to n do if C = 0 then candidate ← data[i]; if candidate = data[i] then C ← C + 1; else C ← C − 1; end C ← 0; for i ← 1 to n do if candidate = data[i] then C ← C + 1; end if C 6 n2 then return NULL; else return candidate

In order to efficiently discard pairs, items are examined only when they arrive. The candidate variable keep track of the item currently prevailing, and a counter C indicates the minimum number of different items required to reach a tie. In other words, the number of item having the prevailing identity that are waiting to be matched with a different item. If a majority exists, i.e. if an item has support greater than n2 , it will be found. This is granted by the fact that an item is discarded only when paired with a different item, and, for the majority element, this cannot

5.2. Frequent items

77

happen for all of its occurrences. In case the most frequent item has a support smaller than n2 the behavior of the Majority algorithm is unpredictable. Furthermore, we may be interested in finding more than one of the top frequent items. The Frequent algorithm (algorithm 7) is thus a generalization of Majority, and is able to deal with these two cases. Its goal is to find a set of m items containing every item having a relative support strictly greater than m1 . The key idea is to keep a limited number m of counters and, when a new item arrives, decrement every counter, and replace one of the items having the counter value equal to zero, if there is any. In this way an item is always discarded Algorithm 7: Frequent input : data [1]...data [n] output: superset of items having support greater than 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1 m

C ← {}; for i ← 1 to n do if ∃f (data[i], f ) ∈ C then replace (data[i], f ) with (data[i], f + 1); else if ∃item (item, 0) ∈ C then replace (item, 0) with (data[i], 1) in C ; else if |C| < m then insert (data[i], 1) in C ; else foreach (item, f ) ∈ C do replace (item, f ) with (item, f − 1) in C ; end end end return {item : ∃f (data[i], f ) ∈ C}

together with m − 1 occurrences of other symbols, or m when the incoming symbol is discarded too because no counter has reached zero, i.e. a total of m or m + 1 symbols are discarded. Hence if a frequent symbol x is discarded d times, either before of after its insertion in the counter set, then a total of at most d · (m + 1) 6 n n n stream positions will be discarded. Since x is frequent σ(x) > m > m+1 > d. Thus, an item that is frequent in the first n position of the stream will be in the set of counters after the processing of the nth position. In order to manage counters efficiently, a specifically designed data structure is required. In particular the operations of insertion, update and removal of a counter as well as the decrement of the whole set of counter need to be optimized. Both [16] and [30] propose a data structure based on differential support encoding and a mix of hash and double linked list which grants a worst-case amortized time complexity which is O(1), and O(m) worst-case space bound.

78

5. Streaming data

This algorithm, in its original formulation, find just a superset of the frequent items, with no indication on support and no warranty on the absence of false positives. In the case of an ordinary dataset, both issues can be avoided with a second scan over the dataset but, on streaming data, this is not possible. However if we are allowed to use some additional space, it is possible to find also an estimate of the actual support of each items with some upper bound. In order to reach this goal, we need to maintain an additional counter which is never decreased, corresponding to a lower bound the support of each item, and a constant value indicating the maximum number of previous occurrences before the insertion in the counter set. Since the Frequent algorithm is correct, this amount is σmin[1,t] − 1, the maximum integer smaller than the support threshold for the corresponding stream portion. Furthermore, it is possible to exclude from the result set every item having the support under a specified value by increasing the number of counters and applying a post-filter as described in [29] for itemsets. The Lossy count algorithm The Lossy Count algorithm (algorithm 8) was introduced in [33]. Its main advantages versus the original formulation of Frequent are the presence of a constraint on false positive and the computation of an approximate support, similarly to the modified version of Frequent. Furthermore it is easily extensible to frequent itemsets, as we will see later in this chapter. The kind of solution this algorithm find is called an -deficient synopsis and consists in a result set containing every frequent item, but no item having relative support less than minsup − , along with a support approximation that is smaller than the exact relative support by at most . The algorithm manages a set C of items, each associated with a counter and a bound on its error. When a new item x arrives and x is known, its counter is incremented. Otherwise a new entry (item,  1  1, bucket − 1) is inserted in C, where bucket is the number of blocks of w =  elements seen so far, and bucket − 1 is the maximum number of previously missed occurrences of item x. The algorithm is granted to maintain correctly the support in the -deficient synopsis. Hence, at the beginning of a new block it is possible to delete every counter having a best-case estimated support less than the error it would have if reinserted from scratch, which is equal to bucket − 1. Since estimated frequencies are less than true frequencies by at most , in order to get every frequent item but no item having relative support less than minsup −  it is enough to return only the items having an upper bound for support, f + ∆, greater than or equal to minsup − . The Sticky Sampling algorithm Both [16] and [33] propose also some non deterministic methods. The idea is to keep the most frequent counters and delete the others in order to free space for new potentially frequent items. The way this is done, however, is different in the

5.2. Frequent items

79

Algorithm 8: Lossy Count input : data[1]...data[n] minsup,  output: set containing every item having support greater than minsup · n and no item whose support is less than (minsup − ) · n 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

bcurrent ← 1 ; C ← {}; for i ← 1 to n do if (∃f, ∆) (data[i], f, ∆) ∈ C then replace (data[i], f, ∆) with (data[i], f + 1, ∆); else insert (data[i], 1, bcurrent − 1) in C ; end   if i mod 1 = 0 then bcurrent ← bcurrent + 1 ; foreach (item, f, ∆) ∈ C do if f + ∆ < bcurrent − 1 then remove (item, f, ∆) from C ; end end end end return {item : (∃f, ∆) (item, f, ∆) ∈ C ∧ f + ∆ > (minsup − ) · n)}

80

5. Streaming data

Algorithm 9: Sticky Sampling input : data[1]...data[n] minsup, , δ output: set containing every item having support greater than minsup · n and no item whose support is less than (minsup − ) · n with probability of failure δ 1 2 3 4 5 6 7 8 9 10 11

12 13 14 15 16 17 18 19 20

C ← {}; 1 ; t ← 1 · log minsup·δ block len ← 2 · t; rate ← 1; for i ← 1 to n do if i mod block len then rate ← 2 · rate ; block len ← t · rate ; correct counters ; foreach (item, f ) ∈ C do while binomial(1, 12 )=0 do replace (item, f ) with (item, f − 1); end end if ∃f (data[i], f ) ∈ C then replace (data[i], f ) with (data[i], f + 1) ; 1 )=0 then else if binomial(1, rate insert (data[i], 1) in C ; end end return {item : ∃f (data[i], f ) ∈ C ∧ f > (minsup − ) · n}

5.2. Frequent items

81

two cases. Probabilistic-Inplace [16] discard one-half of the counters every r received items and select the first items found immediately after the discard occurs. Sticky Sampling [33] (algorithm 9) use, instead, a uniform sampling strategy over the whole stream. In order to keep the number of counters probabilistically bounded, the sampling rate is decreased for increasing stream lengths, and the previously known frequencies are corrected to reflect the new rate using a stochastic method.

5.2.3

Sketch-based algorithms

As Count-based algorithms, also Sketch-based ones maintain a set of counters but, instead of associating the counters with particular items, they are associated with different overlapping groups of items. The analysis of the values of the counters for the various groups containing an item allows us to give an estimate of its support. In this approach there is no notion of monitored item, and the support estimation is possible for any item. Algorithms included in this family, as in the case of the Count-based family, share the same basic skeleton. The main differences are in the management of the counters, the kind of other queries that can be answered by using the same count-sketch and the exact function used for support estimation, which directly influence the space requirements based on the user selected acceptable error probability. In [15] G.Cormode and S.Muthukrishnan present their particularly flexible Count-Min Sketch data structure as well as a good comparison to other state of the art sketch techniques. We will adopt their unification framework in order to describe a generic sketch based algorithm. A sketch is a two dimensional array of dimension w by d. Let m be the number of distinct items, h1 . . . hd be hash functions mapping {1 . . . m} into {1 . . . w} and let g1 . . . gd be other hash functions defined on items. The (j, k) entry of the sketch is defined to be X σ(i) · gk (i) i:hk (i)=j

In other words, when an item i arrives, for each k ∈ {1 . . . d} the entry (hk (i), k) is increased by the amount gk (i), which is algorithm dependent. Thus, the update time complexity is O(d) and the space complexity is O(wd), provided that the hash functions can be stored efficiently. The way the data structure is used in order to answer a particular query, the required randomness and independence of the hash functions, as well as the minimum size of the sketch array needed to guarantee the fulfillment of probability of error constraints, are algorithm dependent. A particularly simple count sketch is Count-Min [15]. In its case the values of functions gk (item) are always 1, i.e. each counter is incremented by one each time an item is transformed into its identifier by a hash function. The approximate value is computed as the smallest of the counters associated with an item by any hash function. Since several items can be hashed to the same integer, the approximate value is always greater than the exact one. The two fragments of pseudo-code show

82

5. Streaming data

the simple updateSketch procedure and approxSupport function used by Count-Min sketches. Procedure updateSketch(sketch,item) - Count-Min sketch 1 2 3

foreach k ∈ {1 . . . d} do sketch[hk (item), k] ← sketch[hk (item), k] + 1; end

Function approxSupport(sketch, item) - Count-Min sketch 1

return mink∈{1...d} sketch[hk (item), k];

Other count sketch based methods are described in [12, 17, 28, 14].

5.3

Frequent itemsets

In this section, we introduce a new algorithm for approximate mining of frequent patterns from streams of transactions using a limited amount of memory. In most cases, finding an exact solution is not compatible with limited resources available and real time constraints, but an approximation of the exact result is enough for most purposes. The proposed algorithm consists in the computation of frequent itemsets in recent data and an effective method for inferring the global support of previously infrequent itemsets. Both upper and lower bounds on the support of each pattern found are returned along with the interpolated support. Before introducing our algorithm, we will shortly describe two other algorithms for approximate frequent itemset mining. Then we will give an overview of APStream , our algorithm, followed by a more detailed description and an extensive experimental evaluation showing that APStream yields a good approximation of the exact global result considering both the set of patterns found and their support.

5.3.1

Related work

The frequent itemset mining problem on stream of transactions (input itemsets) poses additional memory and computational issues due to the exponential growth of solution size with respect to the corresponding problem on streams of items. Here we describe two representative approximate algorithms. The Lossy Count algorithm for frequent itemsets Manku and Motwani proposed in [33] an extension of their Lossy Count approximate algorithm to the case of frequent itemsets. A straightforward conversion of Lossy

5.3. Frequent itemsets

83

Count, using the same data structure in order to store the support of patterns as the transactions arrive, is possible but it would be highly inefficient. This is due to the exponential number of patterns supported by each transaction. Actually, it would be the same than computing the full set of itemset with no support constraint and removing periodically infrequent pattern. In order to avoid this issue, the authors process the transactions in blocks, so that the apriori constraint may be applied. The algorithm is much similar to that previously described for items, so we will focus on differences. The most notable  1 is that the transactions are processed in batches containing several buckets of size  . As many transactions as the available memory can fit are buffered and then mined, using the number of buckets β as minimum support. This is roughly equivalent to searching patterns appearing at least once in each bucket, but more efficient. Every pattern x with support f in the transactions currently buffered is inserted in the set of counters as (x, f, bucket − β), where bucket indicates the last bucket contained in the buffer. At the same time the support of every pattern already in the counter set is checked in current buckets, updating the counters if needed and removing patterns that no longer satisfy the f + ∆ > bucket inequality. Clearly, in order to avoid the insertion in the counter set of spurious patterns, β should be a large number. Hence, a larger available memory increase the accuracy and reduce the running time. The Frequent algorithm for frequent itemsets In [29] R.Jin and G.Agrawal propose SARM, a new algorithm for frequent itemset mining based on Frequent [30]. Also in this case, the immediate extension of the base algorithm has serious shortcomings. This is mainly due to the potentially high 1 counters number of frequent patterns. While in the frequent items case just minsup are needed, for frequent itemsets one of the arguments used in the correctness proof is no longer true. In fact, in a stream of n transactions there can be more than n k-patterns having support greater than minsup. More precisely there can minsup

n n n frequent items, 2l · minsup frequent pairs, and in general kl · minsup be l · minsup frequent k-pattern, where l is the length of transactions. Since the maximum length of frequent patterns is unknown before computation, the user would need to specify the maximal pattern length, maxlen, to use in order to correctly size the counter set. Thus the number of counters needed for the computation of frequent itemsets would be maxlen X l 1 minsup k=1 k Furthermore, unless the transactions are processed in batches as in Lossy Count, all the subpatterns of each incoming transaction need to be examined. In order to avoid these side effects, the SARM algorithm maintains separate sets Lk of potentially frequent itemsets, one for each different pattern length k. These sets are updated using a hybrid approach: SARM updates L1 and L2 using the same

84

5. Streaming data

method proposed in Frequent, and at the same time buffers transactions for a levelwise batched processing. When a transaction t arrives, it is inserted in a buffer, and both L1 and L2 are updated either by incrementing the count, for already known 1 , patterns, or inserting the new ones. If the size of L2 exceeds the limit f · minsup· where  ∈ [0, 1] is a factor used for increasing the accuracy, and f is the average number of 2-patterns per transaction, then the size of L2 is reduced by executing the CrossOver operation, consisting in decreasing every counter and removing, as in Frequent, patterns having count equal to zero. Every time this operation is performed, the transaction buffer is processed. For increasing values of k > 2, the k-patterns appearing in the buffer and having all subpatterns included in Lk−1 are used for updating Lk . Then the buffer is emptied and the CrossOver operation is applied to each Lk . The  ∈ [0, 1] factor can be used for enforcing a bound on result accuracy. If  < 1 then no itemset having relative support less than (1−)·minsup will be in the result set. Thus F minsup ⊆ L ⊆ F (1−)·minsup , where L is the result set, and F s is the set of itemset whose support exceed s. When  = 1 the SARM algorithm is not able to give any guarantee on the accuracy, as the Frequent algorithm. Furthermore, both Lossy Count for itemsets and SARM ignore previous potential occurrences of a pattern when it is inserted into the set of frequent patterns. In the case of Lossy Count the maximum number of neglected occurrences is returned along with the support, but no other information available during the stream processing is exploited.

5.3.2

The APStream algorithm

In order to overcome these limitations APStream (Approximate Partition for Stream), the algorithm we propose, uses the available knowledge on the support of other patterns to estimate a support for previously disregarded ones. The APStream algorithm was inspired by Partition [55], a sequential algorithm that divides the dataset into several partitions processed independently and then merges local solutions. The adjectives global and local are referred to temporal locality. So they are used in conjunction with properties of, respectively, the whole stream and just a relatively small and contiguous part of the stream, hereinafter called a block of transactions. Furthermore, we suppose that each block corresponds to one time unit: hence, D[1,n) will indicate the first n−1 data blocks, and Dn the nth block. This hypothesis allows us to adopt a lighter notation and cause no loss of generality. The Streaming Partition algorithm. The basic idea exploited by Partition is the following: if the dataset is divided into several partitions, then each globally frequent pattern must be locally frequent in at least one partition. This guarantees that the union of all local solutions is a superset of the global solution. However, one further pass over the database is necessary to remove all false positives, i.e. patterns that result locally frequent but globally infrequent.

5.3. Frequent itemsets

85

In order to extend this approach to a stream setting, blocks of data received from the stream are used as an infinite set of partitions. A block of data is processed as soon as ”enough” transactions are available, and results are merged with the current approximate result, which is referred to the past part of the stream. Unfortunately, in the stream case, only recent raw data (transactions) can be maintained available for processing due to memory limits, thus the usual Partition second pass will be restricted to accessible data. Only the partial results extracted so far from previous blocks, and some other additional information, can be available for determining the global result set, i.e. the frequent patterns and their supports. One na¨ıve workaround is to avoid the second pass and keep in the result set only patterns having the sum of the known supports, i.e. only those corresponding to patterns that resulted to be locally frequent in the various blocks mined so far, greater than (or equal to) minsup. We will name this algorithm Streaming Partition. The first time a pattern x is reported, its support corresponds to the support computed in the current block. In case it appeared previously, this mean introducing an error. If j is the first block where x is frequent, then this error can be at most σmin[1,j] − 1. This is formalized in the following lemma. Lemma 11 (Bounds on support after first pass). Let P = {1, ..., n} be the set of indexes of the n block received so far. Then let f part(x) = {j ∈ P |σj (x) > minsup · |Dj |} be the set of indexes of the blocks where the pattern x is frequent and let f part(x) = (P r f part) be its complement. The support for a pattern x is no less than the support computed by the Streaming Partition algorithm (σ lower (x)) and is less than or equal to σ lower (x) plus the maximum support the same pattern can have in blocks where it is not frequent: σ(x)lower =

X j∈f part(x)

σj (x)

,

σ(x)upper = σ(x)lower +

X

minsup · |Dj | − 1

j∈f part(x)

Note that when a pattern x is frequent in a block Dj , its local support is summed to both the upper and lower bounds. Otherwise, its local support can range from 0 (no occurrence) to the local minimum support threshold minus one (i.e. minsup · |Dj | − 1), thus the lower bound remains the same, whereas the upper bound is increased. We can easily transform the two absolute bounds defined above into the corresponding relative ones, usable to calculate the Average Support Range, defined in appendix A: n

upper

sup(x)

X σ(x)upper σi (x)lower = , sup(x)lower = , where |D| = |Dj | |D| |D| j=1

Streaming Partition has serious resource usage issues. In order to keep track of frequent itemsets, a counter for each distinct pattern found to be frequent in at least one block is needed. This obviously leads to an unacceptable memory usage in most cases. The only way to overcome this limitation is introducing some kind of forget policy: in the remainder of this paper when we refer to Streaming Partition we

86

5. Streaming data

mean Streaming Partition with the deletion of patterns that resulted to be globally infrequent after each block processing. Another problem with Streaming Partition is that for every pattern the computed support is a very conservative estimate, since it always chooses the lower bounds to approximate the results. Generally, any algorithm returning a support value between the bounds will have better chances of being more accurate. Following this idea, we devised a new algorithm based on Streaming Partition that uses a smart interpolation of support. Moreover, it is resilient to skewed item distributions. The APStream algorithm. The streaming algorithm we propose, APStream , tries to overcome some of the problems encountered by Streaming Partition and other similar algorithms for association mining on streams when the data skew between different incoming blocks is high. The most evident is that several globally infrequent patterns may be locally frequent, increasing both resource utilization and execution time of these algorithms. APStream addresses this issue by means of global pruning based on historical exact (when available) or interpolated support: each locally frequent pattern that is not globally frequent according to its interpolated support will be immediately removed and will not produce any child candidate. Moreover this skew might cause a globally frequent pattern x to result infrequent on a given data block Di . In other words, since σi (x) < minsup·|Di |, x will not be found as a frequent pattern in the ith block. As a consequence, we will not be able to count on the knowledge of σi (x), and thus we cannot exactly compute the support of x. Unfortunately, P Streaming Partition might also deduce that x is not globally frequent, because j,j6=i σj (x) < minsup · |D|. Result merge and interpolation When an input block Di is available for processing, APStream extract its frequent itemsets using the DCI algorithm. Then for each pattern x, included either in past combined results or in the recent FIM results, it computes the approximate global support σ[1,i] (x)interp in different ways, according to the specific situation. The approximate past support (σ[1,i) (x)interp ) was obtained by merging the FIM results of blocks D1 . . . Di−1 using the technique currently discussed. σ[1,i) (x)interp can be either known or not, depending on the presence of x in the past combined results. In the same way, σi (x) is known only if x is frequent in Di . The following table summarizes the possible cases and the action taken by APStream : σ[1,i) (x)interp known known unknown

σi (x) known unknown known

Action sum σi (x) to past support and bounds. recount σi (x) on recent, still available, data. interpolate past support σ[1,i) (x)interp

The first case is the simpler to handle: the new support σ[1,i] (x)interp will be the sum of σ[1,i) (x)interp and σi (x). Since σi (x) is exact, the width of the error interval

5.3. Frequent itemsets

87

will remain the same. The second one is similar, except that we need to look at recent data for computing σi (x). The key difference with Streaming Partition is the handling of the last case. APStream , instead of supposing that x never appeared in the past, tries to interpolate σ[1,i) (x). The interpolation is based on the knowledge of: • the exact support of each item (or optionally just the approximate support of a fixed number of most frequent items) • the reduction factors of the support count of subpatterns of x in current block with respect to its interpolated support over the past part of the stream. The algorithm will thus deduce the unknown support σ[1,i) (x) of itemset x on the part of the stream preceding the ith block as follows: ( interp

σ[1,i) (x)

= σi (x) ∗ min

min

(

σ[1,i) (item) σ[1,i) (x r item)interp , σi (item) σi (x r item)

) )! item ∈ x

In the previous formula the result of the inner min is the minimum among the ratios of supports of items contained in pattern x in past and recent data, and the same values computed for itemsets obtained from x by removing one of its items. Note that during the processing of recent data, the search space is visited level-wise and the merge of the results is performed starting from shorter pattern too. Hence the interpolated supports σ[1,i) (x r item)interp of all the k − 1-subpatterns of a k-pattern x are known. In fact, each support can be either known from the processing of the past part of the stream or computed during the previous iteration on recent data. Example of interpolation. Suppose that we have received 440 transactions so far, and that 40 of these are in the current block. The itemset {A, B, C}, briefly indicated as ABC, is frequent locally whereas it was infrequent in previous data. Table 5.1 reports the support of every subpattern involved in the computation. The first column contains the patterns, the second and third columns contain the supports of the patterns in the last received block and in the past part of the stream. Finally, the last column shows the reduction ratio for each pattern. The algorithm examines itemsets of size k − 1 (two in this simple example), and single items, and choose the one having the minimum ratio. In this case the minimum is 2.5, corresponding to the subpattern {A, C}. Since in recent data the support of itemset x = {A, B, C} is σi (x) = 6, the interpolated support will be σ[1,i) (x)interp = 6 · 2.5 = 15 It is worth remarking that this method works if the support of larger itemsets decreases similarly in most parts of the stream, so that a reduction factor (different for each pattern) can be used to interpolate unknown values. Finally note that, as regards the interpolated value above, the following inequality should hold: σ[1,i) (x)interp < minsup · |D[1,i) |. If it is not satisfied, the interpolated result should

88

5. Streaming data

x σi (x) σ[1,i) (x)interp {A, B, C} 6 ? {A, B} 8 50 {A, C} 12 30 {B, C} 10 100 {A} 17 160 {B} 14 140 {C} 18 160 {} 40 400

σ[1,i) (x)interp σi (x)

? 6.2 2.5 10 9.4 10 8.9 -

Table 5.1: Sample supports and reduction ratios (σmin[1,t) = 20). not be accepted since, otherwise, the exact value σi (x) should have already been found. Hence, in those few cases where the above inequality does not hold, the interpolated value will be: σ[1,i) (x)interp = (minsup · |D[1,i) |) − 1. In the example described in table 5.1 the interpolated support for {A, B, C} is 15 and the minimum support threshold for past data is 20, so the bound is respected. Otherwise, the interpolated support would be forced to 19. The proposed interpolation schema yields a better approximation of exact results than Streaming Partition, in particular with respect to the approximation of the support of frequent patterns. The supports computed by the latter algorithm are, in fact, always equal to the lower bounds of the intervals containing the exact support of any particular pattern. Hence any kind of interpolation producing an approximate result set, whose supports are between the interval bounds, should be, generally, more accurate than picking always its lower bound. For the same reason the computed support values should be also more accurate than those computed by Lossy Count (Frequent does not return any support value). Obviously several other way of computing a support interpolation could be devised. Some are simple as the average of the bounds while others are complex as counting inference, used in a different context in [43]. We chose this particular kind of interpolation because it is simple to calculate, since it is based on data that we already maintain for other purposes, and it is aware of the underlying data enough to allow for accurate handling of datasets characterized by data-skew on item distributions among different blocks. We can finally introduce the pseudo-code of APStream . As in Streaming Partition the transactions are received and buffered. DCI, the algorithm used for the local computations, is able to exactly know the amount of memory required for mining a dataset during the intersection phase. Since frequent patterns are processed sequentially and can be offloaded to disk, the memory needed for efficient computation of frequent patterns is just that used by the bitmap representing the vertical dataset and can be computed knowing the number of transactions and the number of frequent items.

5.3. Frequent itemsets

Procedure processBlock(frequentItems,buffer, globFreq) 1 2 3 4 5 6 7

locF req[1] ← f requentItems ; k←2; while locF req[k − 1].size >= k do locF req[k] ← computeF requent(k, locF req, globF req) ; if k =2 then V D ← f illV erticalDataset(buf f er, f requentItems) ; commitInsert(V D, k, locF req, globF req) ; end

Procedure commitInsert(VertData,k,locFreq, globFreq) 1 2 3 4 5 6 7

foreach pat ∈ globF req[k] : pat ∈ / locF req[k] do compute support of pat in VertData ; if pat is frequent then pre-insert pat in globFreq[k] ; end end replace globFreq[k] with sorted insertBuffer;

Function computeFrequent(k,locFreq, globFreq) 1 2 3 4 5 6 7 8 9

compute local frequent pattern ; foreach locally frequent pattern pat do compute global interpolated support and bounds ; if pat is globally frequent then insert pat in locFreq[k] ; pre-insert pat in globFreq[k] ; end end return Fk ;

89

90

5. Streaming data

Thus, we can use this knowledge in order to maximize the size of the block of transactions processed at once. For the sake of simplicity we will neglect the quite obvious main loop with code related to buffering, concentrating on the processing of each data block. The interpolation formula has been omitted too, in the pseudocode, for the same reason. Each block is processed, visiting the search space level-wise, for discovering frequent patterns. In this way, itemsets are sorted according to their length and the interpolated support for frequent subpattern is always available when required. The processing of patterns of length k is performed in two steps. First frequent patterns are computed in the current block and then the actual insertion into the current set of frequent patterns is carried out. When a pattern is found to be frequent in the current block its support on past data is immediately checked: if it was already known then the local support is summed to previous support and previous bounds. Otherwise, a support and a pair of bounds are inferred for past data and summed to the support in the current block. In both cases, if the resulting support pass the support test, the pattern is queued for delayed insertion. After every locally frequent pattern of the current length k has been processed, the support of every previously known pattern that is not locally frequent is computed on recent data. Patterns passing the support test are queued for delayed insertion too. Then the set of pre-inserted itemsets is sorted and the actual insertion take place. Bounds on computed support errors As a consequence of using an interpolation method to guess an approximate support value in the past part of the stream, it is very important to establish some bounds on the support found for each pattern. In the previous subsection, we have already indicated a pair of really loose bounds: each support cannot be negative, and if a pattern was not frequent in a time interval then its interpolated support should be less than the minimum support threshold for the same interval. The lower bound is obviously always satisfied, whereas in case a support value σ[1,i−1] (x)interp breaks its upper bound value, it will be forced to (minsup · |D[1,i−1] |) − 1 which is the greatest value compatible with the bound. This criterion is completely true for nonevolving distributed dataset (distributed frequent pattern mining) or for the first two data block of the stream. In the stream case, the upper bound is based on previous approximate results, and could be inexact if the pattern corresponds to a false negative. Nevertheless, it does represent a useful indication. Bounds based on pattern subset The first bounds that interpolated supports should obey, derive from the Apriori property: no set can have a support greater than those of any of its subset can. Since recent results are merged level-wise with previously known ones, the interpolation can exploit already interpolated subset support. When a subpattern is missing during interpolation this mean that it has been examined during a previous level and discarded. In that case, all of its superset

5.3. Frequent itemsets

91

may be discarded as well. The computed bound is thus affected by the approximation of past results: a pattern with an erroneous support will affect the bounds for each of its superset. To avoid this issue it is possible to compute the upper bound for a pattern x simply using the upper bounds of its sub-patterns instead of their support. In this way, the upper bounds will be weaker, but there will be less false negatives due to erroneous bounds enforcement. Bounds based on transaction hash In order to address the issue of error propagation in support bounds we need to devise some other kind of bounds that are computed exclusively from received data and thus are independent of any previous results. Such bounds can be obtained using inverted transaction hashes. This technique was first introduced in the algorithm IHP [26], an association mining algorithm where it was used for finding upper bounds for the support of candidates in order to prune infrequent ones. As we will show this method can be used also for lower bounds. The key idea is that each item has an associated hashed set of counters that are accessed by using transaction id as a key. More in detail, each array hcnt[item] associated with an item is an array of hsize counters initialized to zero. When the tidth transaction t = {ti } is processed, a hash function transforms the tid value into an index to be used for the array of counters. Since tids are consecutive integer numbers, a trivial hash function as h(tid) = tid mod hsize will guarantee an equal repartition of transactions among all hash bins. For each item ti ∈ t the counter at position h(tid) in the array hcnt[ti ] is incremented. The hash function implicitly subdivides the transactions of the dataset. Each partition corresponds to a position in the array of counters, while the value of each counter represents the number of occurrences of an item in a given set of transactions. These hashes are a sort of ”compressed” tid-list and can be intersected to obtain deterministic bounds for the number of occurrences of a specified pattern. Notably these arrays of counters have a fixed size, independent from the number of transactions processed. Let hsize = 1, A and B two items and hA = hcnt[A][0] and hB = hcnt[B][0] the only counters contained in their respective hashes, i.e. hA and hB are the number of occurrences of items A and B in the whole dataset. According to the Apriori principle the support σ({A, B}) for the pattern {A, B} can be at most equal to min(hA , hB ). Furthermore, we are able to indicate a lower bound for the same support. Let n[i] be the number of transactions associated with the ith hash position, which, in this case, corresponds to the total number of transactions n. We know from the inclusion/exclusion principle that σ({A, B}) should be greater than or at least equal to max(0,hA + hB − n). In fact if n − hA transactions does not contains the item A then at least hB − (n − hA ) of the hB transactions containing B will also contain A. Suppose that n = 10, hA = 8, hB = 7. If we represent with an X each transaction supporting a pattern and with a dot any other transaction we obtain the following diagrams:

92

5. Streaming data Best case(ub(AB)= 7) A: XXXXXXXX.. B: XXXXXXX... AB: XXXXXXX...

Worst case(lb(AB)=5) XXXXXXXX.. ...XXXXXXX ...XXXXX..

Then no more than 7 transactions will contain both A and B. At the same time at least 8 + 7 − 10 = 5 transactions will satisfy that constraint. Since each counter represents a set of transaction, this operation is equivalent to the computation of the minimal and maximal intersections of the tid-lists associated with the single items. Usually hsize will be larger than one. In that case, the previously explained computations will be applied to each hash position, yielding an array of lower bounds and an array of upper bounds. The sums of their elements will give the pair of bounds for pattern {A, B} as we will show in the following example. Let hsize = 3, h(tid) = tid mod hsize the hash function, A and B two items and n[i] = 10 be the number of transactions associated with the ith hash position. Suppose that hcnt[A] = {8, 4, 6} and hcnt[B] = {7, 5, 6}. Using the same notation previously introduced we obtain: h(tid)=0 Best case Worst case A: XXXXXXXX.. XXXXXXXX.. B: XXXXXXX... ...XXXXXXX AB: XXXXXXX... ...XXXXX.. supp 7 5

h(tid)=1 Best case Worst case A: XXXX...... XXXX...... B: XXXXX..... .....XXXXX AB: XXXX...... .......... supp 4 0

h(tid)=2 Best case Worst case A: XXXXXX.... XXXXXX.... B: XXXXXX.... ....XXXXXX AB: XXXXXX.... ....XX.... supp 6 2

Each pair of columns represents the transactions having a tid mapped into the corresponding location by the hash function. The lower and upper bounds for the support of pattern AB will be respectively 5 + 0 + 2 = 7 and 7 + 4 + 6 = 17. Both lower bounds and upper bounds computations can be extended to larger itemsets by associativity: the bounds for the first two items are composed with the third element counters and so on. The sums of the elements of the last pair of resulting arrays will be the upper and the lower bounds for the given pattern. This is possible since the reasoning previously explained still holds if we considers the occurrences of itemsets instead of those of single items. The lower bound computed in this way will be often equal to zero in sparse dataset. Conversely, on dense datasets this method did proved to be effective in narrowing the two bounds. Experimental evaluation In the final part of this section, we study the behavior of the proposed method. We have run the APStream algorithm on several datasets using different parameters. The goal of these tests is to understand how similarities of the results vary as the stream length increases, the effectiveness of the hash-based pruning, and, in general, how dataset peculiarities and invocation parameters affect the accuracy of the results. Furthermore, we studied how execution time evolves in time when the stream length increases.

5.3. Frequent itemsets

93

Similarity and Average Support Range. The method we are proposing yields approximate results. In particular APStream computes pattern supports which may be slightly different from the exact ones, thus the result set may miss some frequent patterns (false negatives) or include some infrequent patterns (false positives). In order to evaluate the accuracy of the results we use a widely used measure of similarity between two pattern sets introduced in [50], and based on support difference. To the same end, we use the Average support Range (ASR), an intrinsic measure of the correctness of the approximation introduced in [61]. An extensive description of both measures and a discussion on their use can be found in the appendix A. Experimental data. We performed several tests using both real world datasets, mainly from the FIMI’03 contest [1], and synthetic dataset generated using the IBM generator. We randomly shuffled each dataset and used the resulting datasets as input streams. Table 5.2 shows a list of these datasets along with their cardinality. The datasets having the name starting with T are synthetic datasets, which mimic the behavior of market basket transactions. The sparse dataset family T20I8N5k has transactions composed, on average, of 20 items, chosen from 5000 distinct items, and include maximal patterns whose average length is 8. The dataset family T30I30N1k was generated with the parameters synthetically indicated in its name and is composed of moderately dense datasets, since more than 10,000 frequent patterns can be extracted even with a minimum support of 30%. A description of all other datasets can be found in [1]. Kosarak and Retail are really sparse datasets, whereas all other real world dataset used in experimental evaluation are dense. Table 5.2 also indicates for each dataset a short identifying code that will be used in our charts. Dataset Reference accidents A kosarak K retail R pumbs P pumbs-star PS connect C T20I8N5k S2..6 T25I20N5k S7..11 T30I30Nf1k D1..D9

#Trans. 340183 990002 88162 49046 49046 67557 77302..3189338 89611..1433580 50000..3189338

Table 5.2: Datasets used in experimental evaluation.

Experimental Results. For each dataset and several minimum support thresholds, we computed the exact reference solutions by using DCI [44], an efficient sequential algorithm for frequent pattern mining (FPM). Then we ran APStream for

94

5. Streaming data

different values of available memory and number of hash entries. The first test is focused on catching the effect of used memory on the behaviour of the algorithm when the block of transactions processed at once is sized dynamically according to the available resources. In this case, data are buffered as long as all the item counters, and the representation of the transactions included in the current block fit into the available memory. Note that the size of all frequent itemsets, either mined locally or globally, is not considered in our resource evaluation, since they can be offloaded to disk if needed. The second test is somehow related to the previous one. In this case, the amount of required memory is variable, since we determine a-priori the number of transactions to include in a single block, independently of the stream content. Since the datasets used in the tests are quite different, in both cases we used really different ranges of parameters. Therefore, in order to fit all the datasets in the same plot, the numbers reported in the horizontal axis are relative quantities, corresponding to the block sizes actually used in each test. These relative quantities are obtained by dividing the memory/block size used in the specific test by the smallest one for that dataset. For example, the series 50KB, 100KB, 400KB thus becomes 1,2,8. The first plot in figure 5.1 shows the results obtained in the fixed memory case, while the second one refers to the case of a fixed number of transactions per block. The relative quantities reported in the plots refer to different base values of either memory or transactions per blocks. These values are reported in the legend of each plot. In general, when we increase the number of transaction processed at once, either statically or dynamically on the basis the memory available, we also improve the results similarity. Nevertheless, the variation is in most cases small and sometimes there is a slightly negative trend caused by the nonlinear relation between used memory and transactions per block. In our test we noted that choosing an excessively low amount of available memory for some datasets lead to performance degradation and sometimes also to similarity degradation. The last plot shows the effectiveness of the hash-based bounds on reducing the Average Support Range (zero corresponds to an exact result). As expected the improvement is evident only on more dense datasets. The last batch of tests makes use of a family of synthetic datasets with homogeneous distribution parameters and varying lengths. These datasets are obtained from the larger dataset of the serie by truncating it to simulate streams with different lengths. For each truncated dataset we computed the exact result set, used as reference value in computing the similarity of the corresponding approximate result obtained by APStream . The first chart in figure 5.2 plots both similarity and ASR as the stream length increases. We can see that similarity remains almost the same, whereas the ASR decreases when an increasing amount of stream is processed. Finally, the last plot shows the evolution of execution time as the stream length increases. The execution time increases linearly with the length of the stream, hence the average time per transaction is constant if we fix the dataset and the execution parameters.

5.4. Conclusions

95

Similarity(%)

Similarity(%)

95

95

90

90

85

85

80

80 %

100

%

100

75

75

70

70 A (minsupp= 30%, base mem=2MB) C (minsupp= 70%, base mem=2MB) P (minsupp= 70%, base mem=5MB) PS (minsupp= 40%, base mem=5MB) R (minsupp= 0.05%, base mem=5MB) K (minsupp= 0.1%, base mem=16MB)

65 60 55 1

2

4 Reletive available memory

A (minsupp= 30%, base trans/block=10k) C (minsupp= 70%, base trans/block=4k) K (minsupp= 0.1%, base trans/block=20k) P (minsupp= 70%, base trans/block=8k) PS (minsupp= 40%, base trans/block=4k) R (minsupp= 0.05%, base trans/block=2k)

65 60 55

8

16

1

2 4 8 Relative transaction number per block

16

Average support range(%) 5

A (minsupp= 30%) C (minsupp= 70%) K (minsupp= 0.1%) P (minsupp= 70%)

4.5 4 3.5

%

3 2.5 2 1.5 1 0.5 0 0

100

200 Hash entries

300

400

Figure 5.1: Similarity and Average Support Range as a function of available memory, number of transactions per block, and number of hash entries.

Acknowledgment The datasets used during the experimental evaluation are some of those used for the FIMI’03 (Frequent Itemset Mining Implementations) contest [1]. Thanks to the owners of these data and people who made them available in current format. In particular Karolien Geurts [21] for Accidents, Ferenc Bodon for Kosarak, Tom Brijs [10] for Retail and Roberto Bayardo for the conversion of UCI datasets. Other datasets were generated using the publicly available synthetic data generator code from the IBM Almaden Quest data mining project [6].

5.4

Conclusions

In this chapter we have discussed APStream , a new algorithm for approximate frequent pattern mining on streams, and described several related algorithms for frequent item and itemset mining. APStream exploits a novel interpolation method to infer the unknown past counts of some patterns, which are frequents only on recent data. Since the support values computed by the algorithm are approximate, we have also proposed a method for establishing a pair of upper and lower bounds

96

5. Streaming data

dataset: T30I30N1k min_supp=30%

dataset: T30I30N1k min_supp=30%

100

0.2 32

99.8

0.1 99.4

16 Relative time

99.6

ASR (%)

Similarity (%)

0.15

8

4

0.05 99.2

2 Similarity ASR

99 1

2

4 8 Stream length (/100k)

16

0 32

relative time

1 1

2

4 8 Stream length (/100k)

16

32

Figure 5.2: Similarity and Average Support Range as a function of different stream lengths.

for each interpolated value. These bounds are computed using the knowledge of subpattern frequency in past data and the intersection of a hash based compressed representation of past data. Experimental tests show that the solution produced by APStream is a good approximation of the exact global result. The comparisons with exact results consider both the set of patterns found and their support. The metric used in order to assess the quality of the algorithm output is the similarity measure introduced in [50]. The interpolation works particularly well for dense dataset, achieving a similarity close to 100% in best cases. The adaptive behaviour of APStream allows us to limit the amount of used memory. As expected, we have found that a larger amount of available memory corresponds to a more accurate result. Furthermore, as the length of the processed stream increases, the similarity with the exact result remains almost the same. At the same time, we observed a decrease in the average difference between upper and lower bounds, which is an intrinsic measure of result accuracy. This means that when the stream length increase, the relative bounds on support get closer. Finally, the time needed to process a block of transactions does not depend on the stream length, hence the total execution time is linear with respect to the stream length. In the future, we plan to improve the proposed method by adding other stricter bounds on the approximate support and to extend it to closed patterns.

Conclusions The knowledge discovery process and, particularly, its data mining algorithmic part, have been extensively studied in the literature during the last twenty years, and is still an active discipline. Several problem and analysis methods have been proposed, and the extraction of valuable and hidden knowledge from operational databases is, currently, a strategic issue for most medium and large companies. Most of these organizations are geographically spread by nature, and distributed database systems are widely diffused due to either logistic, failure resilience, or performance reasons. Banks, telecommunication companies, wireless access provider, are just some of the users of distributed system for the management of both historical and operational data. Furthermore, in several cases, the data are produce and/or modified continuously and at a sustained rate. The usage of data mining algorithms in distributed and stream settings may introduce several challenging issues. Problems may be either technical, related to the network infrastructure and the huge amount of data, political, related to privacy, company interest or ownership of data. The issues to solve, however, depend on the kind of knowledge we are interested to extract from data. In this thesis, we have analyzed in detail the issues related to the Association Rules Mining, and more precisely to its most computationally expensive phase, the mining of frequent patterns in distributed dataset and data stream, where these patterns can be either itemsets (FIM) or sequences (FSM). The core contribution of this work is a general framework for adapting an exact Frequent Pattern Mining algorithm to a distributed or streaming context. The resulting algorithms are able to find efficiently an approximation of the exact results with a strong reduction of communication size, in the distributed case, and memory usage, in the stream case. In both cases, the approximate support of each pattern is returned along with an interval containing the true value. The proposed methods have been evaluated in a distributed setting and in a stream one, using several real world and synthetic datasets. The results of our test show that this framework gives a fairly accurate approximation of exact results, even only exploiting simple and generic interpolation schemas as those used in the tests. In the distributed case, the interpolation based method exhibits linear speedup as the number of partitions increases. In the stream case, the time that is required to process a block is on average constant, hence the total execution time is linear with respect to the length of the data stream. At the same time, both the similarity to the exact results and the absolute width of the error interval are almost constant. Thus, the algorithm is suitable for mining infinite amount of data. One further original contribution presented in this thesis is an algorithm for

100

Conclusions

frequent sequence mining with gap constraints. CCSM is a novel algorithm for the discovery of frequent sequence patterns with constraints on the maximum gap between the occurrences of two part of the sequence (maxGap). The proposed algorithm has been compared with cSPADE, a state of the art algorithm, obtaining better performance result for significant value of the maxgap constraint. Thanks to the particular transversal order of the search space exploited by CCSM, the intermediate results are highly reused, and the output is ordered. This is particularly important and allows to efficiently integrate the CCSM algorithm in the proposed distributed/stream framework, as explained in the next section.

Future works Frequent Sequence Mining on distributed/stream data The methods presented for frequent itemset extraction can easily be extended to the other kind of frequent patterns considered in this thesis: the frequent sequences. This only involves minor modifications of the algorithms: replacing the interpolation formula with one suitable for sequences, and the FIM algorithm with a FSM algorithm. The CCSM algorithm is a suitable FSM candidate to be inserted in our distributed and stream framework, since it is level-wise and returns ordered set of frequent sequences. This ordering allows for merging on-the-fly the sequence patterns as they arrive, and the level-wise behavior makes more information available to be exploited by the interpolation schema in order to give a better approximation. Furthermore, the on-the-fly merge reduces both memory requirement and computational cost of the merge phase. As the overall framework remains exactly the same, all the improvements and limits that we have explained for frequent itemsets are still valid. The only differences are those originated by the intrinsic difference between frequent itemset and frequent sequences, which make the result of FSM potentially larger and more likely to be affected by combinatorial explosion. Frequent Itemset Mining on distributed stream data The proposed merge/interpolation framework can be extended seamlessly to manage distributed streams in several ways. The most straightforward one is based on the composition of APInterp , followed by APStream . Each slave is responsible for extracting frequent itemsets from its local streams. The results of each processed block are sent to the master and merged, first among them using APInterp , and then with the past combined results as in APStream . The schema on the left of Figure III illustrates this framework. Resnode,i is the FIM result on the ith block of the node stream, whereas Resi is the result of the merge of all local ith results, and Hist Resi is the historical global result, i.e., from the beginning to the ith block.

Conclusions

101

Figure C.3: Distributed stream mining framework. On the left distributed merge followed by stream merge, on the right local stream merge followed by distributed merge.

A first improvement on this base idea could be the replacement of the two cascaded merge phases, one distribution related and the other stream related, with a single one. This would allow for better accuracy of results and stricter bounds, thanks to the reduction of cumulated errors. Clearly, the recount step, used in APStream for assessing the support of recently non-frequent itemsets that were frequent in past data, is impossible in both cases. Since the merge is performed in the master node, only the received locally frequent patterns are available. However, this step proved to be effective in our preliminary tests on APStream , particularly for dense datasets. In order to introduce the local recount phase, it is necessary to move the stream merge phase to the slave nodes. In this way, recent data are still available in the reception buffer, and can be used to improve the results. Each slave node then sends its local results, related to the whole history of its streams, to the master node that simply merges them like in APInterp . Since these results are sent each time a block is processed, it would be advisable to send only the differences in the results related to the last processed block. This involves rethinking the central merge phase but, in our opinion, it should yield good results. The schema on the right of Figure III

102

Conclusions

illustrates this framework. DCI result streams are directly processed by APStream , yielding Hist Resnode,i , i.e. the results on the whole node stream at time i. APInterp collect these results and output the final result Hist Resi . The last aspect to consider is synchronization. Each stream evolves, potentially at a different rate with respect to other streams. This means that when the stream reception buffer of a node is full other nodes could be still collecting data. Thus, the collect and merge framework should allow for asynchronous and incremental result merge, with some kind of forced periodical synchronization, if needed. Limiting the combinatorial explosion of the output It should be noted that, both in the distributed and in the stream settings, the actual time needed to process a partition is mainly related to the statistical properties of data. This problem is not specific to our algorithms. Instead, it is a peculiarity of the frequent itemset/sequences problems, and is directly linked to the exponential size of the result sets. Our goal was to find an approximate solution as close as possible to the exact one, and this is exactly what we achieved. However, this means that in case the exact solution is huge, the approximate solution will be huge too. In this case, if we want to ensure that data can be processed at a given rate, choosing a different approach is mandatory. Two approaches can be devised: the first one is based on alternative representations of results, such as closed/condensed/maximal frequent patterns. As quickly explained in the related works of chapter 2, both the result size and the information on support of patterns decrease from the first to the last of the three problems, but the presence of a pattern in the results is always certain. The second one, instead, aim at discovering only a useful subset of the result, as in the case of alignment patterns [31]. We have done some preliminary work on approximate distributed closed itemset mining [32], but also the second approach will be matter of further investigations. We believe it should be particularly effective in the sequence case, which is more affected by the combinatorial explosion problem.

A Approximation assessment The methods we are proposing yields approximate results. In particular APInterp computes pattern supports which may be slightly different from the exact ones, thus the result set may miss some frequent pattern (false negatives) or include some infrequent pattern (false positives). In order to evaluate the accuracy of the results we need a measure of similarity between two pattern sets. A widely used one has been introduced in [50], and is based on support difference. Definition 12 (Similarity). Let A and B respectively be the reference (correct) result set and the approximate result set. supA (x) ∈ [0, 1] and supB (y) ∈ [0, 1], where x ∈ A and y ∈ B, correspond to the relative support found in A and B respectively. Note that since B corresponds to the frequent patterns found by the approximate algorithm under observation, A − B thus corresponds to the set of false negatives, while B − A are the false positives. The Similarity is thus computed as P max{0, 1 − α ∗ |supA (x) − supB (x)|} Simα (A, B) = x∈A∩B |A ∪ B| where α > 1 is a scaling parameter, which increase the effect of the support dissimilarity. Moreover, α1 indicates the maximum allowable error on (relative) pattern supports. We will use the notation Sim() to indicate the default case for α, i.e. α = 1. In case absolute supports are used instead than relative ones, the parameter α will be smaller than or equal to 1. We will name this measure Absolute Similarity, indicated as SimABS (A, B). This measure of similarity is thus the sum of at most |A ∩ B| values in the range [0, 1], divided by |A ∪ B|. Since |A ∩ B| 6 |A ∪ B|, similarity lies in [0, 1] too. When a pattern appears in both sets and the difference between the two supports is greater than α1 , it does not improve similarity, otherwise similarity is increased according to the scaled difference. If α = 20, then the maximum allowable error in the relative support is 1/20 = 0.05 = 5%. Supposing that the support difference for a particular pattern is 4%, the numerator of the similarity measure will be increased by a small quantity: 1 − (20 ∗ 0.04) = 0.2. When α is 1 (default value), only patterns

104

A. Approximation assessment

whose support difference is at most 100% contribute to increase similarity. On the other hand, when we set α to a very high value, only patterns with a very similar supports in both the approximate and reference sets will contribute to increase the similarity measure (which is roughly the same than using Absolute Similarity with α close to 1). It is worth noting that the presence of several false positives and negatives in the approximate result set B contributes to reduce our similarity measure, since this entails an increase in A ∪ B (the denominator of the Simα formula) with respect to A∩B. Moreover, if a pattern has an actual support that is slightly less than minsup but the approximate support (supB ) is slightly greater than minsup, similarity is decreased even if the computed support was almost correct. This could be an undesired behavior. While a false negative can constitute a big issue, because some potentially important association rules will be not generated at all, a false positive with a support very close to the exact one could be tolerated by an analyst. In order to overcome this issue we propose a new similarity measure, f pSim (where f p stand for false positive). Since this measure consider every pattern included in the approximate result set B (instead of A ∩ B), it can be used in order to assess whether false positives have an approximate support value close to the exact one or not. A high value of f pSim compared with a smaller value of Sim simply means that in the approximate result set B there are several false positive with a true support close to minsup. Definition 13 (fpSimilarity). Let A and B respectively be the reference (correct) result set and the approximate result set. supB (x) ∈ [0, 1], where x ∈ B, corresponds to the support found in result sets B, while sup(x) ∈ [0, 1] is the actual support of the same pattern. fpSimilarity is thus computed as P max{0, 1 − α ∗ |sup(x) − supB (x)|} fpSimα (A, B) = x∈B |A ∪ B| where α > 1 is a scaling parameter. We will use the notation Sim() to indicate the default case for α, i.e. α = 1. Note that the numerator of this new measure considers all the patterns found in the set B, thus also false positives. Hence finding a pattern with a support close to the true one is considered a ”good” result in any case, even if this pattern is not actually frequent. For example, suppose that minimum support threshold is 50% and x is an infrequent pattern such that sup(x) = 49.9. If supB (x) = 50%, it will result to be a false positive. However, since supB (x) is very close to the exact support sup(x), the value of f pSimα () will be increased. In Definition 13 we used sup(x) instead of supA (x) to indicate the actual support of itemset x since it is possible, as in the example case, that a pattern is present in B even if it is not frequent (hence not present in A). In both definitions above, we used sup(x) to indicate the (relative) support, ranging from 0 to 1. In the remainder of the paper, in particular in the algorithm

105

description, we will also use the notation σ(x) = sup(x) · |D| to indicate the support count (absolute support), ranging from 0 to the total number of transactions. When bounds on the support of each pattern are available, an intrinsic measure of the correctness of the approximation is the average width of the interval between the upper bound and the lower bound. Definition 14 (Average support range). Let B be the approximate result set, sup(x) the exact support for pattern x and sup(x)lower and sup(x)upper the lower and upper bounds on sup(x), respectively. The average support range is thus defined as: ASR(B) =

1 X sup(x)upper − sup(x)lower |B| x∈B

Note that, while this definition can be used for every approximate algorithm, how to compute sup(x)lower and sup(x)upper is algorithm specific. In the next section, we will present a way that is suitable for the class of algorithms containing the one we are proposing. Other, less accurate, similarity measures can be borrowed from the Information Retrieval theory: Definition 15 (Recall & Precision). Let A and B respectively be the reference (correct) result set and the approximate result set. Note that since B corresponds to the frequent patterns found by the approximate algorithm under observation, A − B thus corresponds to the set of false negatives, while B − A are the false positives. Let P (A, B) ∈ [0, 1] be the Precision of the approximate result, defined as follows: B∩A B Hence the Precision is maximal (P (A, B) = 1) iff B ∩ A = B, i.e. the approximate result set B is completely contained in the exact one A, and no false positives occurs. Let R(A, B) ∈ [0, 1] be the Recall of the approximate result, defined as follows: P (A, B) =

B∩A A Hence the Recall is maximal (R(A, B) = 1) iff B ∩ A = A, i.e. the exact result set A is completely contained in the approximate one B, and no false negative occurs. R(A, B) =

According to our remarks above concerning the benefits of the f pSim measure (Def. 13), we have that a ”good” approximate result should be characterized by to a very high Recall, where the supports of the possible false positive patterns should be however very close to the exact ones. Conversely, in order to minimize the standard measure of similarity (Def. 12), we need to maximize both Recall and Precision, while keeping small the difference in the approximate supports of frequent patterns.

106

A. Approximation assessment

Bibliography [1] Workshop on frequent itemset mining implementations FIMI’03 in conjunction with ICDM’03. In fimi.cs.helsinki.fi, 2003. [2] R. Agarwal, C. Aggarwal, and V. V. V. Prasad. A tree projection algorithm for generation of frequent itemsets. Parallel and Distributed Computing, 2000. [3] R. Agarwal, C. Aggarwal, and V.V.V. Prasad. Depth first generation of long patterns. In KDD ’00: Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 108–118, New York, NY, USA, 2000. ACM Press. [4] R. Agrawal, T. Imielinski, and A. Swami. Mining association rules between sets of items in large databases. In Proceedings of the 1993 ACM SIGMOD International Conference on Management of Data, pages 207–216, Washington, D.C., 1993. [5] R. Agrawal and J.C. Shafer. Parallel mining of association rules. In IEEE Transaction On Knowledge and Data Engineering, 1996. [6] R. Agrawal and R. Srikant. Fast algorithms for mining association rules. In Proc. 20th Int. Conf. Very Large Data Bases, VLDB, pages 487–499. Morgan Kaufmann, 1994. [7] R. Agrawal and R. Srikant. Mining sequential patterns. In Proc. 11th Int. Conf. Data Engineering, ICDE, pages 3–14. IEEE Press, 1995. [8] J. Ayres, J. Gehrke, T. Yiu, and J. Flannick. Sequential pattern mining using bitmaps. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2002. [9] R. J. Bayardo Jr. Efficiently Mining Long Patterns from Databases. In Proc. of the ACM SIGMOD Int. Conf. on Management of Data, pages 85–93, Seattle, Washington, USA, 1998. [10] T. Brijs, G. Swinnen, K. Vanhoof, and G. Wets. Using association rules for product assortment decisions: A case study. In Knowledge Discovery and Data Mining, pages 254–260, 1999. [11] D. Burdick, M. Calimlim, and J. Gehrke. Mafia: a maximal frequent itemset for transactional databases. In Proc. of the International Conference on Data Endineering ICDE, pages 443–452. IEEE Computer Society, 2001.

108

Bibliography

[12] M. Charikar, K. Chen, and M. Farach-Colton. Finding frequent items in data streams. In ICALP ’02: Proceedings of the 29th International Colloquium on Automata, Languages and Programming, pages 693–703, London, UK, 2002. Springer-Verlag. [13] D.W. Cheung, J. Han, V.T. Ng, A.W. Fu, and Y. Fu. A fast distributed algorithm for mining association rules. In DIS ’96: Proceedings of the fourth international conference on on Parallel and distributed information systems, pages 31–43, Washington, DC, USA, 1996. IEEE Computer Society. [14] G. Cormode and S. Muthukrishnan. What’s hot and what’s not: tracking most frequent items dynamically. In PODS ’03: Proceedings of the twentysecond ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, pages 296–306. ACM Press, 2003. [15] G. Cormode and S. Muthukrishnan. An improved data stream summary: the count-min sketch and its applications. J. Algorithms, 55(1):58–75, 2005. [16] E.D. Demaine, A. L´opez-Ortiz, and J.I. Munro. Frequency estimation of internet packet streams with limited space. In ESA ’02: Proceedings of the 10th Annual European Symposium on Algorithms, pages 348–360, London, UK, 2002. Springer-Verlag. [17] C. Estan and G. Varghese. New directions in traffic measurement and accounting: Focusing on the elephants, ignoring the mice. ACM Trans. Comput. Syst., 21(3):270–313, 2003. [18] U. M. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy, editors. Advances in Knowledge Discovery and Data Mining. AAAI Press, 1998. [19] V. Ganti, J. Gehrke, and R. Ramakrishnan. Mining Very Large Databases. IEEE Computer, 32(8):38–45, 1999. [20] M.N. Garofalakis, R. Rastogi, and K. Shim. SPIRIT: Sequential pattern mining with regular expression constraints. In The VLDB Journal, pages 223–234, 1999. [21] K. Geurts, G. Wets, T. Brijs, and K. Vanhoof. Profiling high frequency accident locations using association rules. In Proceedings of the 82nd Annual Transportation Research Board, Washington DC. (USA), January 12-16, page 18pp, 2003. [22] E-H.S. Han, G. Karypis, and V. Kumar. Scalable parallel data mining for association rules. In IEEE Transaction on Knowledge and Data Engineering, 2000.

Bibliography

109

[23] J. Han and M. Kamber. Data Mining: Concepts and Techniques. Morgan Kaufmann Publishers, 1st edition, 2000. [24] J. Han, J. Pei, B. Mortazavi-Asi, Q. Chen, U. Dayal, and M.-C. Hsu. Freespan: Frequent pattern-projected sequential pattern mining. In In Proc. ACM 6th Int. Conf. on Knowledge Discovery and Data Mining, pages 355–359, 2000. [25] J. Han, J. Pei, and Y. Yin. Mining frequent patterns without candidate generation. In Proc. of the ACM SIGMOD Int. Conference on Management of Data, 2000. [26] J.D. Holt and S.M. Chung. Mining association rules using inverted hashing and pruning. Inf. Process. Lett., 83(4):211–220, 2002. [27] V.C. Jensen and N. Soparkar. Frequent itemset counting across multiple tables. In In 4th PAcific Asia Conference on Knowledge Discovery and Data Minig, 2000. [28] C. Jin, W. Qian, C. Sha, J.X. Yu, and A. Zhou. Dynamically maintaining frequent items over a data stream. In CIKM ’03: Proceedings of the twelfth international conference on Information and knowledge management, pages 287– 294, New York, NY, USA, 2003. ACM Press. [29] R. Jin and G. G. Agrawal. An algorithm for in-core frequent itemset mining on streaming data. To appear in ICDM’05, 2005. [30] R.M. Karp, S. Shenker, and C.H. Papadimitriou. A simple algorithm for finding frequent elements in streams and bags. ACM Transactions on Database Systems (TODS), 28(1):51–55, 2003. [31] H. Kum, J. Pei, W. Wang, and D. Duncan. ApproxMAP: Approximate mining of consensus sequential patterns. In Proceedings of the Third International SIAM Conference on Data Mining, 2003. [32] C. Lucchese, S. Orlando, R. Perego, and C. Silvestri. Mining frequent closed itemsets from highly distributed repositories. In Proc. of the 1st CoreGRID Workshop on Knowledge and Data Management in Grids in conjunction with PPAM2005, September 2005. [33] G. Manku and R. Motwani. Approximate frequency counts over data streams. In In Proceedings of the 28th International Conference on Very Large Data Bases, August 2002. [34] H. Mannila and H. Toivonen. Discovering generalized episodes using minimal occurrences. In Knowledge Discovery and Data Mining, pages 146–151, 1996.

110

Bibliography

[35] H. Mannila, H. Toivonen, and A. I. Verkamo. Discovering Frequent Episodes in Sequences. In Proceedings of the First International Conference on Knowledge Discovery and Data Mining (KDD-95), Montreal, Canada, 1995. AAAI Press. [36] H. Mannila, H. Toivonen, and A.I. Verkamo. Discovery of frequent episodes in event sequences. Data Mining and Knowledge Discovery, 1(3):259–289, 1997. [37] F. Masseglia, F. Cathala, and P. Poncelet. The PSP approach for mining sequential patterns. In Principles of Data Mining and Knowledge Discovery, pages 176–184, 1998. [38] F. Masseglia, P. Poncelet, and M. Teisseire. Incremental mining of sequential patterns in large databases. Technical report, LIRMM, France, January 2000. [39] F. Masseglia, P. Poncelet, and M. Teisseire. Incremental mining of sequential patterns in large databases. Data and Knowledge Engineering, 46(1):97–121, 2003. [40] J. Misra and D. Gries. Finding repeated elements. Technical report, Ithaca, NY, USA, 1982. [41] A. Mueller. Fast sequential and parallel algorithms for association rules mining: A comparison. Technical Report CS-TR-3515, Univ. of Maryland, 1995. [42] S. Orlando, P. Palmerini, and R. Perego. Enhancing the apriori algorithm for frequent set counting. In DaWaK ’01: Proceedings of the Third International Conference on Data Warehousing and Knowledge Discovery, pages 71–82, London, UK, 2001. Springer-Verlag. [43] S. Orlando, P. Palmerini, R. Perego, C. Lucchese, and F. Silvestri. kDCI: a multi-strategy algorithm for mining frequent sets. In Proceedings of the workshop on Frequent Itemset Mining Implementations FIMI’03 in conjunction with ICDM’03, 2003. [44] S. Orlando, P. Palmerini, R. Perego, and F. Silvestri. Adaptive and resourceaware mining of frequent sets. In Proc. of the 2002 IEEE International Conference on Data Mining, ICDM, 2002. [45] S. Orlando, P. Palmerini, R. Perego, and F. Silvestri. An efficient parallel and distributed algorithm for counting frequent sets. In Proc. of Int. Conf. VECPAR 2002 - LNCS 2565, pages 197–204. Spinger, 2002. [46] S. Orlando, R. Perego, and C. Silvestri. CCSM: an efficient algorithm for constrained sequence mining. In Proceedings of the 6th International Workshop on High Performance Data Mining: Pervasive and Data Stream Mining, in conjunction with Third International SIAM Conference on Data Mining, 2003.

Bibliography

111

[47] S. Orlando, R. Perego, and C. Silvestri. A new algorithm for gap constrained sequence mining. To appear in Proceedings of ACM Symposim on Applied Computing SAC - Data Mining track, Nicosia, Cyprus, March 2004. [48] B. Park and H. Kargupta. Distributed Data Mining: Algorithms, Systems, and Applications. In Data Mining Handbook, pages 341–358. IEA, 2002. [49] J.S. Park, M.S. Chen, and P.S. Yu. An Effective Hash Based Algorithm for Mining Association Rules. In Proceedings of 1995 ACM SIGMOD Int. Conf. on Management of Data, pages 175–186. [50] S. Parthasarathy. Efficient progressive sampling for association rules. In Proceedings of the 2002 IEEE International Conference on Data Mining (ICDM’02), page 354. IEEE Computer Society, 2002. [51] S. Parthasarathy, M.J. Zaki, M. Ogihara, and S. Dwarkadas. Incremental and interactive sequence mining. In CIKM, pages 251–258, 1999. [52] J. Pei, J. Han, B. Mortazavi-Asl, H. Pinto, Q. Chen, U. Dayal, and M.-C. Hsu. Prefixspan: Mining sequential patterns efficiently by prefix-projected pattern growth. In Proceedings of the 17th International Conference on Data Engineering, page 215. IEEE Computer Society, 2001. [53] J. Pei, J. Han, and W. Wang. Mining sequential patterns with constraints in large databases. In Proc. of Proceedings of the 11-th Int. Conf. on Information and Knowledge Management (CIKM 02), pages 18–25, 2002. [54] N. Ramakrishnan and A. Y. Grama. Data Mining: From Serendipity to Science. IEEE Computer, 32(8):34–37, 1999. [55] A. Savasere, E. Omiecinski, and S.B. Navathe. An efficient algorithm for mining association rules in large databases. In VLDB’95, Proceedings of 21th International Conference on Very Large Data Bases, pages 432–444. Morgan Kaufmann, September 1995. [56] A. Schuster and R. Wolff. Communication Efficient Distributed Mining of Association Rules. In ACM SIGMOD, Santa Barbara, CA, April 2001. [57] A. Schuster, R. Wolff, and D. Trock. A High-Performance Distributed Algorithm for Mining Association Rules. In The Third IEEE International Conference on Data Mining (ICDM’03), Melbourne, FL, November 2003. [58] T. Shintani and M. Kitsuregawa. Hash based parallel algorithms for mining association rules. In PDIS: International Conference on Parallel and Distributed Information Systems. IEEE Computer Society Technical Committee on Data Engineering, and ACM SIGMOD, 1996.

112

Bibliography

[59] C. Silvestri and S. Orlando. Distributed association mining: an approximate method. In Proceedings of 7th International Workshop on High Performance and Distributed Mining, in conjunction with Fourth International S, April 2004. [60] C. Silvestri and S. Orlando. Approximate mining of frequent patterns on streams. In Proc. of the 2nd International Workshop on Knowledge Discovery from Data Streams in conjunction with PKDD2005, October 2005. [61] C. Silvestri and S. Orlando. Distributed approximate mining of frequent patterns. In Proceedings of ACM Symposim on Applied Computing SAC - Data Mining track, March 2005. [62] R. Srikant and R. Agrawal. Mining sequential patterns: Generalizations and performance improvements. In Proc. 5th Int. Conf. Extending Database Technology, EDBT, volume 1057, pages 3–17. Springer-Verlag, 1996. [63] R. Wolff and A. Schuster. Mining Association Rules in Peer-to-Peer Systems. In The Third IEEE International Conference on Data Mining (ICDM’03), Melbourne, FL, November 2003. [64] X. Yan, J. Han, and R. Afshar. Clospan: Mining closed sequential patterns in large datasets. In Proc. 2003 SIAM Int.Conf. on Data Mining (SDM’03), 2003. [65] M.J. Zaki. Fast mining of sequential patterns in very large databases. Technical Report TR668, University of Rochester, Computer Science Department, 1997. [66] M.J. Zaki. Parallel and distributed association mining: A survey. In IEEE Concurrency, 1999. [67] M.J. Zaki. Parallel sequence mining on shared-memory machines. In LargeScale Parallel Data Mining, pages 161–189, 1999. [68] M.J. Zaki. Scalable algorithms for association mining. IEEE Transactions on Knowledge and Data Engineering, 12:372–390, May/June 2000. [69] M.J. Zaki. Sequence mining in categorical domains: incorporating constraints. In Proceedings of the ninth international conference on Information and knowledge management, pages 422–429. ACM Press, 2000. [70] M.J. Zaki. Spade: An efficient algorithm for mining frequent sequences. Machine Learning, 42(1-2):31–60, 2001.

List of PhD Thesis TD-2004-1 Moreno Marzolla ”Simulation-Based Performance Modeling of UML Software Architectures” TD-2004-2 Paolo Palmerini ”On performance of data mining: from algorithms to management systems for data exploration” TD-2005-1 Chiara Braghin ”Static Analysis of Security Properties in Mobile Ambients” TD-2006-1 Fabrizio Furano ”Large scale data access: architectures and performance” TD-2006-2 Damiano Macedonio ”Logics for Distributed Resources” TD-2006-3 Matteo Maffei ”Dynamic Typing for Security Protocols” TD-2006-4 Claudio Silvestri ”Distributed and Stream Data Mining Algorithms for Frequent Pattern Discovery”