Optimal Redesign Policies to Support Dynamic ... - Semantic Scholar

Optimal Redesign Policies to Support Dynamic Processing of Applications on a Distributed Relational Database System1 Kamalakar Karlapalem

Department of Computer Science University of Science and Technology Clear Water Bay, Kowloon, Hong Kong. Phone: + 852 358 6991 Fax: + 852 358 1477 e-mail: [email protected] Shamkant B. Navathe

Mostafa Ammar

College of Computing Georgia Institute Of Technology Atlanta, GA 30332-0280 e-mail:fsham,[email protected]

Technical Report HKUST-CS94-16, September 1994. This research has been funded by RGC CERG grant No. HKUST 609/94E and NSF Grant No. IRI 8716798. 1

1

Abstract An application processing center consists of a set of well-de ned, well-designed and that are dynamically executed over a period of time. We assume that there is a set of candidate distributed database designs each of which is optimal for some applications. The random execution of applications on a distributed database design is modeled as a discrete Markov process, and the problem of selecting the candidate design for each execution of an application is solved by using sequential Markovian decision process analysis to generate an optimal redesign policy vector. The scope of the methodology developed in this paper is applicable to environments similar to application processing centers. The viability of this methodology is illustrated by means of a case study conducted at Georgia Tech. well-tested applications

1 Introduction Relational database system technology has been under development for more than two decades. There are many ecient commercial relational database systems in existence today. During these two decades most of the high throughput business database processing was done on either hierarchical database management systems (e.g., IMS), or network database management systems (e.g., IDMS and DMS1100), or at le systems. The relational database technology is at a stage where it can eciently process high throughput database applications. There are various reasons for this:

 relational database system technology has matured, there are very ecient commercial

implementations of relational database systems. These systems are being widely used for day to day data processing.  relational databases support non-navigational data access (meaning that the relationships between the database instances are not hard{coded in the implementation model) that enables easy conceptualization and implementation of a variety of distributed relational database con gurations using client-server facilities.  with the advent of standards for remote data access (RDA) and SQL access, it is now possible to access data from dierent relational database systems.  powerful multiprocessor based high end workstations have further enhanced the power of the relational database technology; a clear separation between the logical and physical levels facilitates ecient parallel processing of database operations.

It is possible that distributed relational database technology will be used to support large scale business data processing [OV91a] extensively by the end of this decade. Most of the research in distributed relational database systems has concentrated on the systems aspect; not much research has been done on the design and management of distributed databases [OV91b]. Therefore, the users of the distributed relational database technology will face the distributed database design/redesign problem fairly soon. In this paper, we concentrate on supporting an application processing center based on a distributed database system. The application processing center consists of applications that 2

need to be executed eciently. Because of the diverse data requirements of these applications, no single distributed database design will be able to support all the applications eciently. Therefore, the relational distributed database may need to be frequently redesigned for all the applications to be executed eciently. By eciently, we mean the minimum amount of time taken to execute all the applications over a period of time including the time taken to redesign the distributed database. We present a methodology to support ecient processing of applications in an application processing center based on a distributed relational database system. This methodology consists of redesigning the distributed database whenever necessary to eciently process all the applications. The methodology generates an optimal redesign policy specifying whether or not to redesign a distributed database. If a redesign is necessary, the policy speci es which distributed database design to use. This methodology has been used for a case study using administration databases at Georgia Institute of Technology. The rest of the paper is organized as follows: section 2 presents the related work in distributed relational database design/redesign methodologies, section 3 describes a model for dynamic execution of applications in an application processing center, section 4 describes the methodology for calculating application processing cost required for generating optimal redesign policy vectors, section 5 describes by means of an example the process of generating optimal policy vectors, section 6 presents a case study illustrating the applicability and viability of the proposed methodology for generating optimal redesign policy vectors. Finally, section 7 presents a summary and the conclusions.

2 Related Work In any distributed database system there are two main factors that aect the performance of the applications (set of transactions), namely, the number of disk accesses and data transfer cost or communication delay [CP84]. For a given distributed database environment (DDE) the performance of applications is dependent on the distributed database design (henceforth, referred to as the design). There are two ways by which the performance of the applications can be improved by a design: fragmentation and allocation. Fragmentation is the process of clustering relevant columns and tuples accessed by an application into a unit called a fragment. This reduces the amount of irrelevant data accessed (thus, number of disk accesses) by the application. A fragmentation scheme partitions a relation either vertically (by clustering columns) or horizontally (by clustering rows). Algorithms have been developed to fragment a relation vertically [NCWD84, NR89, MCVN93, and others] or horizontally [CNP82, CNW83, and others]. Allocation is the process of placing the fragments generated by the fragmentation scheme at the sites of the distributed database system so as to minimize the data transfer cost and the number of messages needed to process a given set of applications [Ape88, CN83, and others]. The result of an allocation process is an allocation scheme. The design consists of a fragmentation scheme and an allocation scheme. In [NKR94], we developed a mixed fragmentation methodology which consists of fragmentation and allocation algorithms as shown in Figure 1. Mixed fragments are generated by (i) 3

vertically fragmenting a relation [NR89], (ii) horizontally fragmenting a relation, (iv) generating grid cells by simultaneously applying vertical and horizontal fragmentation schemes, and (v) merging the resulting grid cells, to create larger fragments so as to reduce the overall transaction processing cost. There are four dierent kinds of parameters (shown in Figure 1), namely, logical parameters, transaction parameters, database system parameters, and system parameters that are used by these algorithms. These parameters are used by the algorithms to generate the fragmentation and allocation schemes, to merge the resulting grid cells to form a set of mixed fragments and to allocate these mixed fragments by using the allocation algorithms. We shall use mixed fragmentation methodology to generate candidate designs for the preventive redesign methodology. A tool based on this methodology to design distributed databases has been implemented at the Hong Kong University of Science and Technology [Chu94, Man94]. Logical Parameters • Relational Schema • Application Processing Characteristics

Fragmentation Algorithms Vertical Horizontal Horz_Frag( ) Vert_Frag( )

Transaction Parameters • Columns Accessed • Rows Accessed (Predicates)

Grid_Opt( )

• Frequency • Sites of Origin Allocation Algorithm

Database System Parameters • Cardinality of Relations • Column Sizes • Indexes Used • Query Processing Strategy

Grid Optimization

Mixed Fragments (Fragmentation Scheme)

Alloc( ) Allocation Scheme Distributed Database Design

System Parameters • Network Topology • Baud Rates • Node Parameters (MIPS, Disk I/O time)

Materialization Algorithm (for Redesign) Query_generator( )

Figure 1: Distributed Database Design Methodology

2.1 Redesign of Distributed Relational Databases Distributed database environment (DDE) is a dynamic system; there are many physical and logical changes that cause a deterioration in the performance of the applications. For each application there is a design that can be generated using the mixed fragmentation methodology. The problem is to redesign the distributed database so as to eciently process the application stream over a period of time on a distributed database system. The redesign problem was classi ed by Wilson and Navathe [WN86] into limited redesign and total redesign. In limited redesign, only the allocation scheme changes but not the fragmentation scheme. In the case of total redesign both the allocation and fragmentation schemes change. The limited redesign problem was addressed in [WN86], wherein an iterative algorithm was developed to reallocate the fragments to the sites of the DDE based on a change in the relative distribution of the transactions across the various sites. The redesign process can be corrective, preventive, or adaptive. Corrective redesign is 4

easy because all the changes in the DDE are known beforehand. Preventive redesign is dicult because it requires a priori knowledge about what changes will take place and when in DDE. Adaptive redesign [Kar94] is a dicult problem because DDE has to be monitored to detect the changes taking place, the relevance of these changes need to be evaluated, and then the redesign of the distributed databases has to be initiated. Any change in the design requires that the new design be materialized. Materialization refers to a global reorganization of the local databases to conform to the new fragmentation and allocation scheme. Materialization due to limited redesign has been addressed by [RVVN90], and materialization of totally redesigned distributed relational databases was addressed by [KN94]. Note that materialization of a design is a costly operation [KN94] and hence plays an important role in deciding whether or not to redesign. This research is the rst step in our work towards developing distributed relational database design/redesign methodologies which are very much required [OV91a].

3 A Model for an Application Processing Center In this section, we shall describe a model for an application processing center. The components of this model are the applications and the designs. The dynamic application processing behavior is modeled by using discrete Markov processes. The interaction between the applications and the designs is modeled by using sequential Markovian decision process (SMDP) which is also used to generate the optimal redesign policies.

3.1 Application Processing Center An application processing center is a data center which has computers and database systems that support processing of dierent applications. In this paper, we concentrate on supporting application processing centers based on a distributed relational database system. An application processing center is the life line for a business organization such as a bank, insurance company, utility company's billing and accounting processing, retailing and warehousing company, manufacturing and distribution organization, etc. A representative example of an application processing center is that of a bank or an insurance company that processes various classes of transaction oriented database applications. Examples of applications are those executed daily like day time applications for processing loans, or creating new accounts, versus night time batch applications for generating consolidated teller machine activities. Other applications include weekly salary processing, monthly account statement generation, and yearly W-2 form generation. These applications are well-de ned, well-designed and well-tested. It is very critical for the organization that these applications be executed in a timely and ecient manner. This would require a design that is optimal for all the applications. An optimal design is a design that gives the least average transaction response time for the entire set of applications. But the optimality of a single design for all the applications cannot be guaranteed. Because of the dierent data and processing requirements for each application, there will exist an optimal design for each application which will outperform a single optimal design for all the applications. In order to execute all the applications eciently, it may be necessary to redesign the distributed database so 5

that an application is always executed on an optimal design generated for it. Note that not all applications are executed at the same time. Typically, there is a schedule for executing the applications, where each application is processed during some intervals of time. This dynamic application processing behavior forms a case of preventive redesign wherein the changes in the distributed database environment are known but when these changes take place is not known. A preventive redesign methodology is developed by modeling the dynamic application processing behavior so as to generate the optimal redesign policies.

3.2 Applications Each application is de ned by a set of transactions initiated by it. A transaction accesses one or more relations of the distributed database. In an application processing center an application can be any one of the following: Large scale applications A large integrated module of applications (both batch and online) that support a wide variety of functions. Large applications can be classi ed on whether the application is executed as a whole or whether it is broken up into modules each of which is executed as a whole application. For example, a bank-teller machine application consists of a set of application modules which are executed together as a single large application. But the student information system of a university consists of a set of independent modules which can be rede ned as applications such as \registration module", \grade reporting module" and \class scheduling module". Each module is executed independently at some pre-speci ed time points during the time interval. Small scale applications Small scale applications are developed based on some business need, and are executed frequently. If these kind of applications access large amount of resources in a DDE or if their performance is very crucial for the business data processing then they need to be incorporated as an application for the preventive redesign methodology. In fact, any set of well de ned transactions in the DDE which take up a large amount of resources, and need to be executed eciently at dierent intervals of time can be grouped as an application for the purposes of this redesign methodology. Most database management systems provide utilities to collect and report data regarding resource consumption by the transactions, frequency of their initiation and the relations they access. This information can be used by the distributed database administrators to de ne these applications. Granularity of applications In an application processing center at any point of time there can be more than one executing application. Because of this we need to consider the concept of de ning a class of applications and come up with the candidate designs for classes of applications. Intuitively, a class of applications is a set of applications that are frequently executed together over a period of time. A class of applications can be de ned by observing the applications being executed over a period of time and clustering those applications that have high anity of being executed together. In an environment where applications are all prede ned and have a schedule for their execution it is possible for the distributed database administrator to de ne classes of applications. Overlapping classes of applications will have common data access characteristics which can potentially reduce the number of candidate designs and the cost of materialization. We shall make no distinction between a class of applications and an application for the rest of this paper. 6

Given A1 ; A2; : : :; An, where each Ai is an application. The main characteristic of an application processing center is that at some pre-speci ed time points (known as change points) an application is initiated and the time duration of its execution known (more on this in Section 3.2.1). Given that an application is being executed currently; the application to be executed at the next change point can be any one of the applications fA1; A2; : : :; An g. At each change point there is a probability for selecting one of the applications from fA1 ; A2; : : :; An g to be executed next. This dynamic process of applications being executed after a change point is modeled as a discrete Markov process. A discrete Markov process consists of a set of states and transitions from one state to another are de ned by the state transition probability matrix (stochastic matrix). A stochastic matrix P = [pij ] gives the probability of executing an application Aj after a change point given that the application Ai is currently being executed before the change point, where 1  i; j  n.

Example 1 Consider an application processing center with three applications A1; A2, and A3. Let the currently executing application be A1 , suppose that application A1 will continue executing with probability 0.8, or application A3 will be executed with probability 0.2 after the change point. Similarly, suppose that the application A2 will continue to be executed with probability 0.8, or application A1 will be executed with probability 0.2 after the change point, and that application A3 will continue to be executed with probability 0.8, or application A2 will be executed with probability 0.2 after the change point. This is speci ed by the following stochastic matrix. 3 2 0:8 0:0 0:2 P = 64 0:2 0:8 0:0 75

0:0 0:2 0:8

2

In order to use this model of application processing center we brie y describe the concepts of change points and execution durations. Furthermore, we brie y describe how the stochastic matrix can be generated empirically by monitoring the application processing center.

3.2.1 Change points and Execution Duration In any application processing center applications are executed according to a well planned schedule. This schedule is governed by the requirements for processing of the data due to various business policies. The schedule gives the time points when the applications are initiated. Since not all applications are executed all the time, there is an execution duration for each initiation of an application. The execution duration for each initiation of an application can be tabulated and averages calculated. Each initiation of an application is treated as a change point because the distributed database system has to eciently process the new set of transactions in addition to those it is already processing. Similarly, termination of the execution of an application is considered as a change point. An execution duration for an application is the time interval between its initiation and termination. Large on-line or batch applications tend to have longer execution durations than small applications or transactions. Hence the distributed database administrators have to evaluate all 7

the applications based on execution duration, resources consumed, and their importance to the business processing. This evaluation helps in short listing and de ning the applications, identifying change points and calculating execution durations for these applications. Note that the complexity of the preventive redesign methodology depends on the number of applications and change points, therefore it is preferable to short list the number of applications based on 80-20 rule (i.e. 80% of all processing is accounted by 20% of applications).

3.2.2 Generating the Stochastic Matrix The dynamic process of initiating and executing the applications in the application processing center is modeled as a discrete Markov process. In order to de ne this Markov process, the stochastic matrix P giving the probability of change from executing application \before the change point" to \after the change point" needs to be generated. The previous two subsections described how the distributed database administrators de ne a set of applications which need to be eciently executed in an application processing center. The relative frequencies of applications initiated after the change point are used to calculate the probability of change de ning the stochastic matrix. This is an empirical approach because it is based on monitoring the processing of applications and collecting the statistics. The distributed database administrator can also de ne the stochastic matrix based on a well-planned schedule for executing the applications.

3.3 Designs Each application has an optimal design on which to execute. As the applications are well de ned, the transactions initiated, the frequencies with which they are initiated from dierent sites, the columns accessed, the predicates used to access the tuples of the relations, the average length of a tuple in a relation, and the cardinality of the relation are all known in advance. This information is sucient for generating the optimal design based on a mixed fragmentation methodology [NKR94] for an application. The mixed fragmentation methodology generates the optimal distributed database design for each application Ai . This optimal design eciently processes the application Ai , but may give a very bad performance for other applications, thus motivating the need to redesign. The materialization of redesign is costly and frequent redesign of the distributed database is not cost eective. Therefore, a sub-optimal design which processes both the applications with acceptable performance and eliminates the need for frequent redesign will be cost eective in the long run. Thus, in addition to the optimal designs for each of the applications additional sub-optimal designs for some subsets of applications need to be generated. The stochastic matrix P giving the probabilities of change in the application can be used [Kar92] to decide which subsets of applications need to be considered for generating these suboptimal candidate distributed database designs. The number of candidate designs depends on the variation in the data accessed by the applications. If there is a large amount of variation, then a distributed database design for one application will give bad performance when used for another application thus requiring a large number of candidate designs. But if there is only a little variation then a single 8

design will provide reasonable performance for most of the applications. Let D1; D2; : : :; Dm denote the m candidate designs. Potentially each application Ai can be executed on any one of the D1 ; D2; : : :; Dm designs2 .

3.4 Modelling Applications and Designs In this section, we shall present a model for the interaction of applications executing on designs and the dynamic process of applications executing in a application processing center. Note that every application Ai can be executed on any design Dj where 1  i  n and 1  j  m. Therefore, the application processing center can have at any moment an application Ai executing on design Dj . We now need to model the eect of executing an application on a design after a change point. This is modelled using the Sequential Markovian Decision Process (SMDP) (see [How60]). The states of the SMDP are s1 ; s2; s3; : : :; snm , where n is the number of applications, and m is the number of designs on which the applications can be executed. Each state represents an application executing on a design and is speci ed by a pair identifying the application and the design on which it is executed. Note that an application can use only one out of the m possible designs every time that the application is executed. Therefore after a change point the application selected for execution can execute on any one of the m candidate designs. The decision about which design to select for the application to be executed after the change point has to be taken. This decision has to be taken for each one of the applications A1; A2; : : :; An that could be potentially executed after the change point. Given that application Ai is running against a design Dj before a change point, then if application Aj ; 1  j  n is executed after the change point, then one needs to decide on the design Djk ; 1  jk  m that needs to be used. Table 1: A Redesign Policy for a state (Ai ; Dj )

8 > Application A1 is executed > > > > Application A2 is executed > > > > If the current state of SMDP > < Application A3 is executed . is (Ai ; Dj ) and if after the > > > . change point > > > > > . > > : Application An is executed

Then use Design Di1 Then use Design Di2 Then use Design Di3 . . . Then use Design Din

A redesign policy as illustrated in table 1 is de ned to specify the design that needs to be selected at a change point. A redesign policy is applicable for each state of the SMDP. The redesign policy shown in table 1 describes that if the application Ai is being currently executed on design Dj then after the change point if application A1 is to be executed then use design Di1 , if application A2 is to be executed then use design Di2 , and so on. This redesign policy is denoted by (A1 ! Di1 ); (A1 ! Di2 ); : : :; (A1 ! Din ) as This assumption can be easily relaxed by constraining each application to execute only on a subset of the candidate designs 2

9

Table 2: Redesign Policies for 3 Applications and 3 Designs Policy Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

Policy

(A1 ! D1); (A2 ! D1 ); (A3 ! D1) (A1 ! D1); (A2 ! D1 ); (A3 ! D2) (A1 ! D1); (A2 ! D1 ); (A3 ! D3) (A1 ! D1); (A2 ! D2 ); (A3 ! D1) (A1 ! D1); (A2 ! D2 ); (A3 ! D2) (A1 ! D1); (A2 ! D2 ); (A3 ! D3) (A1 ! D1); (A2 ! D3 ); (A3 ! D1) (A1 ! D1); (A2 ! D3 ); (A3 ! D2) (A1 ! D1); (A2 ! D3 ); (A3 ! D3) (A1 ! D2); (A2 ! D1 ); (A3 ! D1) (A1 ! D2); (A2 ! D1 ); (A3 ! D2) (A1 ! D2); (A2 ! D1 ); (A3 ! D3) (A1 ! D2); (A2 ! D2 ); (A3 ! D1) (A1 ! D2); (A2 ! D2 ); (A3 ! D2) (A1 ! D2); (A2 ! D2 ); (A3 ! D3) (A1 ! D2); (A2 ! D3 ); (A3 ! D1) (A1 ! D2); (A2 ! D3 ); (A3 ! D2) (A1 ! D2); (A2 ! D3 ); (A3 ! D3) (A1 ! D3); (A2 ! D1 ); (A3 ! D1) (A1 ! D3); (A2 ! D1 ); (A3 ! D2) (A1 ! D3); (A2 ! D1 ); (A3 ! D3) (A1 ! D3); (A2 ! D2 ); (A3 ! D1) (A1 ! D3); (A2 ! D2 ); (A3 ! D2) (A1 ! D3); (A2 ! D2 ); (A3 ! D3) (A1 ! D3); (A2 ! D3 ); (A3 ! D1) (A1 ! D3); (A2 ! D3 ); (A3 ! D2) (A1 ! D3); (A2 ! D3 ); (A3 ! D3)

shown in table 2. As each of the Di1 can be any one of the m designs fD1; D2; : : :; Dmg the number of redesign policies possible for the state (Ai; Dj ) is nm . These nm redesign policies are numbered from from 1 to nm . For each state (Ai ; Dj ) of the SMDP a particular redesign policy is selected. These selected redesign policies for all the states form a redesign policy vector Policy = [p1 ; p2; : : :; pnm ]T 3 where 8j; 1  pj  nm . The elements of the policy vector have one-one correspondence with the states of SMDP enumerated as f(A1; D1); (A1; D2); : : :; (A1; Dm); (A2; D1); (A2; D2); : : :; (A1; Dm); : : :: : : (An ; D1); (An ; D2); : : :; (An ; Dm)g. That is, the rst policy element of policy vector is for the state (A1; D1), second policy element is for the state (A1 ; D2), and in general the jth policy element is for the state (Adj=me ; Dj ?bj=mcm ) in the SMDP.

Example 2 Consider the three applications A ; A and A with the stochastic matrix as 3

VT

1

denotes the transpose of a matrix V .

10

2

3

0.8 0.8 0.8

(A 1, D3 )

(A 1, D2)

(A 1, D1 )

0.2

0.8 0.2

0.2

0.2

0.2

0.8 (A2 , D1 )

0.8

(A 2 , D3 )

(A2 , D2 ) 0.2

0.2

0.8

(A 3 , D1)

0.2

(A 3 , D2 )

0.8

(A 3 , D3 )

0.8

Figure 2: SMDP for Policy Vector [1; 4; 5; 6; 3; 19; 4; 8; 11]t given in Example 1. Let D1; D2, and D3 be the three candidate designs for these applications. Since there are 3 applications and 3 designs there will be 33 = 27 redesign policy decisions as enumerated in Table 2. Each row of the table corresponds to one redesign policy. A policy vector [1; 4; 5; 6; 3; 19; 4; 8; 11]T corresponds to the redesign policy decisions taken by the states f(A1; D1); (A1; D2); (A1; D3); (A2; D1); (A2; D2); (A2; D3); (A3; D1); (A3; D2); (A3; D3)g respectively. This policy vector de nes the SMDP as shown in Figure 2. This policy vector will have an expected cost value for processing all the applications A1; A2 and A3 over a period of time. The objective of this paper is to come up with a policy vector that minimizes the expected cost of processing for all the applications. 2

The optimal redesign policy vector is generated by applying the value-determination and policy improvement routine iteratively so as to minimize the application processing cost over time [How60] (see also section 5). Before we describe the algorithm we shall present an overview of application processing cost calculation.

4 Cost Values Calculation 4.1 Application Processing Cost The average transaction response time and materialization cost are used by the SMDP to generate the optimal policy for the distributed database redesign methodology in the application processing center. We present an overview of a simulation model to estimate average transaction response time in DDE. This model is an extension of the model proposed by [YDR+ 85]. The method we are using to estimate the average transaction response time 11

is one of the many that can be used. Other methods to estimate average transaction response time include dierent simulation models (like [CL88]), analytical models [YDL93], and actual measurements on the DDE. Our goal here is to present a method to estimate the average transaction response time. In the formulation of the application processing cost in SMDP, we can use any of the above mentioned methods to estimate average transaction response time. Based on the average transaction response time r, and the average number of transactions l initiated during the execution duration of an application, the average application processing cost CA per transaction executed in seconds is given by:

CA = ClM + r

(1)

where CM is the materialization cost. Note that if there is no change in design after the change point then CM = 0 and CA = r. But if there is a change in design then the materialization cost CM  0 and CA = ClM + r. The materialization cost when there is a change in design is amortized over the l transactions executed by the application during its execution duration. If a state transition from (Ai ; Dj ) ?! (Ak ; Dl) takes place in a SMDP, then the cost of processing application Ak is given by: j ! Dk ) + r(A ) CA(Ak ) = CM (D k l(A ) k

(2)

where CA (Ak ) is the application processing cost for application Ak , CM (Dj ! Dk ) is the cost of materializing design Dk from design Dj , l(Ak ) is the number of transactions initiated by the application Ak during its execution duration, and r(Ak ) is the average transaction response time for application Ak executing on a distributed database using design Dk . The objective is to minimize the total time taken to execute all the applications in an application processing center based on a DDE. This total time takes into consideration the materialization of redesign cost whenever there is a change in redesign and the time taken to process all the transactions initiated by the applications. In the next two subsections we present the overview of approaches taken to calculate the materialization cost and average transaction response time.

4.2 Materialization Cost The distributed database design (F; A) consists of a fragmentation (F ) and an allocation (A) scheme. The total redesign generates a new fragmentation and allocation scheme from the current design represented by (F 0 ; A0). The following approach has been developed [Kar92] to materialize (F 0 ; A0) from (F; A) for a populated distributed database. Query Generator In this approach each of the fragments in the new fragmentation scheme is expressed as a query on the distributed database so as to let the distributed database system materialize the new fragmentation scheme. This approach depends on the ability to generate correct queries for de ning each of the fragments. The distributed database system undertakes the task of materialization of the new design. The module which generates the above set of statements is known as the query generator. In [Kar92] we discuss 12

in detail the automatic generation of queries based on the speci cation of the redesign. The query generator algorithm to materialize redesigned distributed databases has been implemented at the [Chi94]. Since the cost model is based on the methodology to materialize the distributed database design and the focus of this paper is on redesign methodology for an application processing center, the details and cost model are not presented here. The materialization cost is the sum of the following three cost values.

 All the fragments are split into sub-fragments so as to facilitate formation of the

fragments of the new design. The cost of splitting a fragment is given by the number of disk I/O's required to split it into all its sub-fragments. This number is based on the size of the fragment.  There is a communication delay involved in transferring the sub-fragments to the sites where corresponding new fragments need to be formed. This cost is based on the average baud rate of the network in the DDE.  Once all sub-fragments are located according to the new allocation scheme, they need to be merged to form the new fragments. This cost is given in terms of a number of disk I/O's required to merge the sub-fragments, and is based on the size of the new fragments.

4.3 Simulation Model for Average Transaction Response Time Estimation The simulation model to estimate the average transaction response time is based on the frequency of transactions initiated, the number of transactions initiated, the fragmentation and allocation schemes, and the system parameters (like data access time, communication delay and the MIPS of the CPU). The average number of transactions initiated by an application and their frequencies is calculated by monitoring the applications processed and collecting the statistics. Since the applications to be processed in the application processing center are well de ned, it is easy to develop utilities to collect these statistics. The database management system provides tools that calculate the number of transactions initiated from a given site and the times when these transactions were initiated. The DDE that we are considering in this paper consists of a set of nodes of a network each of the which has a relational database system that can coordinate as a distributed database system. Following are the characteristics of the DDE: (i) All the computers are connected by means of a network with a certain data transfer rate. (ii) Each computer system has a database management system that manages a number of locks. (iii) Each computer has a distributed database system that enables access to data from any other site (computer) in the DDE. A transaction is an atomic unit of consistent, reliable and durable computation. In DDE there are two kinds of transactions: local transactions and distributed transactions. A local transaction accesses data only from the site where it was initiated. A distributed transaction accesses data from more than one site. An application executed in the DDE consists of a set of transactions. An execution of a transaction consists of loading the transaction initiating application program, accessing and processing the data from the database, and nally committing or aborting. Figure 3 shows the phases of transactions described below. 13

Local Transaction

Distributed Transaction

Begin Transaction

Begin Transaction

Initiation Phase Initiation Phase Sub-transaction Spawning

Database Access & Processing Phase

Initiation Phase

Database Access & Processing Phase

Commit or Abort Phase End Transaction

Commit or Abort Phase End Transaction

Figure 3: Local and Distributed Transaction Phases A local transaction has the following phases during its lifetime: Initiation The program initiating the local transaction has to be loaded into the main memory and then executed. This incurs some disk I/O's and some CPU processing. Database access The transaction then goes through a set of actions. An action being \a request for a lock", \release a lock", and \data access". The data access is from the database and hence the time for it will be dierent from the time required to do a disk I/O. There is some \CPU processing" between any two successive actions. Commit or Abort The transaction then either commits by writing a log record or aborts. A commit incurs writing a log record to stable storage and then releasing the locks held. A distributed transaction has the following phases during its lifetime: Initiation The program initiating the distributed transaction has to be loaded into the main memory and then executed. This incurs some disk I/O's and some CPU processing. Spawning The distributed transaction spawns a number of sub-transactions as local transactions at dierent sites. This incurs some communication delay to initiate sub-transactions at dierent sites. These local transactions access and process the data from the database as given above. Once the local sub transactions of the distributed transaction nish their processing, there is some data that is transferred to the site where the distributed transaction is initiated. Commit or Abort Once the local sub-transactions nish processing the data accessed from the local database, the distributed transaction initiates a two phase commit protocol. The distributed and its local sub-transactions commit or abort based on the outcome of the commit protocol. 14

On any given computer of the DDE, both local and distributed transactions are executed concurrently. Hence they may contend for the locks on the data item they access. In case a transaction A contends for a lock with transaction B , then transaction A has to wait until either transaction B either commits or aborts. The wait time for a transaction A during contention is directly proportional to the lock hold time of the transaction B . The model for locking mechanism developed by [YDR+ 85] is extended to model the contention in a distributed relational database transaction processing environment. The eect of contention on transaction response time is cyclic in nature. That is, the contention increases the transaction response time, which increases average lock hold time, which in turn increases the probability of contention. This cyclic aect of contention needs to be incorporated into the simulation model. If the DDE being simulated is stable, then the system stabilizes to a value of probability of contention. The simulation models the cyclic nature of the eect of probability of contention on transaction response time and vice versa by using a feedback process. The probability of contention [CDY90] is proportional to the lock request rate and average lock hold time (which is a part of average transaction response time). The initial estimate of probability of contention is based on average transaction response time when there is no contention. During the simulation, the probability of contention is calculated based on the average transaction response time of the previous few thousand transactions. This new calculated probability of contention is used for the next few thousand transactions. This is done iteratively till there is no appreciable change in the values for the probability of contention for two consecutive runs. A comprehensive set of simulations in [Kar92] were done to evaluate the simulation model for a range of data access times, changes in arrival rate, changes in number of data accesses and range of communication delays. Note that this simulation model is used to estimate the average transaction response time for an application on a design that has not been materialized before. Once a design has been materialized the average transaction response time can be empirically calculated.

5 Optimal Policy Vector Generation The optimal policy vector generation is based on the SMDP technique developed by Howard [How60] in his PhD thesis. The algorithms for generating these optimal policy vectors are well known. Below a brief description of the technique along with an example is presented.

5.1 Modeling of Redesign Problem Given n applications A1 ; A2; : : :; An that need to be processed in the application processing center. These applications can be executed on any one of the m candidate distributed database designs. Let P = [pij ] be the stochastic matrix of order n  n giving the probability of executing an application Aj after completing the execution of an application Ai . C = [cij ] be the cost matrix of order m  m giving the time taken to materialize design dj from design di . R = [rij ] is the matrix of order n  m giving the time taken to execute the application Ai on design dj . 15

Example 3 Let us consider the three applications A ; A and A and three designs D ; D 1

2

3

1

2

and D3 as described in Example 1. Let the matrix C give the cost of materializing one distributed database design from another. The cost values for the example have been selected to emphasize the utility of applying the SMDP to generate optimal redesign policy vectors for an application processing center environment. In the next section, we shall present a case study where realistic cost values are calculated and used. If there is no change in distributed database design to be used after a change point, then it has zero cost. The cost materializing distributed database design D2 from D1 , D3 from D2, and D1 from D3 are all 15 seconds4. And the cost of materializing distributed database design D3 from D1 , D1 from D2, and D2 from D3 are all 10 seconds. 3

2 0:0 C = 64 ?10:0

?15:0 ?10:0 7 0:0 ?15:0 5 ?15:0 ?10:0 0:0

The matrix R gives the cost of executing an application on a distributed database design. Without loss of generality and for illustrating the SMDP technique we assume that each application initiates only one transaction for this example. Therefore in equation 1 we have l = 1 and CA = CM + r. In this example, CM values are given by the elements of matrix C , and r values are given by the elements of matrix R. 3

2 ?10:0 6 R = 4 ?15:0

?15:0 ?15:0 7 ?10:0 ?15:0 5 ?15:0 ?15:0 ?10:0

The cost of executing an application A1 on distributed database design D1 is the cheapest at 10 seconds, and on other designs it takes 15 seconds. Similarly, the costs of executing A2 on D2, and A3 on D3 is 10 seconds, and on all other distributed database designs it is 15 seconds. 2

For these three applications in the application processing center following observations can be made:

 The probability of an application to continue executing after a change point is four times the probability that there will be a change in the application to be executed from matrix P give in example 1. There is a schedule for executing the applications, which is A1 ?! A3 ?! A2 ?! A1 .  Each distributed database design executes a single application optimally. An application A1 is eciently executed by distributed database design D1 , A2 by D2, and A3 by D3.

Since the SMDP maximizes bene t values, and the goal of redesign policy is to minimize cost, we represent cost values as negative bene t value. 4

16

 The materialization of redesign cost is the cheapest along the schedule. That is, cost of materializing D from D is cheaper than D from D , cost of materializing D from D is cheaper than D from D , cost of materializing D from D is cheaper than D from D . 3

1

2

2

3

1

1

1

2

2

3

3

0.8 0.8 0.8 (A 1, D1 )

(A 1, D3 )

(A 1, D2) 0.2

0.2

(A2 , D1 )

0.2

0.8

0.8

0.2

(A2 , D2 )

0.8

(A 2 , D3 )

0.2

0.2 0.8 0.2

(A 3 , D1)

0.2

(A 3 , D2 )

0.8

(A 3 , D3 )

0.8

Figure 4: SMDP De ned by the Optimal Policy Vector The SMDP consists of two phases: value-determination process and the policy improvement process [How60]. The process starts by considering an initial policy vector, determining the values, and evaluating if the policy could be improved further. This iteration between value-determination and policy improvement ends when in successive iterations there is no change in the policy vector. Howard's algorithms use the input provided by the matrices P , C , and R, to generate the optimal policy vector after ve iterations. In [Kar92] we implemented these algorithms and generated the optimal policy vector. The optimal policy vector is: Policy = [3; 2; 3; 4; 4; 4; 3; 5; 6]T (i.e. for state (A1 ; D1) the policy number is 3, for state (A1 ; D2) the policy number is 2, for state (A1 ; D3) the policy number is 3, for state (A2; D1) the policy number is 4, and so on). The SMDP de ned by this policy vector is illustrated in Figure 4. It de nes the policies for the 9 states of applications; the policy numbers are described in Table 2. In the long run the SMDP de ned by the optimal policy vector converges to the process de ned by (A1 ; D1) ! (A2; D2) ! (A3 ; D3) ! (A1; D1). This means that the application processing center always needs to use design D1 when processing applications A1 , design D2 when processing applications A2, and design D3 when processing applications A3 . Therefore, whenever there is a change in applications that need to be executed there is a change in design. For this example with these cost values, it is bene cial to change the distributed database design whenever there is a change 17

of applications to be processed.

6 Marketing and Recruiting System: A Case Study A case study was undertaken to illustrate the distributed database design/redesign methodology proposed in this paper. The Oce of Information Technology at Georgia Tech has been developing a student and nancial information system based on a relational database system. There is a plan to \connect" the data from a variety of sources into a DDE. The proposed methodology to generate optimal redesign policies based on SMDP would help in deciding how to fragment and allocate existing relations. The marketing and recruiting system is a module of this information system and consists of a set of applications. This system is used by Georgia Tech personnel to recruit quality students to their undergraduate and graduate programs. The applications de ned in this case study are conceptualized from this marketing and recruiting system. Following are the characteristics of the case study (some of the details on transactions and designs are elaborated in the Appendix 1):

 There were ve sites and four relations forming the distributed database system. All

sites were fully connected.  15 applications were de ned. These applications initiated 70 transactions.  Three applications were de ned, two on-line applications (A1 and A2), and one batch application(A3). Three candidate designs (D1, D2, and D3) were generated, each one being optimal for one application.  Two application processing scenarios were tested. One scenario consisted of a well planned schedule and other one had some randomness in selecting the application to be executed. The approach taken for the case study was as follows:

 For each application, the mixed fragmentation methodology [NKR94] was used to generate the candidate design.

 The average transaction response time for each application on each candidate design was calculated as follows:

1. All the transactions executed by each application are mapped to the sites from which they are initiated. 2. For each transaction initiated from each site, the following information is tabulated: { Whether it is a distributed or local transaction: if distributed, the number of sites involved in processing the transaction. { Number of fragments accessed by the transaction. 18

{ For transactions accessing more than two fragments, whether a distributed

3.

4.

5. 6.

join, or a distributed union, or a local join, or a local union is performed. Average number of data accesses for a local and distributed transaction are estimated based on above information. In an implemented system, this information can be gathered from the database system utilities, and transaction trace analysis. At each site, the arrival rate of local and distributed transactions are calculated by using the frequency with which each of the transactions are initiated from a given site. From this the average arrival rate of local and distributed transactions at each site is calculated. Based on the estimates for data accesses for each transaction, the average number of data accesses for local and distributed transactions are calculated. Finally, nine simulations were run to estimate the average transaction response time for each combination of an application and a candidate distributed database design.

 The materialization cost was calculated by using the formulae given in [Kar92].  Finally, Howard's algorithm was executed to generate the optimal redesign policy. We shall now present the results of the redesign methodology for both the scenarios.

6.1 Well Planned Schedule For the three applications that follow a strict schedule for their execution A1 ! A2 ! A3 ! A1, the probability of change in application to be executed at the next change point is given by the stochastic matrix P . The execution duration for each of the on-line applications (A1 and A2) is ve hours during daytime and for the batch application (A3) and seven hours during the night. The remaining seven hours are allocated for the materialization of the redesigned distributed databases if required. 3 2 0:0 1:0 0:0 P = 64 0:0 0:0 1:0 75

1:0 0:0 0:0

Then the matrix R giving the bene t (negative of cost) values in seconds for executing each application on each of the three candidate designs is: 3

2 ?0:703 R = 64 ?1:464

?0:924 ?1:232 7 ?1:234 ?1:283 5 ?1:685 ?1:409 ?0:949

The columns denote the designs D1, D2 and D3 respectively and the rows denote applications A1, A2 and A3 respectively. The (i; j )th of the matrix R above gives the average 19

transaction response time when application Ai is executed on design Dj . Note that it is optimal to run application A1 on design D1, application A2 on design D2 and application A3 on design D3. But in order to redesign before executing a dierent application, one needs to take into consideration the overhead due to materialization of the distributed database design. This overhead in seconds per transaction is calculated by dividing the amount of time taken to materialize the distributed database by the average number of transactions initiated before the next change point as given in equation 2. The number of transactions are calculated based on the arrival rate of transactions at each node of the DDE. Note that this arrival rate is also used by the simulation model to estimate the average transaction response time. For the case study we used the cost model developed in [Kar92] to calculate the amount of time to materialize the new design, and from that we calculate the overhead of materialization cost per transaction. The cost of materialization was: 3

2 0:0 6 C = 4 ?3897:0

?3325:0 ?3975:0 7 0:0 ?3746:0 5 ?2818:0 ?2561:0 0:0

That is, it takes 3325 seconds to materialize design D2 from design D1, 3975 seconds to materialize design D3 from D1, 3897 seconds for D1 from D2, 3746 seconds for D3 from D2, 2818 seconds for D1 from D3, and 2561 seconds for D2 from D3. Note that seven hours had been allocated for materializing the redesigned distributed relational databases which is sucient in this case. The overhead due to materialization cost per transaction is given by matrix Overhead. The application name in the parenthesis speci es the application for whose execution after the change point this overhead value5 is applicable. 3

2 0 Overhead = 64 ?0:06(A1)

?0:07(A2) ?0:024(A3) 7 0 ?0:023(A3) 5 ?0:04(A1) ?0:053(A2) 0

The rows and columns denote the designs D1, D2 and D3 respectively. The overhead per transaction due to materialization of design D2 from design D1 is 0.07 seconds for executing application A2, for design D3 from D1 is 0.024 seconds for executing application A3, for design D2 from D1 is 0.06 seconds for executing application A1, for design D3 from D2 is 0.023 seconds for executing application A3, for design D1 from D3 is 0.04 seconds for executing application A1, and for design D2 from D3 is 0.053 seconds for executing application A2 after the change point. Therefore, if dierent applications are to be executed than the overhead to materialize the design also changes. That is, overhead for state transition (A1; D1) ?! (A2; D2) is dierent than the overhead for the state transition (A1; D1) ?! (A3; D2) even though the same change in design (i.e. from D1 to D2) is taking place. Thus the overhead depends on which application is going to be executed after the change point on which design. This information can be gathered from the state The overhead for the materialization cost all the state transitions that arise are not presented here due to lack of space. 5

20

transitions (as speci ed equation 2). If the state transition is from (A1, D1) to (A2,D2) then D2 has to be materialized from D1 to execute application A2. In the case study equation 2 is used to calculate the application processing cost which calculates the overhead due to materialization cost by considering all the alternatives that arise due to various state transitions. Note that materialization cost given in C is independent of the application to be executed, but the overhead due to materialization depends on the application to be executed after the change point. Then by using the probability matrix P, the cost matrices R and C, the optimal policy vector Policy 6 generated by the SMDP algorithms is: Policy = [4; 4; 7; 3; 3; 3; 1; 1; 1]T . 1.0 A1 D1

1.0

A2

1.0

A3 D3

D1

Figure 5: Long Run SMDP For Well Planned Schedule. Figure 5 shows the long run SMDP (the states being (A1,D1), (A2,D1), and (A3,D3)) de ned by the above policy. This policy implies that in the long run it is always optimal to use distributed database design (D1) for executing the on-line application (A1) and the of on-line application (A2). But for executing the batch application (A3) it is preferred to use the candidate distributed database design (D3). The design (D1) is preferred for processing on-line applications (A1) and (A2) in the long run because it gives the least average transaction response time. For the same reason, design (D3) is preferred for the batch application (A3). Thus there is a change in design whenever there is a change from processing the on-line applications to processing the batch application and vice versa.

6.2 Random Execution of the Applications A scenario with randomness in the applications initiated is considered next. After executing the application (A1) during working days, the batch application (A3) is executed for during weekends. Similarly, after executing the application (A2) during working days, the batch application (A3) is executed during weekends. But after executing the batch application, one of the on-line applications f(A1), (A2)g is executed with equal probability. This scenario incorporates the randomness in the applications to be initiated. The probability of change matrix will be given by the following stochastic matrix: 3 2 0:0 0:0 1:0 P = 64 0:0 0:0 1:0 75

0:5 0:5 0:0

The cost of executing the applications on the candidate designs and materialization cost does not change, but the overhead due to the cost of redesign changes this is because the execution duration of application changes (i.e. the number of transactions initiated also 6

Please refer to Table 2 for the meaning of each of the policy numbers in the policy vector.

21

changes). For this case, the optimal policy vector generated by the SMDP was: Policy = [3; 3; 3; 3; 3; 3; 4; 4; 7]T . A1 1.0 D1 0.5

A3 D3

1.0 A2 0.5 D3

Figure 6: Long Run SMDP For Random Execution of the Applications. The analysis of the above policy vector shows that in the long run (see Figure 6), distributed database design (D3) will be the least costly to use for the on-line applications (A2) and batch applications (A3). And the candidate distributed database design (D1) will be the least costly to run the on-line applications (A1). The optimal policy shows that it is bene cial to do redesign for this case also. Thus the case study showed the applicability of the methodology presented in this paper to a realistic application processing scenario.

7 Conclusions In this paper, we addressed the problem of preventive redesign where there is a random behavior in the applications being executed in the DDE over a time period. This random behavior is modeled as a SMDP. A set of candidate distributed database designs are generated for a set of applications, and the problem is to select the optimal distributed database design for each execution of an application in the long run. There are two cost values that govern this selection. The rst one is the cost of using a particular candidate design by an application. The second one is the cost of materializing the populated redesigned distributed database. A preventive distributed database redesign methodology based on SMDP has been developed. The SMDP uses the cost of materialization, the average transaction response time, and the stochastic matrix de ning the random behavior in executing the applications to generate an optimal policy. This optimal policy speci es the candidate distributed database design that needs to be used by any application to be executed. This optimal policy is guaranteed to process all the applications eciently in the long run. Finally, a real life case study has been developed by conceptualizing a set of distributed databases and applications in use at Georgia Tech. Currently, there is no single distributed DBMS that interconnects all databases. But we assume for this case study that such a system is in existence. Two application processing scenarios were modeled as SMDP, and the SMDP was used to generate the optimal policies. These optimal policies speci ed that a switching between two distributed database designs is required to eciently execute the given applications in the long run for both the application processing scenarios. In summary, the methodology of preventive redesign for application processing center consists of following steps: 22

 The algorithms developed for mixed fragmentation are used to generate a set of can-

didate designs for an application.  A simulation model is used to estimate the average transaction response time for each application executed on each of the candidate distributed database designs.

 The algorithms for materialization are used to calculate the cost of redesign from one candidate distributed database design to another.  SMDP is used to generate the optimal policy vector that speci es the design which is to be used by an application that is to be executed after a change point.

The scope of this work is limited to an application processing center environment with well de ned applications and sucient time must be allowed for materialization of the distributed relational databases. We address the problem of adaptive redesign in [Kar94] which is not limited by the application processing center environment, initiates a redesign which is incremental and hence takes lot less time to materialize than complete change of design as it is done for preventive redesign methodology. The applicability of the preventive redesign methodology has been shown by means of a case study. Many application processing centers based on distributed relational database systems will nd this methodology viable.

References [Ape88] [CDY90] [Chi94]

[Chu94]

[CL88] [CN83]

P. M. G. Apers. Data allocation in distributed database systems. ACM Transactions on Database Systems, 13(3):263{304, September 1988. B. Ciciani, D. M. Dias, and P.S. Yu. Analysis of replication in distributed database systems. IEEE Transactions on Knowledge and Data Engineering, 2(2):247{261, June 1990. Lee Wai Ching. Implementation of a query generator algorithm for a distributed relational database design tool. BEng, Final Year Project Report, June 1994. Department of Computer Science, Hong Kong University of Science and Technology. Ngan Mei Chun. Implementation of fragmentation algorithms for a distributed relational database design tool. BEng, Final Year Project Report, June 1994. Department of Computer Science, Hong Kong University of Science and Technology. M. J. Carey and M. Livny. Distributed concurrency control performance: A study of algorithms, distribution, and replication. In Proceedings of International Conference on Very Large Databases, 1988. S. Ceri and S. B. Navathe. A comprehensive approach to fragmentation and allocation of data in distributed databases. In Proceedings of the IEEE COMPCON Conference, pages 426{431, 1983. 23

[CNP82]

S. Ceri, M. Negri, and G. Pelagatti. Horizontal data partitioning in database design. In Proceedings of ACM SIGMOD International Conference on Management of Data, pages 128{136, 1982.

[CNW83] S. Ceri, S. B. Navathe, and G. Wiederhold. Distribution design of logical database schemas. IEEE Transactions on Software Engineering, SE-9(5):487{ 504, July 1983. [CP84] S. Ceri and G. Pelagatti. Distributed Databases: Principles and Systems. McGraw Hill, New York, 1984. [How60]

R. A. Howard. Dynamic Programming and Markov Processes. M.I.T. Press, 1960.

[Kar92]

Kamalakar Karlapalem. Redesign of Distributed Relational Databases. PhD thesis, College of Computing, Georgia Institute of Technology, December,1992.

[Kar94]

Kamalakar Karlapalem. An active database system supported adaptive redesign of distributed relational databases. under preparation for publication, 1994. K. Karlapalem and S. Navathe. Materialization of distributed relational databases. submitted to a journal, 1994. Wu Yu Man. Implementation of a system catalog for a distributed relational database design tool. BEng, Final Year Project Report, June 1994. Department of Computer Science, Hong Kong University of Science and Technology.

[KN94] [Man94]

[MCVN93] J. Muthuraj, S. Chakravarthy, R. Varadarajan, and S. B. Navathe. A formal approach to the vertical partitioning problem in distributed database design. In Proc. of Second International Conference on Parallel and Distributed Information Systems, San Diego, California, 1993. [NCWD84] S. B. Navathe, S. Ceri, G. Wiederhold, and J. Dou. Vertical partitioning algorithms for database design. ACM Transactions on Database Systems, 9(4):680{ 710, 1984. [NKR94] S. B. Navathe, K. Karlapalem, and M. Ra. A mixed fragmentation approach for initial distributed database design. Journal of Computers and Software Engineering, To appear in 1994. [NR89]

S. B. Navathe and M. Ra. Vertical partitioning for database design: A graphical algorithm. In Proceedings of ACM SIGMOD International Conference on Management of Data, 1989.

[OV91a]

M. T. Ozsu and P. Valduriez. Distributed database systems: Where are they now? IEEE Computer, 24(8), August 1991. T. Ozsu and P. Valduriez. Principles of Distributed Database Systems. PrinticeHall Inc., 1991.

[OV91b]

24

[RVVN90] P. I. Rivera-Vega, R. Varadarajan, and S. B. Navathe. Scheduling data redistribution in distributed databases. In Proceedings of International Conference on Data Engineering, IEEE, February 1990. [WN86] [YDL93]

B. Wilson and S. B. Navathe. An analytical framework for the redesign of distributed databases. In Proceedings of the 6th Adavanced Database Symposium, Tokyo, Japan., 1986. Philip S. Yu, Daniel M. Dias, and Stephen S. Lavenberg. On the analytical modeling of database concurrency control. Journal of the Association for Computing Machinery, 40(4):831{872, September 1993.

[YDR+ 85] P. S. Yu, D. M. Dias, J. T. Robinson, B. R. Iyer, and D. W. Cornell. Modelling of centralized concurrency control in multi-system environment. In Proceedings of ACM SIGMETRICS, pages 183{191, 1985.

25

1 Applications, Relations, Transactions, and Designs There are ve nodes in the distributed database environment. Each node has a computer, and a distributed relational database management system. The nodes are College of Computing (CoC), College of Engineering (CoE), College of Management (CoM), College of Science (CoS), and the Central Administrative Building (CAB). Each of these colleges have Recruiters and maintain the information about the Prospects. All the ve nodes are fully connected by a campus wide LAN. All the ve nodes have the same computer, with same MIPS rating, Disk I/O time, and the DBMS. This environment with a set of applications accessing the relations de nes the application processing center scenario.

1.1 Applications and Relations Table 3: Relation Recruiters Column No c1 c2 c3 c4 c5 c6 c7 c8

Column Name Recr{id Recr{type Recr{name Recr{add Recr{major Recr{college Recr{comments Recr{date

Size Char(9) Char(4) Char(40) Char(250) Char(10) Char(30) Char(1024) Char(10)

Comment Unique Identi er Type Name Address Major for which the Prospects are recruited College the recruiter represents Any comments about the Recruiter Date on which Recruiter started recruiting

The application A1 consists of transactions related to following activities: 1. Access Recruiter Information. 2. Access Prospect Information. 3. Access Event Calendar. 4. List Prospects to whom Information Packets need to be sent. 5. List Prospects to whom Birthday Greeting Card has to be sent. 6. Schedule Events for Prospects and Recruiters. The application A2 consists of transactions related to following activities: 1. Match Recruiters to the Prospects. 2. Maintain Prospect Information: that is, to update the prospect information, delete or insert prospects. 26

Table 4: Relation Prospects Column No c9 c10 c11 c12 c13 c14 c15 c16 c17 c18 c19 c20 c21 c22 c23 c24 c25 c26 c27 c28 c29 c30 c31

Column Name Pros{id Pros{type Pros{name Pros{sex Pros{ms Pros{state Pros{add Pros{place Pros{dob Pros{resi Pros{app{id Pros{term{yr Pros{college Pros{gpa Pros{sat{verb Pros{sat{quan Pros{sat{anal Pros{gre{verb Pros{gre{quan Pros{gre{anal Pros{gmat Pros{toe Pros{international

Size Char(9) Char(4) Char(40) Char(1) Char(1) Char(2) Char(250) Char(60) Char(10) Integer Char(7) char(10) Char(30) Float Integer Integer Integer Integer Integer Integer Integer Integer Char(1)

Comment Unique Identi er Type `ug' or `grad' Name Sex Marital Status State of Residency Address Place of residence Date of Birth Residency achieved ag Application Identi er Term and Year applied for College the prospect is interested in Cumulative GPA SAT Verbal score SAT Quantitative score SAT Analytical score GRE Verbal score GRE Quantitative score GRE Analytical score GMAT score TOEFL score Whether out of country

3. Maintain Recruiter Information: that is, to update the recruiter information, delete or insert recruiters. The application A3 consists of transactions related to following activities: 1. Insert or Update or Verify GRE/TOEFL/GMAT/SAT scores. 2. Evaluate all the Prospects. 3. Evaluate all the Recruiters.

1.2 Transaction and Relation Characteristics Table 7 gives the cardinalities of the relations in the distributed database environment. Table 8 gives the predicates used to access the relations and the approximate cardinality of the tuples satisfying the predicates. Table 9 gives the frequency of the transactions. The batch applications generate more transactions/sec than on-line applications. Also applications which are active throughout 27

Table 5: Relation Events Column No c32 c33 c34 c35 c36 c37 c38 c39 c40

Column Name Event{id Event{name Event{type Event{place Event{state Event{st{date Event{end{date Event{max{recr Event{max{pros

Size Char(9) Char(40) Integer Char(60) Char(2) Char(10) Char(10) Integer Integer

Comment Unique Identi er Name Type Place held State of Place Start Date End Date Maximum number of recruiters Maximum number of prospects

Table 6: Relation ProsRecr Column No c41 c42 c43 c44 c45

Column Name Pros{id Recr{id Adv{st{date Adv{fn{date Num{recr

Size Char(9) Char(9) Char(10) Char(10) Integer

Comment Start date of advising Last date of advising Number of recruiter

the day initiate fewer transactions/sec than applications which are active for brief intervals of time. In order to use a standard unit of time interval to calculate frequency, for some transactions the values of frequency are less than one. As some of these applications are running concurrently, the total arrival rate of transactions will be the sum of frequencies of all transactions initiated by these applications. This frequency of transactions will be used to come up with the distributed database designs.

1.2.1 Representative Transactions Each transaction that is initiated by an application accesses some tuples and columns of the relations. Therefore, the part of the relation accessed by each of the transaction initiated by the applications can be described by SQL SELECT statements. Here we present a sample of representative transactions from the 70 transactions used in the case study.

T1. SELECT Recr{id, Recr{name, Recr{add, Recr{type FROM Recruiters WHERE Recr{college = `CoC';

T17.SELECT Pros{id, Pros{gpa, Pros{sat{verb, Pros{sat{quan, Pros{sat{anal FROM Prospects

28

Table 7: Cardinalities of the Relations Relation Recruiters Prospects Events ProsRecr

Cardinality 200 40000 100 40000

WHERE Pros{college = `CoC' and Pros{type = `ug';

T26.SELECT Pros{id, Pros{gpa, Pros{gre{verb, Pros{gre{quan, Pros{gre{anal FROM Prospects WHERE Pros{college = `CoE' and Pros{type = `grad';

T33.SELECT Pros{id, Pros{name, Pros{add, Pros{sex, Pros{ms, Pros{dob FROM Prospects WHERE Pros{international = 1 and Pros{toe > 600;

T51. SELECT Recr{id, Recr{name, Pros{id, Pros{name, Event{id, Event{name, Event{place, Pros{add, Recr{add FROM Recruiters, Prospects, Events WHERE Event{place = Pros{place and Pros{college = Recr{college and Pros{type = Recr{type and Event{type = Pros{type;

T67. SELECT Recr{id, Recr{name, Recr{type, Recr{college, Pros{id, Pros{sex, Pros{gpa, Pros{gre{verb, Pros{gre{quan, Pros{gre{anal FROM ProsRecr, Prospects, Recruiters WHERE Prospects.Pros{id = ProsRecr.Pros{id and ProsRecr.Recr{id = Recruiters.Recr{id and Pros{type = `grad';

1.3 Distributed Database Designs The vertical and horizontal fragmentation algorithms are executed using the transaction and relation characteristics given in the previous section. The distributed database designs for the relation Prospects for executing application A1 are presented here. In a similar manner the distributed database designs for relation Recruiters are generated, the details are given in [Kar92]. The application processing costs in Section 6 were calculated using these distributed database designs for relations Prospects and Recruiters. The vertical fragments are: 29

Table 8: Selectivity of Predicates in the Relations Relation Name Predicate Recruiters Recr{type = `ug' Recr{type = `grad' Recr{college = `CoC' Recr{college = `CoE' Recr{college = `CoS' Recr{college = `CoM' Prospects Pros{type = `ug' Pros{type = `grad' Pros{college = `CoC' Pros{college = `CoE' Pros{college = `CoS' Pros{college = `CoM' Pros{state = `GA' Pros{gpa > 3.00 Pros{gpa > 3.25 Pros{international = 1 Pros{toe > 600 Events Event{type = `ug' Event{type = `grad'

1. 2. 3. 4. 5. 6.

Pros{id, Pros{name, Pros{sex, Pros{ms, Pros{add, Pros{dob. Pros{id, Pros{resi, Pros{app{id, Pros{term{yr. Pros{id, Pros{gpa, Pros{sat{verb, Pros{sat{quan, Pros{sat{anal. Pros{id, Pros{gpa, Pros{gre{verb, Pros{gre{quan, Pros{gre{anal. Pros{id, Pros{gpa, Pros{gmat. Pros{id, Pros{type, Pros{college, Pros{state, Pros{place, Pros{toe , Pros{international.

The horizontal fragments are:

a. b. c. d. e. f.

Cardinality 150 50 30 80 60 30 30000 10000 6000 16000 12000 6000 20000 15000 8000 3000 1500 70 30

Pros{type = `ug' and Pros-college = `CoC'. Pros{type = `grad' and Pros-college = `CoC'. Pros{type = `ug' and Pros-college = `CoS'. Pros{type = `grad' and Pros-college = `CoS'. Pros{type = `ug' and Pros-college = `CoE'. Pros{type = `grad' and Pros-college = `CoE'. 30

Table 9: Transaction Initiation Frequencies Frequency/sec 0.005 0.05 0.5 1.0 2.0

Transactions T1{T33 T34{T35, T41{T48 T36{T38, T56{T68 T39, T40, T49{T51 T52{T55

Table 10: Design for relation Prospects for Application A1 Fragment Id

p1 p2 p3 p4 p5 p6 p7 p8 p9

p10

Fragment Representation (1(a;b)) (1(c;d)) (1(e;f )) (1(g;h)) (2(a;b)) (2(c;d)) (2(e;f )) (2(g;h)) (3(a;b;c;d;e;f;g;h); 4(a;b;c;d;e;f;g;h) 5(a;b;c;d;e;f;g;h)) (6(a;b;c;d;e;f;g;h))

Site CAB CAB CAB CAB CoC CoS CoE CoM CAB CAB

g. Pros{type = `ug' and Pros-college = `CoM'. h. Pros{type = `grad' and Pros-college = `CoM'. The set of mixed fragments formed by merging the grid cells generated by above set of vertical and horizontal fragments were allocated to the sites of the DDE is shown in Table 10. The fragment represented by (1(a;b)) is formed by merging grid cells de ne by vertical fragment 1 and horizontal fragments a, b. Similarly the fragment represented by (3(a;b;c;d;e;f;g;h); 4(a;b;c;d;e;f;g;h) is formed by merging grid cells de ned by vertical fragments 3, 4 and horizontal fragments a,b,c,d,e,f,g,h. The details about fragment representation is given in [NKR94]. We do not present the details for generating allocation scheme the details are given in [Kar92]. The set of horizontal fragments for application A2 are the same as those for the application A1 the set of vertical fragments are given below. The optimal design is shown in Table 11. 10. Pros{id, Pros{name, Pros{sex, Pros{ms, Pros{add, Pros{state, Pros{resi, Pros{app{ id, Pros{term{yr. 20. Pros{id, Pros{gpa, Pros{sat{verb, Pros{sat{quan, Pros{sat{anal. 31

Table 11: Design for relation Prospects for Application A2 Fragment Id

p01 p02 p03

Fragment Representation 10(a;b;c;d;e;f;g) (20(a;b;c;d;e;f;g;h); (30(a;b;c;d;e;f;g;h) 40(a;b;c;d;e;f;g;h)) (50(a;b;c;d;e;f;g;h))

Site CAB CAB CAB

Table 12: Design for relation Prospects for Application A3 Fragment Id

p001 p002 p003 p004 p005

Fragment Representation (100(a;c;e;g)) (100(b;d;f;h)) (500(a;b;c;d;e;f;g;h)) 00 (2(a;b;c;d;e;f;g;h); 300(a;b;c;d;e;f;g;h) 400(a;b;c;d;e;f;g;h)) (600(a;b;c;d;e;f;g;h))

Site CAB CAB CAB CAB CAB

30. Pros{id, Pros{gpa, Pros{gre{verb, Pros{gre{quan, Pros{gre{anal. 40. Pros{id, Pros{gpa, Pros{gmat. 50. Pros{id, Pros{type, Pros{college, Pros{place, Pros{dob, Pros{toe , Pros{international. The set of horizontal fragments for application A3 are the same as those for the application A1 the set of vertical fragments are given below. The optimal design is shown in Table 12. 100. Pros{id, Pros{name, Pros{add.

200. Pros{id, Pros{sex, Pros{gpa, Pros{sat{verb, Pros{sat{quan, Pros{sat{anal. 300. Pros{id, Pros{gre{verb, Pros{gre{quan, Pros{gre{anal. 400. Pros{id, Pros{gmat. 500. Pros{id, Pros{toe , Pros{international. 600. Pros{id, Pros{type, Pros{ms, Pros{state, Pros{place, Pros{dob, Pros{app{id, Pros{ term{yr, Pros{college.

32