Design and Evaluation of a Hierarchical Application ...

Design and Evaluation of a Hierarchical Application Placement Algorithm in Large Scale Clouds Hendrik Moens, Jeroen Famaey, Steven Latr´e, Bart Dhoedt and Filip De Turck Ghent University - IBBT Department of Information Technology Gaston Crommenlaan 8/201, B-9050 Gent, Belgium Telephone: +3293314938, Fax: +3293314899 e-mail: [email protected]

Abstract—As the requirements and scale of cloud environments increase, scalable management of the cloud is needed. Centralized solutions lack scalability and fully distributed management systems only have a limited overview of the system. One of the often-studied problems in cloud environments is the application placement problem, used to decide where application instances are instantiated and how many resources to allocate to the instances. In this paper a general approach is introduced for using centralized cloud resource management algorithms in a hierarchical context, increasing the scalability of the management system while maintaining a high placement quality. The management system itself is executed on the cloud, further increasing scalability and robustness. The proposed method uses aggregation techniques to generate input values for a centralized application placement algorithm which is run in all management nodes. Decoupling ensures management nodes can function independently. Subsequently, we compare the performance of hierarchical application placement method with that of a fully centralized algorithm. The results show that a solution, within 5% of the optimum placement when using the centralized algorithm, can be achieved hierarchically in less than 25% of the time needed for execution of the centralized algorithm. Index Terms—Hierarchical systems, Distributed computing, Clouds

I. I NTRODUCTION In recent years the adoption of cloud computing has increased greatly. The increasing scale of clouds complicates management, which leads to to scalability issues of the management system itself, compromising the scalability of hosted applications. Centralized management systems are being replaced by distributed management infrastructures, that are often fully decentralized, lacking a full overview of the system, and making it more difficult to achieve a global optimum. Our previous theoretical work [1] indicates that a hierarchical structuring of control nodes enables good sharing of context, information of the current state of the system, with a relatively small communications cost, while at the same time having a high scalability. In this paper, we investigate how centralized algorithms can be modified to work in large-scale environments by using a hierarchical management system. The different levels of the hierarchy have a different view of the system, with highestlevel controllers having an overview of the system based on aggregated values, increasing the scalability of a hierarchy compared to a purely centralized solution. To this end, we

evaluate how centralized solutions for one specific problem, the cloud application placement problem, can be incorporated into a hierarchical management system. One of the key challenges in cloud management is quickly adjusting application resource allocation in the face of changing demands, and doing this in a scalable way. Determining which servers in the cloud need to execute which applications is done by means of solving the application placement problem. This problem is NP-hard, and many different solutions have been proposed[2], [3], [4], [5]. Currently the two common approaches to application placement are centralized and fully decentralized solutions. In centralized solutions, a controller gathers monitoring information, calculates an optimal placement, and enforces the configuration. These algorithms tend to be highly complex and slow to execute, which makes scalability an issue. Decentralized approaches on the other hand optimize using only local information, leading to suboptimal placement. Our results demonstrate that a hierarchical approach leads to scalable and fast cloud application placement, as the structure scales better than the centralized approach, and has a higherlevel overview of the total system compared to the fully decentralized model. A generic template for using centralized application placement algorithms in a hierarchical fashion is presented, based on information aggregation and decoupling of management levels. The aggregation and decoupling techniques demonstrated in this paper will enable the use of various centralized algorithms in a hierarchical fashion, greatly increasing scalability of existing cloud management solutions. In the next section, we will discuss related work. Afterwards, in Section III we will give an overview of the system architecture. Section IV contains a generalized formal description of the application placement problem. Following this, we discuss the hierarchical management itself in Section V, where both the creation of a management hierarchy and the modification of centralized algorithms to function in the system are described in-depth. In Section VI we evaluate the proposed solution. Finally, Section VII contains our conclusions. II. R ELATED WORK Much work has been done concerning the application placement problem. Most of the work can be divided into two categories: centralized and fully distributed.

One of the first papers on the subject of application placement was published in 2003 [5]. A solution is generated centrally using linear programming and genetic algorithms. The solution is more geared towards consolidation of datacenter resources rather than dynamic cloud provisioning in large scale clouds, as is the case here. Our solution is designed to scale where centralized solutions no longer function, be it because of CPU limitations, which would be the bottleneck when using linear programming, or bandwidth bottlenecks, which would occur when using genetic algorithms. Multiple centralized solutions have been proposed [2], [6], [7], [8], and many of the algorithms use similar principles. The centralized solution proposed in [2] uses multiple mincost max-flow problems to generate a suitable solution and has a complexity of O(n2.5 ), better than older solutions. The algorithm was further expanded in [8], yielding slightly better performance and results. As centralized solutions have access to all information concerning managed nodes, the placement quality is generally very good. These solutions work well for smaller datacenters, but do not scale well for large datacenters. Our solution uses centralized algorithms on clusters which are limited in size, leading to a much higher scalability at a cost to placement quality due to the smaller overview of the system resources. A distributed peer to peer system, used for cloud management was demonstrated in [9]. In this solution, every node contains a database of management information, which is selectively flooded towards other nodes. This ensures every node has the relevant information for its management, but due to its design, no single node has a full overview of the network. In our work, we focus specifically on a subproblem, the application placement problem, and we examine how application placement can be made to scale while maintaining a global overview. In [3] a fully decentralized approach is used, based on a gossip-protocol. Here individual nodes manage themselves and continually exchange information. Nodes continually improve their configuration by exchanging information and shifting load between them. It is shown that this leads to an optimal configuration if memory constraints are omitted. This decentralized approach has a very high scalability, but convergence to an optimum is slower when compared to the centralized approach, and as each step leading to a configuration causes the migration of applications, these steps are expensive. Another decentralized approach, proposed in [4] uses an economical approach in which every actor tries to maximize its own gain. In doing so, a good global solution is obtained, but like all fully decentralized solutions there is no higher-level overview of the network. In contrast to these fully decentralized solutions, ours executes application placement on different hierarchy levels in which higher-level nodes have better overview of the system and are able to achieve good placement quality, while still maintaining good scalability. While most application placement approaches are based on CPU and memory requirements, [10] executes application placement based on the physical location and bandwidth

requirements of the servers, trying to put as many application components as possible close to each other and taking into account the physical system configuration. The solution is based on a central placement manager of which multiple, synchronized instances can exist. It combines properties of both the centralized approach and decentralized approaches, but needs much synchronization between management node instances. In this paper we propose a management system in which the different management nodes are more loosely coupled, needing only limited amounts of communication between nodes. Hierarchical techniques in provisioning were used in [11], but this system is dependent upon hierarchical application and system descriptions and executes placement using only bandwidth information. By contrast, our solution uses hierarchies for the management itself without needing additional descriptions. Automatic hierarchical node ordering in peer to peer systems was demonstrated in [12]. TreeP is used to hierarchically structure nodes in peer to peer systems, mainly used to structure object lookup, and its structure is inspired by that of B-Trees. Each node can occur at multiple levels in the tree. Our solution uses a similar approach, but has different goals and thus uses a different structure. We use dedicated management nodes, without strict ordering. Because of this our solution can grow and recover faster, as no node ordering is needed in our management system. III. S YSTEM A RCHITECTURE The cloud computing datacenter consists of multiple servers. All the servers have two applications installed on them: a cloud environment, such as OpenNebula, and management middleware, the functionality of which will be explained later in this section. Most of the servers are execution servers, used to execute application instances. Some servers are used by the management system to control the execution servers. These servers, used to manage the cloud system are management servers. They execute a dedicated management application in their cloud environment instead of regular applications. This approach allows every server to be either a management server or an execution server, enabling dynamic scaling of the management system itself. The servers of the datacenter are connected hierarchically, where the execution servers act as leaves and the management servers are nodes of the management tree. The different system components are shown in Figure 1. Each server in the system contains a server manager. This is a lightweight middleware component, responsible for maintaining a relationship with the servers parent, gathering management information and sending it to the parent management server. Its main function is abstracting the cloud infrastructure present on the server. As this component is present on all servers, it can also be used to increase robustness of the system by monitoring its parent, and initiating leader election should one of the management nodes fail. A cache of neighbours,

Execution Server Server Manager Cloud Environment Application

...

Application

TABLE I S YMBOLS

Management Server manages

Server Manager Cloud Environment Hierarchy Controller Placement Function

Symbol A S Γ Ra Rd M

Fig. 1.

The system components on management and execution servers Γs

nodes with which the server has communicated with recently, can be used to reconnect to the management system. A second component, the hierarchy controller is only present on management servers and gathers service performance information of their children. These children are either application servers or other controllers. The controller executes application placement based on the locally present management information. It also aggregates it and forwards it to its own parent. The parent node executes application placement on a higher level, determining which clusters execute which applications, and how much of each application to execute. This hierarchy controller, unlike the server manager is a heavyweight component, which is executed like the other applications in the cloud (e.g.: in the case of OpenNebula as a Virtual Machine (VM)). It is responsible for executing the application placement algorithm. As every server in the cloud can be used as a controller, the management system itself can scale when needed and can be made more reliable. The server manager and hierarchy controller form the backbone of the architecture, but to enable full cloud functionality, additional supporting components would be needed: An application request router, responsible for routing successive application requests to application instances and session management, an application image repository containing images of the various hosted applications, and a policy repository containing placement restrictions. In this paper we focus on the application placement problem itself, so these components were not implemented. IV. F ORMAL P ROBLEM D ESCRIPTION The application placement problem itself has previously been described formally[2], [3], [4], [9]. In this section we formally describe and generalize common inputs and outputs of centralized application placement algorithms [2], [8], [10]. An overview of the symbols used here is shown in Table I. A cloud consists of a set of servers S on which a set of applications A are executed. The cloud management system considers a set of multiple resources types Γ, such as memory, CPU and bandwith. For each resource type r ∈ Γ, every server s ∈ S has available resources Rars , and every application a ∈ A has a resource demand Rdra . A centralized application placement algorithm takes a set of inputs and delivers as output a placement matrix M . For application a, server s and resource r contains the amount resource r to allocate on server r, Ms,a s for application a.

Γl Rdl Rds

Description The set of all applications. The set of all servers. The various resources considered by the system. The resource availability of the various servers. Rars determines the available amounts of resource r on server s. The resource demand of the applications. Rdra is a value containing the demand of application a for resource r. r contains the amount of resource The placement matrix, Ms,a r allocated to server s for application a. Resources for which the demand is strict. They have a fixed demand per-instance and without this amount the placement is invalid. Resources for which the demand is loose. The goal of the management system is to maximise loose demand fulfilment. Resource demands for loose resource types. Resource demands for strict resource types.

The inputs generally contain server resource information Ra, the current placement M 0 , and application resource demands Rd. When it comes to resource demands, we differentiate between two resource types which we shall call strict and loose. The set of strict resources Γs ⊂ Γ contains demands that are invariable and per-instance, such as memory use, and in some instances bandwidth. Loose demands Γl ⊂ Γ, such as CPU requirements and sometimes bandwidth, are total demands. The goal of the application placement problem is to optimize the fulfilment of loose requirements, while respecting the strict application requirements. Generally, strict requirements of applications are considered fixed for every application whereas loose requirements are variable[7]. Strict resource demands are indicated by Rds , loose resource demands by Rdl . The complete specification of the application placement function is: applace : Ra × Rds × Rdl × M 0 → M V. H IERARCHICAL MANAGEMENT In this section we will describe the hierarchical management structure. First we will describe the management hierarchy itself, after which we will explain how the centralized algorithm is used in individual management nodes. A. Hierarchical Management Structure A hierarchical management node organisation is dynamically created by executing “add node” operations for each of the servers to add them to an existing management hierarchy. As more nodes are added, the hierarchy will automatically restructure itself. The structure of the management hierarchy is inspired by that of B-Trees[13]. B-Trees are datastructures, used mainly for ordering large amounts of data. The most important characteristics of B-Trees are the large number of children for every node and an equal depth of tree leaf nodes. Every node in a B-tree, except for the root, contains between n and n/2 entries. The root itself is allowed to have any number of nodes between 0 and n.

In the hierarchical management scenario, less restrictions are needed as the hierarchy is only used to structure nodes, and not to order them. Adding nodes can be realised simply by adding them to any controller at the lowest level. Deleting execution nodes can be done trivially, whereas deleting management nodes is achieved by performing a leader election amongst its children. Furthermore, we change the restrictions at the root node: an imbalanced tree is allowed, but only at this level. This enables us to require a higher minimum node count in the root than in a regular B-Tree, as using a dedicated server to manage only a small number of child servers would be a waste of resources. As controllers gain more and more children, the execution time of applace continually increases. Eventually the execution time exceeds a given threshold, indicating that the node is overutilized. In the reverse case, if execution time gets too low, network delay overhead becomes a bigger concern than applace execution time, which indicates the node is underutilized. By executing applace for various server counts the number of children causing overutilization and underutilization can be determined. These values are Cmin and Cmax . If Cmin ≤ Cmax 2 , splitting and merging of nodes can be achieved using basic B-Tree operations. If Cmin > Cmax techniques 2 used in variations on B-Trees can be used where n nodes are considered and split into n + 1 nodes (as opposed to splitting a single node into two nodes in regular B-Trees). The splitting of overutilized nodes is illustrated in Figure 2. A node ni chooses a node nj ∈ children(ni ), which it adds to children(parent(ni )) it then chooses half the nodes remaining in children(ni ), which it adds as children to nj . The opposite situation is demonstrated in Figure 3, where an underutilized node is deleted and its children are added to its neighbours. In a bootstrap scenario server managers connect to each other and execute leader election to determine the root of the management hierarchy. Once this root is chosen the other nodes can be added to it using a default “add server node” operation. Nodes can find each other by providing an initial neighbour cache or by using separate approaches such as the Dynamic Domain Name System. Because there are only a few restrictions, this structure can be created and updated swiftly, enabling dynamic restructuring of the management hierarchy. Furthermore, the equal depth of the various nodes ensures the various nodes at each level are either all management servers or all execution servers, making the set of managed servers at each level more homogeneous. B. Algorithm Details We will now demonstrate how the general applace function can be executed in a hierarchical structure. As servers are grouped hierarchically, each server s has a parent parent(s) = p, p ∈ S and a set of children children(s) ∈ S, except for the root control node, which has no parent, and execution nodes, which have no children. Every management server executes the applace function. We will now consider a single cluster

..... n+1

..

..

⌊n / 2⌋

⌈n / 2⌉

Fig. 2. Solving overutilization by splitting a node and promoting a child node (grey) to peer status.

.....

.....

..

..

..

Fig. 3. Solving underutilization by removing a node and distributing its children (grey) amongst the node’s peers.

with management server m, which will execute the applace function with modified inputs and outputs: ˆ s , Rd ˆ l, M 0 ) ˆ Rd Mm = applace(Ra, m As Rds is generally taken to be static, it can be considered as application information, available to every management node involved in managing a specific application, ˆ s = Rds . M 0 is always present on each management so Rd m node, except in bootstrap scenarios where the general bootstrap scenario of the specific applace function needs to be used. ˆ l , which need to be aggregated across ˆ and Rd This leaves Ra the management tree. 1) Resource availability aggregation: Server resource inˆ at the lowest level can be observed from formation Ra the child nodes and can be used directly, so if a child ˆ r = Ra ˆ r for c ∈ children(m) is an execution server, Ra c c all resource types r. At higher levels, in every management server, an aggregated value of child resource information must be determined. Whereas a single server has a definite resource availability Ras , a cluster of servers does not. As the placement algorithm can only function using definite values for Ra, a value must be determined for the clusters by aggregating the values of its children. As a result, aggregated resource ˆ needs to be determined which indicates how information Ra many resources its parent can realistically expect to allocate on the cluster. This estimation will have to be revised whenever underutilization of resources or unrealistic resource allocation occur. For a resource r ∈ Γl and a cluster c an aggregated ˆ rc can be determined: estimation Ra low max ˆ rc = wEr,c Ra + (1 − w)Er,c low Er,c max Er,c

= =

Ccr

Ucr

+ X

(1) (2)

ˆr Ra n

(3)

n∈children(c)

The complete estimation is denoted in the first equation, with E low an estimate of available resources that will generally

ˆc Ra

ˆs Rd

ψ

D

Management Server c aggregation

be close, but slightly too low and E max an upper bound which will be too high. The actual ratio will be determined by a weight w, which will be determined experimentally. E max is the sum of all available resources of the children. Finally, E low combines two values: Ccr , the amount of resource r currently allocated in the cluster, and Ucr , an estimate of the remaining usable space on all servers on which more application instances can be instantiated:

decoupling ˆl Rd

M' applace

manages

Ccr Ucr

= =

Υ=

X

X

(Mc0 )rs,a s∈children(c) a∈A X X ˆ rs − (Ra (Mc0 )rs,a ) s∈Υ a∈A ˆ r− {s ∈ S|∀r0 ∈ Γs : Ra s

(4)

(Mc0 )rs,a


|S| the hierarchical approach is the same as the centralized algorithm. Once branching occurs a tree is formed. This temporarily increases both placement quality and allocation cost, as the management system repeats the allocation multiple times. As the datacenter size increases, allocation performance decreases while the difference between centralized execution times and hierarchical execution times increases. The higher Cmax , the higher the placement quality, but the slower the execution. Branching causes two types of performance penalties. As additional servers are used for the management process, they can no longer be used to execute applications, decreasing maximum achievable application satisfaction. Furthermore, information is fragmented by the process, allowing for overor under-allocation of application demand on clusters. The effects of the different performance penalties in the hierarchical approach are illustrated in Figure 8. Here we compare the satisfied demand of the centralized algorithm with that of the hierarchical approach and that of an adjusted

Average fulfilled application demand

120 Execution Time (s)

1

Centralized Hierarchical (50) Hierarchical (100) Hierarchical (200)

140

100 80 60 40 20 0 0

100

200

300 400 Server Count

500

600

0.8 0.75 0.7 0.65

100

200

0.9

0.85

0.8

0.75

300 Server Count

400

500

600

Fig. 8. Illustration of the management overhead and performance penalties induced by the hierarchy with a very low branching factor (Cmax = 10) 0.94

Centralized Hierarchical (50) Hierarchical (100) Hierarchical (200)

0.95

0.85

0

Average fulfilled application demand

Average fulfilled application demand

1

0.9

0.6

700

Fig. 6. Execution time of the hierarchical and centralized allocation strategies with varying server and application counts (|A| = |S|)

Centralized Hierarchical (10) Centralized (adjusted)

0.95

Centralized Hierarchical (50) Centralized (adjusted)

0.93 0.92 0.91 0.9 0.89 0.88 0.87 0.86

0.7 0

100

200

300 400 Server Count

500

600

700

0.85 50

100

150

200 250 Server Count

300

350

400

Fig. 7. Comparison of the average satisfied demand of the different allocation strategies (|A| = |S|)

Fig. 9. Illustration of the management overhead and performance penalties with a higher branching factor (Cmax = 100)

centralized algorithm, where servers used in hierarchical management are kept idle. In this case we used a management tree with an unrealistically low Cmax to illustrate both types of performance penalties. While the centralized approach uses all servers but one, where the algorithm itself executes, the adjusted centralized algorithm does not use any of the servers used in the management hierarchy. Because less servers can be used for actual application execution, the achievable placement quality is lower. As the hierarchical placement uses the same servers as management servers, the adjusted centralized placement offers an upper bound for the placement quality. The quality difference between the centralized approach and the adjusted centralized approach (in this specific case ±20%) is caused by the management hierarchy itself and stems from the choice to place the management infrastructure on the cloud itself. Not doing so would require a separate management infrastructure which would be less dynamic. The quality difference between the adjusted centralized approach and the hierarchical approach is caused by fragmentation of information and under-allocation. Picking higher a Cmax causes performance

penalties to decrease significantly while increasing allocation performance by repeated execution of the allocation algorithm, as illustrated in Figure 9, where hierarchical execution at times even surpasses centralized execution due to repeated executions. The performance of the hierarchical system is still impacted by the number of applications, as shown in Figure 10, where a constant number of applications is used (|A| = 50) and significantly better performance is achieved. The cause of this is that the number of applications also has an impact on the performance of the application placement algorithm. When considering a first placement, all applications need to be taken into consideration, leading to very high placement times. After this, some application responsibilities are delegated to specific instances, decreasing costs for subsequent placements. This implies that when many applications are in use, a lower branching factor should be chosen as this leads to more management nodes and thus more delegation of applications. Figure 11 illustrates the number of applications used in application placement per level. The graph illustrates the max-

60

Centralized Hierarchical (50) Hierarchical (100) Hierarchical (200)

Execution Time (s)

50

40

30

20

10

0 0

500

1000 1500 2000 2500 3000 3500 4000 4500 5000 Server Count

Fig. 10. Execution time of the hierarchical and centralized allocation strategies with fixed application counts (|A| = 50). Execution time of the centralized algorithm was measured up until 1000 servers.

Maximum applications per controller (%)

1.2

level 0 level 1 level 2

1

changing environments. At this point, using the hierarchical approach makes sense, as it is much faster and, thus, more scalable. The number of servers managed by each node has a large impact on the execution speed and on the quality of allocation, as it directly influences both the number of servers used in the management system and the quality of the placement itself. These results are in accordance with our earlier work concerning hierarchical management. Introducing hierarchies increases scalability of management systems, and eases distribution of context, in this instance calculated application demands. In future work, we intend to study and improve the robustness of the hierarchical management system. It would also be useful to add adaptiveness to the system by dynamically restructuring the management tree and adjusting system parameters, such as branching factor and estimation weights. These adjustments should lead to improved resource estimates, better resource allocations and faster placements. We also need to examine how the techniques demonstrated in this paper can be applied to other centralized cloud management algorithms. R EFERENCES

0.8

0.6

0.4

0.2

0 0

100

200

300

400 500 600 Server Count

700

800

900 1000

Fig. 11. The maximum number of applications known per node at different management levels (Cmax = 10)

imum number of applications used in application placement at a given tree level. The lower this number, the faster a placement at this level can occur. From the graph, we see that a large part of applications continues to be managed at the root level, but at lower levels, significantly less applications are managed per-node. This implies lower-level nodes can execute placement much faster. This enables the system to react fast based on local data, swiftly yielding a local solution, while still maintaining a globally good solution once higher levels have executed application placement. VII. C ONCLUSION In this paper we presented a hierarchical management system for cloud environments. Management nodes automatically order themselves in a structure inspired by that of B-Trees and each node executes a centralized placement algorithm for which inputs are generated by the management system. The centralized approach leads to higher placement quality at the cost of higher execution times. As datacenters scale, the execution time also increases, leading to slow reactions to

[1] J. Famaey, S. Latr´e, J. Strassner, and F. De Turck, “A hierarchical approach to autonomic network management,” 2010 IEEE/IFIP Network Operations and Management Symposium Workshops, pp. 225–232, Apr. 2010. [2] C. Tang, M. Steinder, M. Spreitzer, and G, “A scalable application placement controller for enterprise data centers,” in 16th international conference on World Wide Web, 2007, pp. 331–340. [3] F. Wuhib, R. Stadler, and M. Spreitzer, “Gossip-based Resource Management for Cloud Environments,” in 6th International Conference on Network and Service Management, 2010, pp. 1–8. [4] Y. Li, “Self-Adaptive Resource Management for Large-Scale Shared Clusters,” Science And Technology, vol. 25, no. 2009, pp. 945–957, 2010. [5] J. Rolia, A. Andrzejak, and M. Arlitt, “Automating Enterprise Application Placement in Resource Utilities,” IFIP International Federation For Information Processing, pp. 118–129, 2003. [6] T. Kimbrel, M. Steinder, M. Sviridenko, and A. Tantawi, “Dynamic application placement under service and memory constraints,” in International Workshop on Efficient and Experimental Algorithms, Apr. 2005. [7] A. Karve, T. Kimbrel, G. Pacifici, M. Spreitzer, M. Steinder, M. Sviridenko, and A. Tantawi, “Dynamic placement for clustered web applications,” in Proceedings of the 15th international conference on World Wide Web. ACM, 2006, pp. 595–604. [8] D. Carrera, M. Steinder, I. Whalley, J. Torres, and E. Ayguad´e, “Utilitybased placement of dynamic web applications with fairness goals,” in Network Operations and Management Symposium, 2008. NOMS 2008. IEEE, 2008, pp. 9–16. [9] C. Adam and R. Stadler, “Service Middleware for Self-Managing LargeScale Systems,” IEEE Transactions on Network and Service Management, vol. 4, no. 3, pp. 50–64, Dec. 2007. [10] C. Low, “Decentralised Application Placement,” Future Generation Computer Systems, vol. C, no. 21, pp. 281–290, 2005. [11] S. Schneider, A. Meisel, and W. Hardt, “Communication-Aware Hierarchical Online-Placement in Heterogeneous Reconfigurable Systems,” in IEEE/IFIP International Symposium on Rapid System Prototyping, 2009, 2009, pp. 61 –67. [12] B. Hudzia, M.-T. Kechadi, and A. Ottewill, “TreeP: A Tree Based P2P Network Architecture,” 2005 IEEE International Conference on Cluster Computing, pp. 1–15, Sep. 2005. [13] R. Bayer and E. McCreight, “Organization and Maintenance of Large Ordered Indices,” in Proceedings of the 1970 ACM SIGFIDET (now SIGMOD) Workshop on Data Description, Access and Control. Boeing Scientific Research Laboratories, 1970, pp. 107–141.