AbstractâWe propose a suite of market-oriented task schedul- ing algorithms to build ... distributed computing platforms, such as P2P, grid, and cloud computing ...
AuctionNet: Market Oriented Task Scheduling in Heterogeneous Distributed Environments Han Zhao (Ph.D. Student, Fourth Year) and Xiaolin Li (Advisor) Scalable Software Systems Laboratory, Department of Computer Science Oklahoma State University, Stillwater, OK 74078, USA Email: {haz, xiaolin}@cs.okstate.edu Abstract—We propose a suite of market-oriented task scheduling algorithms to build an AuctionNet for heterogeneous distributed environments. In heterogeneous distributed environments, computing nodes are autonomous and owned by different organizations, for example peer-to-peer systems, desktop grids/clouds. To address such diverse heterogeneity and dynamism in systems, applications, and local policies, efficient and fair task scheduling becomes a challenging issue. To cope with such complexity in a distributed and noncooperative environment, we propose to use market-oriented incentive mechanisms to regulate task scheduling in a distributed manner. Further, to accommodate multiple objectives and criteria, we adopt a combined approach leveraging the advantage of both hypergraph theory and incentive mechanisms. We first formulate a general framework of market-oriented task scheduling in distributed systems. We then present two algorithms for task-bundle scheduling. Preliminary results demonstrate the satisfactory performance of our proposed algorithms. The remaining work to complete the PhD dissertation is then presented. The proposed research carries significant intellectual merits and potential broader impacts in the following aspects. (1) We propose the notion of task-bundle for the first time in the literature. Product-bundle has been a common marketing strategy in our daily life for a long time. In the emerging commercial clouds and desktop clouds, task-bundle could be a useful concept for computing and storage markets. (2) We propose efficient distributed mechanisms that are very suitable for such distributed systems. A novel algorithm combining hypergraph and incentive mechanisms achieves multi-objective optimization. (3) We conduct rigorous analytical study and prove that our algorithms ensure efficiency and fairness and in the meantime maximize social welfare. (4) Overall, this proposal lays a solid foundation and sheds light on future research and realworld applications in the broad area of task scheduling in distributed systems.
I. I NTRODUCTION The fast development of parallel and distributed computing paradigms, driven by increasing demand for computing power and network bandwidth, spurs a variety of massively distributed computing platforms, such as P2P, grid, and cloud computing emerging in academic and industrial communities. Of all the research issues in the field of distributed computing, task scheduling, which concerns the mapping of parallel tasks to heterogeneous computers, is among the most important research topics. Conventional methods for task scheduling, mostly centralized, are difficult to adapt to the growing complexity of modern heterogeneous distributed environments. The research presented in this paper is supported in part by National Science Foundation (grants CNS-0709329 and OCI-0904938).
This growing complexity is mainly caused by two reasons. Firstly the system scale increases significantly when compared to the past, growing from hundreds of machines in a single laboratory to thousands of geographically distributed cooperative computers. Secondly, each single computer features autonomy and volatility, making central coordination extremely hard. To overcome these difficulties, researchers have proposed and investigated numerous scheduling methods to accommodate the new computing environments both theoretically and practically. One of the promising method is to use market oriented mechanisms to regulate the scheduling decision making process. The market oriented mechanisms regard each distributed computing machine as economically rational individuals in the human society, and try to characterize the cooperation and competition process using economic theory. Such marketoriented scheduling models are very suitable because they precisely capture essential features of modern heterogeneous distributed environments. This research proposal is aimed to model the large-scale distributed systems, and to design effective mechanism enabling site cooperation towards solving computationally hard problems. As a key subsystem of our CoopNet project, AuctionNet offers holistic mechanisms and policies to ensure efficient and fair resource sharing and task scheduling in distributed systems. As a holistic autonomic distributed system, CoopNet consists of three subsystems: AuctionNet for resource management (self-optimization), TrustNet for trust management (self-protection), and AmberNet for networking and messaging (self-configuration and self-healing). The rest of this proposal is organized as follows. Section II presents an overview of related work and identifies weakness of current research. Section III presents our proposed research with preliminary results. Finally Section IV presents future research plans. II. R ELATED W ORK In recent years we have witnessed a burst of research efforts, investigating the application of economic and game theory for task and resource scheduling in heterogeneous distributed systems [1], [3], [13]. This trend is largely contributed to the following observations: design similarity of economic market mechanisms and distributed system scheduling principles; and role similarity of realistic rational individuals and egocentric heterogeneous computers. Therefore market oriented methods derived from game theory is extremely helpful in modeling
Job Route
In this section we present two preliminary proposed research projects. The first work proposes a bargaining protocol using combinatorial auction, and the second one proposes a novel task scheduling framework based on directed hypergraph for multi-criteria scheduling performance optimization. Preliminary studies have been accepted for publications in CCGrid 2009 and IPDPS 2010 [14], [15]. A. Task Scheduling Using Bargaining Based Self-Adaptive Auction System Model: As the scale of the grid system grows, the flat organization of traditional grids no longer holds. Instead, the fast development of communication technologies paves the way for enhanced grid cooperation among different organizations. In this research we adopt the hierarchical grid structure modeling large-scale computational grid system, as shown in Fig. 1. Computational jobs submitted by the grid user (GU) are decomposed into embarrassingly parallel tasks (e.g., BoT [6]) and inserted into the FIFO queue waiting to be scheduled. Two levels of schedulers exist in such a model. The first level global scheduler (GS) coordinates intersite cooperations, while the second level local site schedulers (LSS) institute intra-site interactions. The end grid users and computing peers (CP) are functionally symmetric, indicating that the users are also resource contributors. Proposed Approaches: In this work we focus on modeling and designing scheduling strategies at intra-site level. Given task valuation functions for each node, our goal is to devise
GU CP
Job FIFO Job
Registration
Queue
FIFO Job Queue
LSS GS
Task Bidding
III. P ROPOSED R ESEARCH
Task Route
Job Bidding
behaviors of benefit-driven agents. Methods using game theory converge to system equilibrium state on the basis of revenue maximization. The key challenge for researchers is to identify a suitable objective function that defines target performance optimization in terms of utility. Example applications include dynamic resource sharing [11], promoting incentives in grid and P2P systems [8], and some practical projects Popcorn [10], Nimrod/G [3] and G-commerce [13]. Market oriented methods can be categorized as cooperative [9] and noncooperative [7]. We adopt a hybrid approach that at global level sites are more willing to cooperate with each other while at local level each resource contributor is selfish, seeking maximum revenues for their own interests. This assumption fits well in modern distributed environments, e.g., P2P desktop grid systems such as Cohesion [12] and OurGrid [5]. Despite significant progress has been made so far, existing market-oriented task scheduling methods are inadequate in several aspects, specifically, (1) inaccurate modeling of egocentric agent behaviors; (2) lack of scheduling mechanism design for multi-criteria optimization; and 3) most marketoriented mechanisms used in the literature are left behind from the progress in related economic studies. This proposal attempts to fill the gap and take advantage of both advanced economic methods and graph/hypergraph theories to tackle challenging task scheduling issues in heterogeneous distributed systems.
CP
P2P Network
CP
FIFO CP
Task Queue
LSS
Execution Feedback
Fig. 1.
Results Collection
Job execution flow chart for hierarchical grid model
an allocation strategy that achieves the highest scheduling efficiency. The evaluation of scheduling efficiency is measured in term of social welfare, which is defined as the maximum of valuation summation for all possible allocations. To accomplish this goal we propose a Bargaining based Self-Adaptive Auction (BarSSA). BarSSA is a combinatorial auction form that is easy to implement and effectively achieves maximum allocation efficiency. It is different from the single-sided oneshot auction forms in that it allows bidders to better express valuations for improved market efficiency. The BarSSA algorithm presents the auction strategies for both bidders (computing peers) and auctioneer (local site scheduler). It proceeds iteratively with adaptive bargaining between auctioneer and bidders until market equilibrium is reached. At each round of negotiation bidders dynamically adjust their values for tasks based on their observations of market demand-supply situation. Through analytical presentations we show that BarSSA effectively achieves efficient allocation result at market clearing price upon convergence. Moreover, we prove that BarSSA promotes truthful bidding behaviors and prevents cheating for all bidders as truthful bidding is weakly dominant and will lead to expost perfect equilibrium. Preliminary Results: In addition to theoretical analysis we perform extensive simulations to validate that BarSSA achieves maximum efficiency and is incentive compatible. The BarSSA algorithm was implemented using SimGrid framework [4] and was conducted on the scale of 20 computing nodes. The results are exhibited in Fig. 2. The simulation was carried on with 32 rounds of jobs arriving. The task number was increased at each round. From Fig. 2(a) we can observe that the resulting social welfare is always higher compared with baseline random allocation. The effects of truthful bidding against trick playing is shown in Fig. 2(b). The observation shows that trick bidding strategies, either humble reporting or bluffing, decreases profits gained by each computing node, which will in turn result in honest bidding. At last in Fig. 2(c) we show that varying the system heterogeneity will impact the scheduling fairness in BarSSA. B. Multi-criteria Hypergraph-based Task Scheduling Scheduling Model Based on Directed Hypergraph: In previous work we proposed BarSSA algorithm for maximum
0.7
hf = 0.15 hf = 0.25 hf = 0.36
0.65 0.6
Single Bidder Profits (G$)
Social Welfare (G$)
3500
Bluffing Humble Truthful
120
Random Bluffing Humble Truthful
3000
2500
2000
100
0.55
Fairness Index
4000
80
60
0.4 0.35 0.3 0.25
40
1500
0.5 0.45
0.2 1000 0
5
10
15
20
25
30
35
0.15
20 0
5
10
Round
15
20
25
30
Round
(a) Social welfare
(b) Single bidder profits
35
0
10
20
30
40
50
60
70
80
Maximum Bidding Number
(c) Effects of heterogeneity
Fig. 2. Simulation results of decomposed task allocation. (a) overall social welfare using different strategies with round; (b) single bidder profits using different strategies with round; (c) fairness index affected by heterogeneity with fixed maximum bidding number.
task allocation efficiency. However another important scheduling feature fairness is not taken into account. In this research we extend previous work under the same system and task model. Using this model we proposed a unified hypergraph scheduling model under which efficiency and fairness are compatible with each other. In our model fairness is defined as envy-freeness [2], indicating that no peer would get better off (utility) by exchanging its current allocation with another peer through rational deals. Given arbitrary initial allocation projecting tasks onto processors, the goal is to develop a task exchange strategy for autonomous peers in order to achieve efficiency and fairness simultaneously. To accomplish this goal the two seemingly contradict objectives should be unified in one model. Hypergraph provides a good modeling solution. A hypergraph is a mathematical extension of conventional graph connecting arbitrary sets of vertices rather than two. In order to represent allocation efficiency and fairness we propose Allocation Matrix (AM) and Envy Matrix (EM) respectively. The first one represents current task allocation situation while the second matrix records envy relations among computing peers. With these two matrices we propose a directed hypergraph model as a 3D matrix. The model is shown in Fig. 3: Envy Matrix (E)
Hyper-node x (Tid, Pcur, Λ) n
n
n n
Allocation Matrix (A)
m
m
n
Fig. 3. Hypergraph Allocation Model. The proposed model is derived from an m×n×n matrix. A hypergraph vertex is represented by nonzero elements on the column of EM matrix (surface). Each processor launches a hyper-arc (not shown) pointing to the hyper-nodes whose corresponding task-bundle is more appreciated on that processor, but allocated to others.
Using this directed hypergraph model we propose the distributed negotiation process for task allocation as follows: each peer negotiates with its neighbors using Rational Deals (RD),
and requests task-bundle represented by hyper-nodes through hyper-arcs. Fig.4 displays an example of task-bundle exchange between two processors.
Pi
payment
Pj
Fig. 4. Example of task-bundle exchange between pi and pj . The ellipses represent computing peers while the diamonds represent tasks waiting to be allocated. Take pi as an example, the red line going out of pi stands for a hyper-arc. Therefore there are three peers in pi ’s envy set including pj . In pj three tasks (hyper-nodes) have higher valuations in pi and they are included in the red dashed circle. Hyper-nodes are moved from pj to pi and corresponding payments are paid by pi .
Proposed Approaches: We consider two scenarios with and without budget constraint respectively. Each peer performs two activities at each deal negotiation: offer making and offer selection. For task allocation without budget constraint (denoted as budget-unawareness), as long as the envy set for each computing peer is not empty and there are offers available, it randomly picks its neighbor using its hyperarc and makes offer using predefined prices. If the offer is accepted the hyper-node is removed and envy set is reevaluated accordingly. For budget-aware negotiation we modified the previous algorithm by proposing a local search negotiation strategy using hill climbing (HCN). When there are multiple valid offers available we proposed three versions of HCN in favor of valuation (V-HCN), envy degree (E-HCN), and profits (U-HCN) respectively. Due to the space limitation we refer the readers to [15] for detailed algorithm description. Preliminary Results: As in the previous work we implemented the algorithms on SimGrid and investigated the performances through extensive simulations. The selected research results are shown in Fig. 5, in which we investigated the impact of different payment strategies and initial budget settings. In Fig. 5(a) shows that more aggressive bidding behaviors will result in higher system profits after each deal. Next we selected E-HCN as the base algorithm and experimented with different
11
160
Rich Middle Class Poor
1200 10
Rich Middle Class Poor
140 1000
Profits
800
600
400
Envious Peer Number
Total Envy Degree (log)
9 8 7 6 5
120
100
80
60
4 200
0 0
5
10
15
20
25
Deal Number
(a) Social welfare
30
40
3
Aggressive Modest Conservative 35
40
20
2 0
5
10
15
20
25
30
35
Deal Number
(b) Single bidder profits
40
45
0
5
10
15
20
25
30
35
40
45
Deal Number
(c) Effects of heterogeneity
Fig. 5. Simulation results of hypergraph-based task-bundle allocation: (a) impact of bidding strategy for modified U-HCN; (b) impact of initial budget assignment for V-HCN: envy degree; (c) impact of initial budget assignment for V-HCN: envious pair number.
amounts of initial budget assignment. The results are shown in Fig. 5(b) and Fig. 5(c) respectively. Based on the observation we draw the conclusion that with abundant initial funds, the convergence to the local minimum envy degree is close to the budget-unaware scenario, whereas for the system with insufficient funds the offer making process is more likely to stuck due to lack of budgets, resulting in longer convergence rate and higher local minimum envy. IV. F UTURE W ORK Our preliminary study shows the strength of marketoriented mechanisms in ensuring efficiency and fairness in task scheduling for distributed systems. We plan to make further efforts in this research area as follows. (1) Holistically address task scheduling issues for both computation- and data-intensive applications. Our previous work focused mainly on computational tasks. Emerging data-intensive applications call for efficient coordinated sharing of resource, dynamic load balancing, and intelligent data placement and replica management. We plan to leverage incentive mechanisms and swarm intelligence algorithms to holistically address several key constraints and design objectives in such a context. (2) Design efficient and fair mechanisms for emerging desktop clouds for resource sharing of both computation cycles and storage under the framework of our CoopNet project. Current data/file sharing systems are centered around peer-to-peer overlay topology management and routing protocols. Little attention has been paid to design incentive mechanisms for emerging desktop cloud platforms. We expect our work will lay a solid foundation to shape an important “market” of the emerging cloud computing and storage economics. R EFERENCES [1] E. Anshelevich, A. Dasgupta, J. Kleinberg, E. Tardos, T. Wexler, and T. Roughgarden. The price of stability for network design with fair cost allocation. In Proceedings of the 45th Annual IEEE Symposium on Foundations of Computer Science (FOCS 04), pages 295–304, 2004. [2] S. J. Brams and A. D. Taylor. Fair division: from cake-cutting to dispute resolution. Cambridge University Press, 1996. 272 pp. [3] R. Buyya, D. Abramson, and J. Giddy. Nimrod/G: An architecture of a resource management and scheduling system in a global computational grid. In Proceedings of the 4th International Conference on High Performance Computing in the Asia-Pacific Region, volume 1, pages 283–289, 2000.
[4] H. Casanova, A. Legrand, and M. Quinson. SimGrid: a Generic Framework for Large-Scale Distributed Experiments. In 10th IEEE International Conference on Computer Modeling and Simulation, 2008. [5] W. Cirne, F. Brasileiro, N. Andrade, L. Costa, A. Andrade, R. Novaes, and M. Mowbray. Labs of the world, unite!!! Journal of Grid Computing, 4(3):225–246, September 2006. [6] A. Iosup, O. Sonmez, S. Anoep, and D. Epema. The performance of bags-of-tasks in large-scale distributed systems. In Proceedings of the 17th international symposium on High performance distributed computing (HPDC 08), pages 97–108, 2008. [7] Y.-K. Kwok, S. Song, and K. Hwang. Selfish grid computing: gametheoretic modeling and NAS performance results. In CCGRID, pages 1143–1150, 2005. [8] K. Ranganathan, M. Ripeanu, A. Sarin, and I. T. Foster. Incentive mechanisms for large collaborative resource sharing. In CCGRID, pages 1–8, 2004. [9] R. Ranjan, M. Rahman, and R. Buyya. A decentralized and cooperative workflow scheduling algorithm. Cluster Computing and the Grid, IEEE International Symposium on, 0:1–8, 2008. [10] O. Regev and N. Nisan. The POPCORN market. online markets for computational resources. Decision Support Systems, 28(1-2):177–189, 2000. [11] K. Rzadca, D. Trystram, and A. Wierzbicki. Fair game-theoretic resource management in dedicated grids. In Proceedings of the Seventh IEEE International Symposium on Cluster Computing and the Grid (CCGRID 07), pages 343–350, 2007. [12] S. Schulz, W. Blochinger, M. Held, and C. Dangelmayr. Cohesion a microkernel based desktop grid platform for irregular task-parallel applications. Future Generation Computer Systems, 24(5):354–370, 2008. [13] R. Wolski, J. S. Plank, J. Brevik, and T. Bryan. G-commerce: Market formulations controlling resource allocation on the computational grid. In Proceedings of the 15th International Parallel & Distributed Processing Symposium (IPDPS 01), pages 46–53, 2001. [14] H. Zhao and X. Li. Efficient grid task-bundle allocation using bargaining based self-adaptive auction. In IEEE International Symposium on Cluster Computing and the Grid (CCGrid 09), volume 0, pages 4–11, 2009. [15] H. Zhao, X. Liu, and X. Li. Hypergraph-based task-bundle scheduling towards efficiency and fairness in heterogeneous distributed systems. In (accepted) Proceedings of the 24th IEEE International Parallel and Distributed Processing Symposium (IPDPS 10), 2010.
