Decentralized Resource Availability Prediction for a ...

Decentralized Resource Availability Prediction for a Desktop Grid

Karthick Ramachandran

Dr. Hanan Lutfiyya

Dr. Mark Perry

Distributed and Grid Systems, Department of Computer Science University of Western Ontario

4th Workshop on Desktop Grids and Volunteer Computing Systems (PCGRID 2010)

Motivation

Focus of our work

Architecture

Implementation and Experiments

Conclusion

Outline

1

Motivation

2

Focus of our work

3

Architecture

4


5

Conclusion

Decentralized Resource Availability Prediction for a Desktop Grid 2 / 25

Motivation

Focus of our work

Architecture


Conclusion

Motivation

“Companies across US are wasting $2.8 billion every year powering 1.8 million unused computers during night” - PC Energy Report 2009 Decentralized Resource Availability Prediction for a Desktop Grid 3 / 25

Motivation

Focus of our work

Architecture


Conclusion

Volunteer Computing Volunteer computing (desktop grid) is a type of distributed computing in which computer owners donate their computing resources (such as processing power and storage) to one or more projects.

Characteristics Leverages the idle CPU cycles for number crunching tasks Answer to the demand for huge processing power


Motivation

Focus of our work

Architecture


Conclusion

The Problem

? Can the system behave the same way irrespective of the number of participating nodes? (Scalability) What is the guarantee that the peer selected for job execution will not terminate the execution because of some user activity? (Predictability)


Motivation

Focus of our work

Architecture


Conclusion

Focus of our work

The focus of this system is to create a desktop grid that minimizes centralization of resource selection process that can do resource availability prediction We build a P2P desktop grid framework which does resource availability prediction

None of the work we surveyed has taken the approach of performing availability prediction in a P2P environment


Motivation

Focus of our work

Architecture


Conclusion

Approach - Scalability

Peer to Peer Network Peers act as equals, merging the roles of clients and server There is no central server managing the network There is no central router


Motivation

Focus of our work

Architecture


Conclusion

Approach - Resource Availability Prediction

Resource Availability Prediction Recording the resource availability pattern over a period of time Using the saved data to calculate the probability of the machine availability at a given time of the day of the week


Motivation

Focus of our work

Architecture


Conclusion

Stakeholders

Desktop User Desktop Grid Service Client Desktop Grid Service Provider Job is a work unit which the Desktop Grid Service Client wants to execute in the desktop user’s machine.


Motivation

Focus of our work

Architecture


Conclusion

Overview Management Server

Job Server

P2P Desktop Cloud (Collection of all desktops)

Monitoring Server

Result Server

Figure: Birds eye view of the system Decentralized Resource Availability Prediction for a Desktop Grid 10 / 25

Motivation

Focus of our work

Architecture


Conclusion

Client Software Code Executor in Sandbox (Java Virtual Machine)

Monitoring and Prediction Engine Resource Discovery Module

Resource and Group Availability Data

Peer Communication Module Underlying P2P Framework (JXTA)

Figure: Components of Client Software Decentralized Resource Availability Prediction for a Desktop Grid 11 / 25

Motivation

Focus of our work

Architecture


Conclusion

Job

Components of work unit are encapsulated in a mobile agent to form a job A mobile agent is a software module which moves from node to node autonomously and is executed at each node it moves to The mobile agent is responsible for the following Migration Communication with Monitoring and Result Server


Motivation

Focus of our work

Architecture


Conclusion

Prediction Engine Group 1

Group 2

Dedicated Desktops Non-dedicated Desktops (Group 1) Non-dedicated Desktops (Group 2)

Figure: Prediction Engine Decentralized Resource Availability Prediction for a Desktop Grid 13 / 25

Motivation

Focus of our work

Architecture


Conclusion

Prediction Engine

The peer-to-peer system is used to classify the network into groups (g1 , g2 ... gn ). Every machine in the group returns the following parameters, in response to the resource request query. Resource Availability Factor (raf ) Group Availability (ga)


Motivation

Focus of our work

Architecture


Conclusion

raf and ga The resource availability factor (raf ) denotes the probability of an individual machine’s availability at a given time of the day in a week. raf (t) =

Navailable (t) Ntotal (t)

The group availability quantifies the total group availability at a given time of the day in a week (t). This is represented by the following: ga(t) = currentga (t) − avgb (t)


Motivation

Focus of our work

Architecture


Conclusion

Probability of Resource Availability

For non-dedicated desktops (university lab), Presourceavailability = (raf (t) ∗ ga(t))

For dedicated desktops (individually owned machines), Presourceavailability = raf (t)


Motivation

Focus of our work

Architecture


Conclusion

Softwares Used

JDK 1.5 Java Axis2 webservices JXTA P2P Framework Aglets agents framework C++ MySQL SqlLite GNU/Linux and MS Windows XP


Motivation

Focus of our work

Architecture


Conclusion

Verification and Validation

Scalability - Real time experiments

Efficiency of Prediction Algorithm - Simulation


Motivation

Focus of our work

Architecture


Conclusion

Scalability Testing Management Server

Job Server 1

Monitoring Server 1

Job Server 2

Monitoring Server 2

Result Server 2

Resource 1

Resource 4

Resource n-2

Resource 2

Resource 5

Resource n-1

Resource 3

Resource 6

Resource n

Result Server 1

HFT Cluster Machines

n >= 20

Desktop Machine (Windows) + HFT Cluster Workstations

Figure: Scalability Testing Decentralized Resource Availability Prediction for a Desktop Grid 19 / 25

Motivation

Focus of our work

Architecture


Conclusion

Simulation of Prediction models

Random Scheduling - Simulating the first come first serve model Single Machine Predictive Scheduling - Simulating the predictive algorithm based on the single machine prediction without group availability Group Predictive Scheduling - Simulating the predictive algorithm based on the group algorithm


Motivation

Focus of our work

Architecture


Conclusion

Results Input (num_of_jobs, duration, time) 10, 30, 2008-12-01 12:00:00

10, 100, 2008-12-01 12:00:00

10, 200, 2008-12-11 10:00:00

10, 300, 2008-12-11 10:00:00

100, 30, 2008-12-01 12:00:00

100, 100, 2008-12-01 12:00:00

100, 100, 2008-12-11 10:00:00

1000, 100, 2008-12-11 10:00:00

Sched. type R GP SP R GP SP R GP SP R GP SP R GP SP R GP SP R GP SP R GP SP

Migrations

Elapsedtime

4 0 1 10 2 6 16 1 0 19 14 18 55 49 50 427 296 284 2 3 3 42 34 33

35 minutes 30 minutes 35 minutes 1:50hrs 1:45 hours 1:45 hours 3:30hrs 3:25hrs 3:20hrs 5:20hrs 5:15hrs 5:15hrs 2:00hrs 1:55hrs 2:00hrs 8:25hrs 7:05hrs 7:05hrs 3:25hrs 3:30hrs 3:30hrs 1 day 3:55hrs 1 day 3:55hrs 1 day 3:55hrs

Table: Simulation Results Decentralized Resource Availability Prediction for a Desktop Grid 21 / 25

Motivation

Focus of our work

Architecture


Conclusion


10, 100, 2008-12-01 12:00:00

10, 200, 2008-12-11 10:00:00

10, 300, 2008-12-11 10:00:00

100, 30, 2008-12-01 12:00:00


Migrations

Elapsedtime

4 0 1 10 2 6 16 1 0 19 14 18 55 49 50 427 296 284 2 3 3 42 34 33


The predictive scheduling decreases the amount of 100, 100, 2008-12-01 12:00:00 migrations when num_of _jobs < num_of _machines 100, 100, 2008-12-11 10:00:00

1000, 100, 2008-12-11 10:00:00


Motivation

Focus of our work

Architecture


Conclusion

Results Input (num_of_jobs, duration, time)


Migrations

Elapsedtime

4 0 1 10 2 6 16 1 0 19 14 18 55 49 50 427 296 284 2 3 3 42 34 33


10, 30, 2008-12-01 12:00:00 predictive scheduling is more During peak hours,

effective 10, 100, 2008-12-01 12:00:00 10, 200, 2008-12-11 10:00:00

10, 300, 2008-12-11 10:00:00

100, 30, 2008-12-01 12:00:00

100, 100, 2008-12-01 12:00:00

100, 100, 2008-12-11 10:00:00

1000, 100, 2008-12-11 10:00:00


Motivation

Focus of our work

Architecture


Conclusion


10, 100, 2008-12-01 12:00:00


Migrations

Elapsedtime

4 0 1 10 2 6 16 1 0 19 14 18 55 49 50 427 296 284 2 3 3 42 34 33


Predictive scheduling also makes a difference with long10, 200, 2008-12-11 10:00:00

duration jobs (> 45 minutes) 10, 300, 2008-12-11 10:00:00

100, 30, 2008-12-01 12:00:00

100, 100, 2008-12-01 12:00:00

100, 100, 2008-12-11 10:00:00

1000, 100, 2008-12-11 10:00:00


Motivation

Focus of our work

Architecture


Conclusion


10, 100, 2008-12-01 12:00:00

10, 200, 2008-12-11 10:00:00

10, 300, 2008-12-11 10:00:00


Migrations

Elapsedtime

4 0 1 10 2 6 16 1 0 19 14 18 55 49 50 427 296 284 2 3 3 42 34 33


When100,num_of _jobs > num_of _machines 30, 2008-12-01 12:00:00

the predictive scheduling is not that effective 100, 100, 2008-12-01 12:00:00

100, 100, 2008-12-11 10:00:00

1000, 100, 2008-12-11 10:00:00


Motivation

Focus of our work

Architecture


Conclusion

Related Work Peer to Peer Grids Kim et.al - P2P desktop grid built over Distributed Hash Tables El-Desoky et.al - Fault tolerant P2P desktop grid Drost et.al - proposed a P2P supercomputing middleware system that was locality aware Resource Availability Selection RIDGE (Budati et.al) - Reliability Ratings over a client/server architecture TMDG (L Shang et.al) - Client/Server with Trust Model to predict machine availability Kondo et.al - Identify groups of internet resource with correlated availability Decentralized Resource Availability Prediction for a Desktop Grid 22 / 25

Motivation

Focus of our work

Architecture


Conclusion

Contribution

Development of a robust desktop grid which takes into account multiple job servers, mobility of the job and desktop volatility An approach to decentralized resource availability prediction of both dedicated and non-dedicated desktops


Motivation

Focus of our work

Architecture


Conclusion

Future Work Resource Discovery Improvements to JXTA’s basic filtering of resources Studying the system’s performance with more than 20 clients Resource Availability Prediction Study of efficiency of models in dedicated resources Implementation of advanced mathematical models and probabilistic models in the system. Trust Socio-Economic Models Fault Tolerance Leveraging Free Disk Space Decentralized Resource Availability Prediction for a Desktop Grid 24 / 25

http://digs.csd.uwo.ca/

Thank you! http://www.csd.uwo.ca/~kramach/presentation/pcgrid2010.pdf