Data Grid: Supporting Data-Intensive applications in Wide-Area Networks. Xiao Qin and Hong Jiang. Department of Computer Science and Engineering.
DataGrid:SupportingData-Intensiveapplications inWide-AreaNetworks
XiaoQinand
HongJiang
DepartmentofComputerScience andEngineering University oNebraska-Lincoln, f Lincoln, NE68588-0 {xqin, jiang}@cse.unl.edu
TechnicalReport No. TR-03-05-01 May2003 tyoNebraska-Lincoln f Lincoln, NE 68588-0115
115
Abstract Adatagridisacollectionofgeographicallydispersedstorag network.Thegoalof
eresourcesoverwidearea
datagridsystemistoprovidealargevirtualstorageframewor
kwith
unlimitedpowerthroughcollaborationamongindividuals,institutions,andresourc study, we first review naumber of data grid services such as,
to just name few, a metadata
dataaccessservice,andperformancemeasurementservice.Wet techniquesforboostingtheperformanceodata f grid,andthesek
service,
heninvestigatevarious eytechniqueshavefalleninto
threecamps,namely,datareplication,scheduling,anddatamovement.Si becomes one othe f prime concerns in grid computing, we review two int indatagrid.Finally andimportantly,wehaveidentifiedfiveinte outsome potentialsolutionsto
es.Inthis
ncesecurityissue eresting issues of security restingopenissues,andpointed
the open problems.
1
1. Introduction A grid icasollection ogeographically f dispersed computing resource
s, providing large a virtual
computingsystemtousers.Therearefourcommonlyusedkindsofres
ourcesingrids:
computation,storage,communications,andsoftware.In whatfollows, four t
ypesof resourcesare
briefly introduced. • Computation. Computingresourcesuppliedbytheprocessorsotfhemachinesisthe commonresourceingrids.Thisresourceihs eterogeneousin ature processorswith variety a ospeeds, f architectures,andsoftwarepl
most
since , iistcombination a of atforms.Ajobsubmittedbyan
endusercanutilizecomputationresourcesbyeitherrunningonasin
gleavailablemachineor
beingbrokendownintonumber a ofsub-jobstobexecutedinparallelonm
ultiplemachinesin
the grid. • Storage. Storage iviewed s athe s second most commonly used resource igan presents an integrated view of storage icoined s a“Data s grid”
rid. The grid that or “Data Grid”. Memory, hard disk,
andotherpermanentstoragemediaarereferredtoasstorager
esources.Inthisstudy,weare
particularly interestedinsecondarystorageinthegrid,since
secondstoragei1000 s timesslower
than the memory attached tprocessor. ao Throughout this report, we
only address the issues of data
gridwithrespecttosecondary storage. Many Networked file systems, which exhibit security and relia applied tdoata grid. These file systems include: Network File S
bility features, have been widely ystem (NFS) [27], Distributed File
System (DFS), Andrew File System (AFS) [18], General Para
llel File System (GPFS), and Parallel
VirtualFile System(PVFS) [15][43]. • Communication. The data grid becomesincreasingly moreattractivewith the rapi communication capacity, since the high performance data grid re movingdataandjobsamongthesitesinthegrid.Toenhancethereliabi 2
dgrowth in
sts on the available bandwidth for lityofthegrid,itis
desirabletoapplyredundantcommunicationpaths,whichics apableoaf ll burdenimposedbyexcessivedatatraffic.Amonitorsystemasso
eviatingthenetwork ciatedwithagridmanagement
canbdeevelopedtoidentify the communicationbottlenecksin grid. a • Software.Incasewheresoftwareisnotinstalledoneverymachineinthe scheduler, inaddition tboalancing the grid load, can assign jobs requirin
grid,thegrid tghe specific software to
machinesthatisable tomeetthe jobs’ needs. Theamountofscientificdatageneratedbysimulationsorcollec experimentsigs enerallylarge,andsuchdatatendstobegeog
tedfromlarge-scale raphicallystoredoverwide-area
networksforthesakeolfarge-scalecollaborations.Thenotionofc
omputationalgridhasbeen
proposedforseveralyears,mainlyfocusingoneffectiveusageofg
lobalcomputationaland
networkresourcesinwide a areanetwork/Internetenvironment.Howeve computationalgrid,wheretheeffectiveusageofstorageresourc
r,theperformanceoaf esisignored,willdecrease
substantiallyivast f majorityoapplications f runningintheg
ridaredata-intensive.Toovercome
thisproblem,varioustechniqueshavebeendevelopedandincorporatedinto soa infrastructure.Inthisreport,wewilltakeacloselookatthese
calleddatagrid techniquespresentedinthe
literature,andpointoutsome openissuesindata grid research. Therestofthereportisorganizedafsollows.Inthesectiontha
follows, t threemajor datagrid
servicesarereviewed.InSection3techniques , forimprovingtheperf described idnetail. Security issue idnata grid ibriefly s re
ormanceodf atagridare viewed iS n ection 4Section . identifies 5
several open issues, and presents the potential and possible solutions t Section6concludesthereportbysummarizingthemainopenproblemsand
tohe open problems. Finally, ourpreliminary
solutions.
2. DataGrid Services Theobjectiveodafatagridsystemistwo-fold.First,tointegr
ateheterogeneousdataarchives 3
storedilarge na numberofgeographically distributedsitesi
nto single a virtualdatamanagement
system.Second,toprovidediverseservicestofittheneedsohighf
performancedistributedand
data-intensivecomputing.Inthissection,wefocusonthreeessential
services:metadataservice,
data accessservice,andperformance measurement. 2.1MetadataServicesfor Data GridSystems TheMetadataodafatagridsystemisthemanagementinformat
ionregardingtothedatagrid.
The metadata includes file instances, the contents of file ins
tances, and variety a ostorage f systems
in the data grid. The goal of metadata service ito sfacilita
te aenfficient means of naming, creating
andretrievingthemetadataofdatagrid[12].Therearetwotypes
ofmetadata:application
metadataandfabricmetadata.Whiletheformerdefinesthelogic
alstructureotfheinformation
representedbydatafiles(e.g.XML[9]),thelattercorres
pondstotheinformationrelatedtothe
characteristicsof the data grid. Sincethedatagridconsistsoalarge f numberosftorageresourc
esgeographicallydispersed
overalarge-scaledistributedenvironment,anessentialrequirement capability obeing f scalable, meaning that the performance
ofthedatagridisthe
can bseustained acatertain level while
supportingalargenumberodf ataentities.Wahleat l.haveproposed
aso-calledLightweight
DirectoryAccessProtocol(LDAP)[40],whichachievesahighsca
labilitybyapplyinga
hierarchicalnamingstructurealongwithrichdatamodels.The operateattribute-basedsearchingandwritingofdataelements developed distributed a directory service for general grid metada
LDAPprotocolisdesignedto [40].Fitzgeraldetal.have ta, which itnurn can baepplied to
representthe metadata odata f grid. Barueal. tproposed metadata a catalog(MCAT)for satorage
resourcebroker [3].MCAT
onlydistinguishedlogicalnamespacefromphysicalnamespace, specificationolafogicalcollectionhierarchy,whichachieved 4
butalsoprovidedthe ha ighscalability.Inaddition,the
not
performance wasimproved inMCAT by introducing containersto aggrega
te smallfiles.
Chervenakeal. trecently developed prototype a Metadata Service (MC the metadata iclassified s into two categories: logical fi Alogicalfilenameinthisprototype
S) for data grids, where
le metadata and physical file metadata [13].
representsauniqueidentifierfordatacontent,whereasa
physicalfilenamedenotesanidentifierforpa hysicalcontent
onstoragesystem.UnlikeMCAT
mentionedearlier,theservicesinMCSarededicatedtodescribing
files,andthephysicalfile
propertiesarenotcoveredbyMCS.WhileMCATutilizespropriet a
arydataexchangeformatto
communicatewiththestorageresourcebrokerserver,MCSemploys
XMLinterface[9]to
communicate withthe clients. 2.2DataAccessServicesfor DatagridSystems Datastoredandretrievedbyadatagridsystemmayreside,
bydesign,indifferentsiteson
different storage devices. Moreover, it is an essential requirem
ent that end users be not necessarily
aware of the physical locations of accessed data. Consequently,
the data grid system has to provide
galobalview of data for applicationsrunning otnhe grid[12]. Data access services, defined bsytorage API and command lines
are , required tboceompatible
withtraditionalfilesystems.Indoingso,applicationsthatare
notoriginallydesignedforgrid
environmentcanrunonagridsystemwithoutcomplicatedmodifications. systems couple with desire a teonrich the interface odata f a
Inaddition,grid
ccess service teoxploit full benefits of
the gridsystems. Beforeproceedingtodiscussingthedataaccessservicesi
nthegridenvironment,webriefly
review twowell-knowndistributed file systems.The transfer s
ize oeach f data accessisrestricted
inNFS[27]duetothefindingthatstorageaccessperformancein
thewideareanetworkgreatly
dependsontransfersize.Whileaccessingdatathatmightbe
merely portion a ofafile,AFS[18]
transfersandcopiesthe entire file,causing unnecessary I/O tr
affic. 5
Globus, serving aan sinfrastructure for grid computing, provides acces
to sremote files through
x-gass, ftp oHTTP r protocols [17]. White eal. thave proposed the Le
gion I/O model that provides
raemoteaccesscapability [41].Legionachieves55-65% offtp’s
write bandwidth and 70-85%of
ftp’sreadbandwidthformasstransfers.TheperformanceoL f egi overhead ithe f transfer sizes are less than 1Mbyte. To allow e
onsuffersduetotheprotocol nd users to exploit context space and
manipulate context structure,Legion also offers some command line ut those iUnix an File System. For example, legion_cp copies files
ilities, which are similar to within context space obetween r a
nativefilesystemandcontextspace.Furthermore,Legionallows
applicationstoaccess
BasicFileObjectsby meansof raemote I/O interface. PattenandHawickproposed Distributed a ActiveResourceArchitect
ure(DARC)thatenables
thedevelopmentofportable,extensible,andadaptivestorageservice
os ptimizedforendusers’
requirements[23].ToevaluatetheeffectivenessoDARC, f daist
ributedfilesystemusing DARC
wasdeveloped[23].Thefilesystem,calledStrata[23],hasaloc hierarchy.InStrata, saitewilling toffer satorageresour In case where client a is going to
ation-transparentdirectory cehastousethemechanisminDARC.
access raemote data service, its site calls for laoca
of the specified data resource, which itnurn communicates with
instantiation l
remote storage services. The data
accessservicesimplementedinStrataincludelookup,create,mkdir
r,ead,write,andreaddir
operations. 2.3Performance Measurement inDataGrid Asmentionedearlier,m a onitorcanbde evelopedtoidentifycommunicat
ionbottlenecksina
grid. In general, performance a monitor plays an important role in t
he grid environment to measure
theperformanceodiverse f resources,suchastoragesystem,n
etwork,andcomputers,therefore
identifying performance problems in different resources. With the can analyze the monitoring data from different point of views. Since 6
monitoring service in place, one multiple resources may affect
oneanotheracrosstimeandspace,performanceproblems(e.g.low
throughputorhighlatency)
mightbeimposedbcombination ay oresources. f Thus,itisimportant
andnon-trivaltodiagnose
performanceproblemsinaccordancewiththeperformancemeasureme
ntsprovidedbythe
monitor. The objective othe f monitoring service ito sobserve performance
bottlenecks, which are likely
tooccurinanycomponentsodf atagridsystems.Previousstudyhas
shownthatmonitoringat
application level is an effective approach for both performance anal
ysis and application debugging
[8][38].Forexample,Snodgrassincorporatedrelationaldatabasesi
ntoamonitoringsystemto
monitor complexsystems[28]. GMA(GridMonitoringArchitecture)proposedbyaworkinggroupdesc
ribesthekey
componentsoagrid f monitoringsystemalongwithsomeessentialint modelusedintheNetworkWeatherService[33]wasrecentlyext
eractions[37].Thedata endedbyLeeeal. ttomeetthe
needs of monitoring data archives in grid environments [20]. More specifi raelational monitoring data archive that was designed teoffici
cally, Lee et al. proposed ently handle high volume streams of
monitoringdata[20].Ahigh-energyphysicsgridprojectreveals gridcanboeptimizedbymeansoanalyzing f thedetailedinstrum
thatthe
datatransfersindata a
entationdatamaintainedby the
monitoring service associated withthe data grid[45].
3. High Performance DataGrid Inthissection,wereviewtheissuesodata f replication,scheduli
ng anddatamovement,which
mustbe addressedinorder toachieve highperformance and scalability
in data a grid.
3.1DataReplication Datareplicationios neothe f keytechniquestoachievebetter
accesstimesandreliabilityin
datagridsystems.Alargebodyofworkhasbeenfoundondatarepli applications[10][31][34].Ingeneral,datareplicationcanbeutili 7
cationfordistributed zedtoimprovedatalocality,
increasedependability,reduceaccessdelay,andenhancescalabili
tyfordistributedsystemsin
wideareacomputingenvironments.Theapproachesoriginallyproposed arealsoapplicabletodatagridsystems,sincedatagridscompr data sites. The existing techniques for
fordistributedsystems iselarge a numberofdistributed
data replication idnata grids can bdivided e into four camps,
namely,(1) Architecture and management for data replication, (2) D
ata replication placement, (3)
Data replicationselection,and (4) Data consistency. • Architecture oadf atareplicationservice Before discussing data grid replication architecture, it is nec
essary troeview the following three
featuresodf atagrid.First,dataids istributedandreplicatedto
geographicallydistributedsites,
implying that data consistency has to bm e aintained
under wide-area environments where network
latencies are generally long [22]. Second, data dnoot change och r
ange infrequently. For example,
data im n any scientific collaboration systems arenot published to
the community until it hasbeen
prepared[14].
Althoughalargenumberofdata-intensiveapplicationssharethisfeat
ure,the
feature does not necessarily apply teovery data-intensive applic
ation ignrid environments. Third,
areplicationserviceigs enerallyimplementedontopoadf atabas
emanagementsystemordata
store,therebyallowingvariousstoragetechniquesappliedtodiffer
entsites.Inotherwords,data
gridstrivestobuilduniform a datareplicationservicethatis
applicabletobroad a rangeodata f
stores.Therefore,datareplicationisimplementedasamiddlewa
recomponentbetweengrid
applicationsandthe physicaldata storage. Ageneralhigh-levelarchitecturewithfourlayersfordata
replicationserviceinadatagrid
systemisdepictedinFig. [112].We describe eachlayer asf
ollows:
(1)Filetransferlayerconsistsofiletransferprotocols
suchas:FTP,HTTP,andGridFTP[2].
Note thatGridFTPian sextendedversionoFTP. f (2) The responsibility othe f Replica catalogues layer is to
map logical file names to physical files. 8
For example, Replica Catalog iG n lobus Data Grid tools [2] is capa copiesof laogicalfile bfyacilitating a
ble of keeping track ophysical f
map fromlogicalfile namesto physicallocations.
4.Replica Selection 3.Replica Manager Replica 2. Catalogue File T1.ransfer FigureArchitecture 1. ofadata ReplicationServiceinadatagridsystem (3) The replica manager layer’s functionalities include create,
move, modify and delete replicas in
thedatagrid.Thereplicacatalogfunctions,suchacsreatingandd
eletinglogicalfilenames,can
both bienvoked bgyridapplications and operations issued ireplica an mana
gement layer. Once a
fileics opiedfromonesitetoanotherbythereplicamangerl
ayerandregisteredinthereplica
catalogue,itcanbreetrieved bayny applicationrunning oanrbitrary
site itnhe grid.
(4)Replicaselectionlayerisresponsibleforchoosingthemost
appropriatereplicafrom
copiesgeographicallydispersedacrossthegrid.Theselectionsorf
those
eplicationpoliciesdependon
readandwriteaccesstodata,whichcanbcelassifiedintothet
wocategories.(1)Read-only data,
(2) writable data withwell-definedfile ownership,
and (3) writable data with varying writers.
• DataReplicationManagement We first take look a areplication t policies designed for read
-only data. A pilot project called the
GridDataManagementPilot(GDMP)hasbeenlaunchedtodevelopaprotot managementwithin data a gridenvironment[26].IntheGDMPsoftware, database files has been used, and high-level a replica catalog
ypeforreplication saecurereplicationof interface iimplemented s buysing the
native objectivity federation catalogue. Consequently, the GDMP sof Objectivity file. However, replicate files of more flexible
tware is restricted troeplicate data types can bseupported bryeplacing
9
the objectivity file catalogue with other catalogues, such aGlobus s
Replica Catalogue [17].
The replication policy inGDMP iasynchronous s with subscription a m
odel. Before describing
thereplicationpolicyimplementedinGDMP,someimportantdefinitions follows. A producer is saite where one omore r files are written, and a going tolocally replicatethese files.The withtheir relatedinformation,whereasthe have not been updated athe t consumer
areintroducedas consumer is the
site that is
exportcatalogue is laistof allnewly created filesalong importcatalogue is list a ofnewly publishedfilesthat site. It is the producer’s responsibility tdoetermine when to
write new filesintothe exportcatalogue andtherefore making the
mavailable for itsconsumers.
Thereplicationcanbeimplementedatdifferentlevelsogf ranular
ity,suchasfileleveland
objectlevel[30].Realizingthatfilereplicationinadataa
nalysisapplicationispotentially
inefficient,Stockingereat l.hasimplementedobjectreplicat
ioninadatagridprototypeusing
GlobusData Gridtools[30]. Foster et al. recently developed general a framework, which defi
nes parameterized a set of basic
mechanismtocreateavarietyofreplicalocationservices[
14].Byconfiguringthesystem
parametersdynamically,oneisabletotunesystemperforman
cebymeansocf onsideringthe
tradeoff among reliability andcommunicationoverhead, and storage update/
accesscosts.
• DataReplicationPlacement In recent a study [24],severalreplicationselectionstrategie beenproposedandevaluated.Comparedtostaticreplication,dynamicr addsanddeletesreplicasinaccordancewithchangingworkload,ther
fsormanaginglargedatasethave eplicationautomatically ebysustainingahigh
performance evenwhendata accesspatternchanges[24]. • DataReplicationSelection Once data replicas are generated and stored amultiple t site
according s tvariety ao oreplication f
placement strategies discussed above, the performance odata f grid 10
can bneoticeably improved by
choosing optimizedreplica location for eachdata access. Replica selection ihasigh level service that helps grid applica
tions to choose replica a based on
systemperformanceanddataaccessfeatures.Vazhkudaietal
.[39]havedesignedand
implementedareplicaselectionservicethatusesinformationwi
threspecttoreplicalocationas
wellasapplication’sspecific requirementstodetermine repl a
ica among allreplica alternatives.
• DataConsistencyinDataGridSystems Thedatareplicationtechniquesindatagridsystemsreviewedabove
mainlyaimatread-only
files,therebyavoidingdataconsistencyproblems.Ingeneral,l
ocalconsistencyateachsiteis
guaranteedbydatabasemanagementsystems(DBMS),whereasgl
obalconsistencyindatagrid
systemsneedstobemaintainedbyaso-calledgridconsistencys
ervice.Todesignapractical
consistency service,one hastotake intoaccountthe following two dynam (1)File and replica catalogues are updated once filesare cre
ic factors[22]: ated, deleted omoved. r There are two
typesorfeplicacatalogues:centralreplicacataloguea
nddistributedcatalogue.Asforacentral
replicacatalogue, daatabasemanagementsystemcanbaepplie information,aswellasupdatethefileinformation.Incasewhere distributed, the replica catalogue itself needsto bseynchronize
dtostorelogicalandphysicalfile thereplicacatalogueis d. For example, creating new a file
meansitsfile name hastobuepdatedianllcataloguereplica
s. Thissynchronization ireferred s to
asmeta-datasynchronizationsincereplicacatalogueim s et
a-informationusedtomanageactual
data. The consistency model proposed bD y üllmann eal. tassumes that
meta-data synchronization
ishandledbytheunderlyingreplicacatalogueservice,andthisass
umptionnarrowsthefocus
downtothe consistency issue oactual f data [22]. (2)Updateofilecontents.Whenreplicateddataisupdatedatonesit propagatedtoandbecome visible taollother replicaswithin speci a
e,itschangesshouldbe fic mount a of time.
Someworkhasbeendoneondataconsistencyindistributedenvironments 11
[32][42].For
example,Sunetal.proposedanovelconsistencymodelwiththreeproper
ties,namely,
convergence,causality-preservation,andintention-preservation[32].Them consistencymodelisreal-timecooperativeeditingsystemsthat
aintargetofthis
allowmulti-user,physically
dispersedpeople,toview andedit sahareddocumentatthe same time
over networks.
Toaddresstheissueofgriddataconsistencymaintenance,Düllmann consistency service [22], whichisupported s beyxisting datagrid s catalogues [12] mentioned earlier (See Fig.1). This service hel
etal.proposeda ervicesthatinclude the replica ps grid applications in updating data
andpropagating the updateddata tother sitesthathave subscribed ttohe
consistency service.
Wenowsummarizefivepossibleconsistencylevelsdiscussedin[22 toleratespossiblyinconsistentcopiesirseferredtoacsonsist
].(1)Thelevelthat encylevel
unabletoguaranteethatafilereplicacopiedfromonesiteto
–1.Atthislevel,oneis
anotheritshelatestversion.(2)
Consistent level g0uarantees that the newly copied replica is
snapshot a of the original file at some
point. (3) At consistency level 1, each replica has been created whe
nw o rite transactions issue on
itsoriginalfile,therebymakingotherclientsusethereplica
withoutanyinternalconsistency
problem.Consistencylevel1istheminimalconsistencyofferedb systemusingtheconceptoftransactions,(4)Sincereplicaandit
yadatabasemanagement os riginaldataarenotpartofa
commondatabasesystem,itisnot saurprisewhentwocopiesaredi
vergent.Atconsistency level
2,itisassumedthateachsitehasanindependentdatastoreandi
us nawareotfhedatastoreof
replicatedsites.Thus,thereplicasofafilearestoredandma
nagedbythelocaldatabase
managementsystems.(5)Consistencylevel3isviewedasthes
trictestlevel,wherecomplex
locking and recovery procedures are involved. In grid a environment, this le
vel of consistency can
beimplementedifalldataaccessoperationsshareacommoninte
rface.Inotherwords,a
distributeddatabasemanagementsystemcanbedirectlyappliedto
thegridenvironmentto
guarantee the dataconsistency athis t strictlevel. However, it
isnotfeasible for mostapplications 12
inthegrid,sinceitincurssevereperformancepenaltieswhendat
avolumeaccessedbythe
applicationgrowsuptoseveralPetabytes. Tomaintainthebasicconsistencylevel,areplicationoperationmus database read transaction [22]. Thus, read a lock ofile an
bt emanipulatedasa
has to bobtained e before site a iready s to
produceanewreplicaofthefile.Thesitereleasestherea
dlockwhenthereplicahasbeen
successfullyduplicated.Likewise,theconsistencyservicerests operationsare issued
onwritelockswhentheupdate
on set a of replicas.
Alldata replicationtechniquesdiscussedabove are summarized iT na
ble I.
TableI.SummaryoData f ReplicationTechniques Researchers Samar andStockinger, [26]
Granularity File
DateType ReadOnly
Consistency No.Producer decides when topublish new files No
Vazhkudai,Tuecke,et File ReadOnly al.[39] Stokinger etal.[30] File/Object ReadOnly No.Con
Chervenak,Deelman, File ReadOnly andFoster [14] Ranganathan and File ReadOnly Foster [24] Lamehamedietal.[19] File ReadOnly Mullmann,etal.[22] File ReadandWrite Sun eal. t[40] String ReadandWrite
sumer decides when to replicateobjects Relaxedconsistency for metadata No No
Yes Yes
Technique Subscription model,partial replication model Replication selection Object replication
Replication location service Replication placement
Replication placement Consi stency service ThreeNovelproperties
3.2SchedulinginDatagridSystems Duetothelargenumberouf sers,datasize,anddistanceinvolved,sch challengingissuetomakedatagridscalableaswellasef
eduling becomesa ficientforlargenumberofjobs.
Therefore,itisinteresting todiscussthe relatedwork regardi
ng to scheduling idnata grid.
Previousresearchhasshownthatschedulingiasfundamentaltechniquef performanceforgridapplications[6][29][1][34].However,thesea
orachievinggood pproachesdonottakeinto
accountthe distributeddata constraints. Schedulingschemesdesignedfordata-intensivejobshavegenerallybe 13
enclassifiedinto
two
majorcategories,namely,movingdatatojobandmovingjobtodata.Ingr
idenvironments,both
approachesneedtoconsiderdatalocationwhendeterminingjobplacem
ent.Toreducethe
frequencyorfemotedataaccessandtheamountoftransferreddata
, datareplicationcanbean
optimizationtechnique awe s discussed inSection3.1. Thain eal. tdeveloped system a where execution sites band together
into I/O communities [35].
EachI/OcommunitycomprisesseveralCPUsandstoragedevicest
hatprovidestoringand
retrievingdataservicesbothforlocalandremotejobs.Forexampl
e,adistributedfilesystem
sharedamongagroupofmembersisviewedasanI/Ocommunity.To constraintsonstoragedeviceswithinI/Ocommunities,Thain
enablejobstodeclare etal.extendedClassAdsthatare
currentlyusedinCondorsystem[21]todescribejobs’requirementsi singleClassAdialist s of(attribute,value)pairs,whereval
nadistributedsystem.A uecanbeitheratomicocomplex r
expressions.Figure 2illustratesthree differentkindsof ClassA
[35]. d
Type = “job” Type = “storage” ype = “machine” TargetType TargetType = “machine” Name = “machine”= “job” Name = “raven” Cmd= “sim.exe” HasCMSData = true Owner = “thain” CMSDataPath = “/cmsdata = “slinux” Requirements= rements= (Owner = “thain”) && Onwer = “thain”) NearestStorage.HasCMSData NearestStorage = (Name = “turkey”) && (Type = “storage Figure2ExamplesofjobClassAd,Machine
”
”)
ClassAd,andStorageClassAd
CondormatchesajobClassAdsinputwithbothmachines’andstorages’C storedinthesystem.Once match a ifsound,thejobwillbeexecut
lassAdsthatare edby themachinelinkedwith
the job’sspecific storage devices. Tominimizeremotedataaccessoverhead,Basneyeal. t haveinvent
edaso-calledexecution
domainframeworktodefineanaffinitybetweenCPUanddataresource
sinthegrid[4].By
applyingtheframeworktotheCondorsystem,data-intensivejobscan
bescheduledtorunon
CPUsthathave accesstorequireddata. 14
To gain aenfficient execution oparameter f sweep applications on
the grid, Casanova et al. have
studiedfouradaptiveschedulingalgorithmsthatconsiderdistributeddat
astorage[11].The
proposed algorithms are heuristic,yet effective iennvironments whe
re some computing nodes are
bestforsometasksbutnotforothers.Thekeyideabehindtheseschedul
ingalgorithmsisto
judiciouslyplacefilesformaximumreuse.Theseschedulingalg
orithmshavebeenimplemented
in user-level a gridmiddleware,whichcentrally handlestask sche
duling.
Comparedwithcentralizedschedulingframework, daistributedscheduli be more scalable and efficient for the grid storage that provides
ngapproachseemsto
grid I/O service for hauge number
of data-intensive jobs. RanganathanandFosterhaveproposedadistributedschedulingframeworkw consistsoaf nexternalscheduler,alocalscheduler,andadatase schedulerofsaiteirsesponsiblefordispatchingthesubmittedjobs
hereeachsite scheduler t [25].Theexternal toanappropriatesite,which
caneitherbeonewiththeleastloadortheonewhererequireddata functionality othe f local scheduler is to decide the order in whic
havebeenstaged.The jhobs are executed athe t site. The
datasetscheduler makesdecisionson whenandwhere to replicate data
.
3.3DataMovement Incomputational a grid,datamovementmechanismplaysanimportant systemperformance.Thisids rivenbytworequirementsimposedby datagriditself.First,applicationsneedtoaccessdatathat
roleinachievinghigh bothgridapplicationsand isnotlocatedatthesitewherethe
computationips erformed.Second,replicationservicesareresponsible
formanagingreplicasby
meansof copying andmoving data throughnetworks. Tofulfillthe above tworequirements, Bester etal.have proposed a service calledGlobal Access to Secondary Storage (GASS) [7]. data movement strategies that are geared for the common I/O pa
data movementand access The service not only incorporates tterns of grid applications, but also
15
supportsprogrammermanagementodf atamovement.Theperformanceeval
uationshowsthat
GASSian sefficientapproach todata movementingridenvironments. Thainetal.havedevelopedadatamovementsystemcalledKanga reliabilityaswellasthroughputofgridapplicationsbyhidingne
roo,whichimproves tworkstoragedevicesbehind
memoryanddiskbuffer[36].Morespecifically,theapproachrelie
sonKangaroo’sabilityto
overlap CPU and I/O processing intervals by using background process
es to move data and handle
errors.
4. SecurityIssue Withtheusageodifferent f machinesiGrid, na theissueosec f
urity,whichhasbeenprovided
bymostexistingdatagrids[12][26][41],becomesoneoftheprime traditionalsecuritytechniques,suchasencryptionandaccesscontrol applied tgrid. ao For example, uaser has to baeuthenticated and author
concerns.Inprinciple, c, anbeconservatively ized before contacting any
remote site. Duetothespacelimitation,inwhatfollows,weonly reviewtwoint
erestingissuesinsecurity
addressedinGDMP[26].SamarandStockingerhavestudiedthemainsec sensitivity odata f andunauthorized use onetwork f
urityissuesof
bandwidth to transfer hugefiles. Specifically,
thesecuritymoduleoG f DMPserver,basedontheGlobusSecurity functionstoacquirecredentials,initiatecontextestablishmenton requestsontheserverside,encryptinganddecryptingmessagesandc Consequently, the security module protects saite from any undesire
Service(GSS)API,offers clientsideandacceptcontext lientauthorization. fdile transfers and rejects any
unauthorizedrequests.
5. Open Issues Although variety a otechniques f have been proposed taochieve high performa therearestillsomeopenissuestobaeddressed.Thissectionlis
nce idnata grids, tstheopenissuesrelatedtohigh
16
performance data grid. 5.1ApplicationReplication Althoughsomeeffortshavebeenmadetoreplicateactualdataac little attention has been paid troeplicating frequently used gri
cessedbygridapplications, adpplications across the grid. Once a
schedulingschemedecidestomoveanapplicationtowardsitsdata,a
replicationselectionpolicy
forapplicationscanbedevelopedtochoosethemostappropriateapplicati
oncopyinthegrid.
There are three reasonsthatmotivate uto sdistinguishapplication
code fromactualdata:
(1)Foremost,anexecutablecodeoan fapplicationaone t sitemightnot
beable troun
site.Onecanstraightforwardlytacklethisproblembyusinga
onanother
java-programmingenvironment,
whichbecomesincreasinglypopular[44].However,alargenumberofexi applications have been developed iF nORTRAN, C or other programming lang problem,onecanhavethesourcecodemovedtodestinationsitesandobtainthe
stingscientific uages. To solve this executablecode
by compiling the source code athe t destinationsites. (2)Theapplicationreplicationtechniquesprovideefficientsupporttoap architecture.Forexample,awell-designedpartitionalgorithm effectively divide data among the processors where the processing
arallel-shared-nothing canbeusedtodynamicallyand application have been staged to
achieve maximumparallelism. (3)Itisreasonably assumedthatdatacanbsetoredaany t sit
e,providedthatithasregisteredaas
member of daatagrid.Unfortunately,thisassumption inot s practical sitesmightnotbe able tofulfillapplications’ hardware or/and sof
for applicationssince some tware requirements.
Tofacilitateaparallelcomputingenvironmentincomputational directionscanbestudyinganapplicationreplicaservice,whichincl managementandapplicationselection.Theapplication duplicate,remove,andupdatecopiesofapplicationinstances,aswella 17
grids,oneofthefuture udesapplicationreplica replicamanagementisdesignedto stomaintainthe
consistencyamongthereplicas.Theapplicationselectionservice
isdevisedtochoosean
optimizedlocationwhere anapplicationreplica hasbeen stored troun t
he job.
5.2ConsistencyMaintenance InSection3.1,wehavediscussedtherequirementsocf onsistencymaint scientificdataandmetadata.Webelievethattheproposedapplicat
ionreplicationservicesmake
theconsistencymaintenanceevenmorecomplex.Thisisbecause,in scalability,theservicesarerequiredtoboefferedidis na
enanceimposedby
ordertoobtainhigh tributedfashion.Inaddition, garoupof
programmers are allowed tcoollaborate with one another to develop in grid a environment,andgridapplicationsare likely tobpeeriodical We realize that, from the summary odata f replication techniques consistencymodelsindatagridworkwellforreadonly data,butli
some large-scale applications ly upgraded. given iT n able Imost , existing ttleattentionhasbeenpaidto
the consistency issuesfor data thatneedsto bw e rittenmultiple
times.In data a grid environment,
manydatasetsaremultiple-writteninnature.Whenagroupofprogr
ammerscollaboratively
develop large-scale a scientificapplicationsonthegrid,somepr
ogrammersmightbeworking on
thesamesourcecode,implying thatthecodenotonly needstobe
writtenmultipletimesbutalso
likelytobewrittenconcurrentlybyseveralprogrammers.The
refore,iw t illbeinterestingto
developarelaxedconsistencymodeltoeffectivelyhandlethecon
sistencyproblemsin
multiple-writtendata discussedabove. 5.3AsynchronizedDataMovement Theimpactofdatamovementontheperformanceoflocalsiteis especiallytrueithe f volumeotransferred f dataihs ugeodat r
enormous,andthisis m a ovementsoccurfrequently.To
alleviate such burden a resulting fromdata movements, paotential of moving data withoutsacrificing the performance oapplications f runni protocol can bdeesigned tm o ove asynchronized data if the loads of sour 18
solutionito sstudy new a way ng olnocalsites. Thus, a ce site and destination site
arebelowacertainthreshold.Comparedwiththeexistingdatamov
ementapproachesthattreat
synchronizedandasynchronizedequally,thepotentialsolutionisexpec
tedtoimprovethe
performance odata f gridwhenthe I/O andnetworkloadsateach site
are bursting inature.
5.4PrefecthingSynchronizedData We notice thatdata prefetching schemesfor synchronized data have
caughtlittle attention. We
believethatdataprefetchingtechnique,whichcomplementsthenewdata
movementapproach
presentedinSection5.3,canoptimize the performance odata f grid in
casewhererequired data is
movedtowardstothe remote site where itsjobresides. By using ClassAd language to explicitly define applications’ r
equired files, static a prefectching
schemecanmakethesefilesavailablebeforeapplicationsneedt
oaccessthem.Asoneotfhe
possiblefuturedirections,anewprefetchingalgorithm,whichis predicating files, will be developed teostablish correlations am
capableofdynamically ong files based ostatistic an model.
This new dynamic prefetching algorithm actively provides grid appli
cations with files by shipping
requiredfilesinadvance before applicationsare locally loading the
files.
5.5DataReplication AscanbseeenfromTableIthe , ideaoreplicating f frequently
accesseddataonmultiplesites
hasbeeneagerlypursued.Thedrawbackofthemostexistingreplication replication granularity ifile, s w a hole big file might be repli
techniquesitshat,as cated even siafmall portion othe f file
isfrequentlyaccessed.Thisproblemisanalogoustofalsesharing system.To
indistributedsharedmemory
provideapotentialsolutiontotacklethisproblem,onecaninvestigat
model,wherethereplicationgranularitycanbedynamicallyadjus
esanewdata tedaccordingtothebenefit
gainedfromreplicasandthe overhead ogenerating f replications. Interestingly,feweffortshavebeenmadetodevelopsomecachinga diskcachereplacementpolicies.Therefore,itwillbeinterest
lgorithmswithefficient ingtostudy caching a algorithm,in
19
which files cached olnocal disks are viewed atemporary s replic
as. The main difference between a
cached file and regular a replica is that the file in disk ca
che might be removed when the disk cache
isunable toaccommodate newly arrivedfiles.
6. Conclusions Wehavereviewedanumberodf atagridservicesandvarioustechni performance odata f grid. Additionally, the issue of security idnat
quesforimprovingthe grid a has been briefly discussed.
Five openissuesandthe possible future directionsare summarized as
below.
(1)While significant a amountofdatagridresearchhasbeendonei
ndatareplication,theideaof
replicatingfrequentlyusedgridapplicationsacrossthegridhasr
eceivedlittleattention.One
possiblefutureworkitsodevelop new a applicationreplicaservice,w
hichcanbbeuiltontopof
data replicationservice. (2)Iitsnoticedthatmostconsistencymodelsintheexisting
datagridshavenotaddressed
consistencyissuesformultiple-writtendata.Asoneotfhefuture
directions,onecandevelopa
relaxedconsistencymodeltoeffectivelyhandletheconsistency
problemsforanumberof
commonfile accesspatternsincluding multiple-written pattern. (3)Iw t illbeinterestingtostudyanewwayofmovingdataa
atime t whenbothsourceand
destinationsitesarelightlyloaded,therebyachievingbetterpe
rformanceindatagridwithout
sacrificing the performance oapplications f running on localsites. (4)Afourthinterestingfuturedirectionistodesign
anewprefetchingalgorithmthatcan
dynamically predicate filesthatare likely tboreetrieved. (5)Lastbutnotleast,oneotfhepossiblefuturedirectionsitsos
tudyafile-cachingalgorithm,
where the replication granularity onew af data model can bdy e
namically tuned ianccordance with
the benefitand the overheadogenerating f replicas.
20
Acknowledgement WewouldliketothankDr.ByravRamamurthy,Dr.SteveGoddard,Dr.Da
vidSwanson,and
Dr.XiaoChengZengfor their valuableandinsightfulcomments
andsuggestions. Thiswork was
supportedbyanNSFgrant(EPS-0091900)andaNebraskaUniversityFoundat
iongrant
(26-0511-0019).
References [1]D.Abramson, J.Giddy, I.Foster, and L.Kotler, “High Performance Pa Killer Application for the Global Grid,”
rametric Modeling with Nimrod /G:
Proc. of the International Parallel and Distributed Processing
Symposium May 2000. [2]BAllcock, . J.Bester,J.Bresnahan,A.Chervenak,IFoster, . e
t.al.,“EfficientDataTransportand
Replica Management for High-Performance Data-Intensive Comput onMassStorageSystems
ing,” Proc. 18th IEEE Symposium
2001. ,
[3]CB . aru,RM . oore,AR . ajasekar,andM.Wan,“TheSDSC
StorageResourceBroker,”
Procof
CASCON Conference1998. [4]J.Basney,ML . ivny,andP.Mazzanti,“UtilizingWidelyDistr Efficiently withExecution Domains”,
ibutedComputationalResources
ComputerPhysicsCommunications,
[5]J.Basney,R.Raman,andM.Livny,“High-throughputMonteCarlo,” Conferenceon ParallelProcessing forScientificComputing
Volume140,2001. Proc.ot heNinthSIAM
March , 1999.
[6]F.Berman,RW . olski,SF . igueira,JS . chopf,andG.Shao,“Applicati
on-LevelSchedulingon
Distributed HeterogeneousNetworks,”Proc.ofSupercomputing 1996. [7]J.Bester,I.Foster,C.Kesselman,J.Tedesco,andSTuecke, . “
GASS:ADataMovementandAccess
ServiceforWideAreaComputingSystems,”Proc.of
SixthWorkshoponI/OinParalleland
Distributed Systems May , 51999. , . [8]WBethel, . B.Tierney,J.Lee,D.Gunter,andSLau, . "UsingHigh-
SpeedWANsandNetworkData
CachestoEnableRemoteandDistributedVisualization,"
Proc.oftheIEEESupercomputing
Conference 2002. [9]TB . ray,JP . aoli,andC.Sperberg-McQueen,TheExtensibleMar
kupLanguage(XML)1.0W#C
Recommendation,World WideWebConsortium, Feb.1998. [10]LBreslau, . P.Cao,L,Fan,G.Phillips,andSShenker, . "WebCac
hingandZipf-likeDistributions:
EvidenceandImplications,"ProceedingsofIEEE Infocom,1999. [11]HCasanova, . G.Obertelli,F.Berman,andR.Wolski,“TheAppLeS User-LevelMiddlewarefor theGrid,” [12]AChervenak, .
ParameterSweepTemplate:
Proc. Supercomputing 2000.
I.Foster,C.Kesselman,C.Salisbury,andS.Tuecke,“TheDataG
ArchitecturefortheDistributedManagementandAnalysisof NetworkStorageSymposium, 2000.
21
rid:Towardsan
LargeScientificDatasets,”
Proc.of
[13]A.Chervenak,E.Deelman,C.Kesselman,
L.Pearlman,andGSingh, . “AMetadataCatalogService
forDataIntensiveApplications”,GriPhyNTechnicalReport,Informa
tionScienceInstitute,August,
2002. [14]A.Chervenak,E. Deelman,I. Foster, etal.,“Giggle:A Frame Location Services,”
work for Constructing ScalableReplica
Proc.IEEESupercompuing Conference
2002 ,
[15]AChing, . A.Choudhary,W.Liao,R.Ross,andW.Gropp,"NoncontiguousI/
OthroughPVFS,"
Proceedingsof2002IEEEInternationalConferenceoC n lusterComputing [16] S. Fitzgerald, I. Foster, C. Kessenlman, G. Laszewski, W. S
mith, and STuecke, . “A Directory Service
forConfiguringHigh-performanceDistributedComputations,” PerformanceDistributed Computing
September, , 2002. Proc.6thIEEESymposiumonHigh
pp.365-375, , 1997.
[17]I.FosterandCKesselman, . “Globus:AMetacomputinginfrastruc
turetoolkit,”InternationalJournal
ofSupercomputer Applications, Vol.11, No.2,pp.115-128, 1997. [18] J. Howard, M. Kazar, S. Menees, D. Nichols, M. Satyanarayanan, R. andPerformanceiDistributed a n FileSystem,”
Sidebotham, and MWest, . “Scale ACMTransactionsoComputer f Systems
Vol6, , No.1,
pp.51-81, 1988. [19] H.Lamehamedi,B.Szymanski,Z.shentu,andE.Deelman,“DataReplicat
ionStrategiesinGrid
Environments,” Proc.Othe f FifthInternationalConferenceonAlgorithmsandA
rchitecturesfor
ParallelProcessing (ICA3PP'02), 2002. [20]J.Lee,D.Gunter,M.Stoufer,andBTiemey, . "MonitoringDataArc
hivesforGridEnvironments,"
Proc.Supercomputing 2002. [21]MLitzkow, . M.Livny,
andM.Mutka,“Condor-AHunteroIfdleWorkstations,”Proc.ofthe8th
InternationalConferenceonDistributedComputing Systems, pp.104-111,1988. [22] D. Mullmann, W. Hosckek, J. Jaen-Martinez, and BSegal, . “M Consistency in DataGrid,”
odels for Replica Synchronizationand
Proc. IEEESymp. on HighPerformanceoD n istr.Computing
[23]C.J.Pattenand K.A.Hawick,“FlexibleHigh-performanceAcce ProcoHigh f PerformanceDistributed Computing
ssto DistributedStorageResources,” 2000. ,
[24]K.RanganathanandIFoster, . “IdentifyingDynamicReplica DataGrid,”
2001. ,
tionStrategiesforaHigh-Performance
Proc.InternationalGridComputing Workshop
Denver, , November 2001.
[25]K.RanganathanandIFoster,“DecouplingComputationandDataSc
hedulinginDistributed
Data-IntensiveApplications,” Proc.ofthe11thIEEESymposiumonHighPerformanceDistributed Computing, 2002. [26] A. Samar, H. Stockinger, "Grid Data Management Pilot (GDM
P): A Tool for Wide Area Replication,"
Proc.ofInternationalConferenceoA n pplied Informatics
Feb. , 2001.
[27]R.Sandberg,D.Goldberg,S.Kleiman,D.Walsh,andBLyon, . “Desig SunNetwork Filesystem,”
Proc. ofUSENIX Conference
[28]RS . nodgrass,"ARelationalApproachtoMonitoringComplexSy
nandImplementationothe f Berkeley, , CA,1985. stem,"ACMTransactionson
Computer Systems,Vol.6,No.2, pp.157-196, 1988. [29]SpringandRWolski, . “Applicationlevelscheduling:Geneseque
22
ncelibrarycomparison,”Proccof
ACMInternationalConferenceoS n upercomputing 1998. [30]H.
Stockinger,AS . amar,BA . llock,IF . oster,KH . oltman,andB.Tierney
Replication in DataGrids,”
“, FileandOjbect
Proc. ofIEEESymp. on High PerformanceDistributed Computing
[31]HeinzStockinger,“DistributedDatabaseManagementSyst
2001. ,
emsandtheDataGrid,”
SymposiumonMassStorageSystemsand9thNASAGoddardConferenceonM
18thIEEE
assStorageSystems
andTechnologie s,San Diego,April17-20, 2001. [32] C. Sun, X. Jia, Y. Zhang, Y. Yang, and DChen, . “Achieving convergence
causality-preservation, , and
intention-preservation in real-time cooperative editing systems,” Computer-Human Interaction,
ACM Transactions on
Vol.5, No.1, March, 1998, pp.63-108.
[33]MSwany . andRWolski, . "RepresentingDynamicPerformance
InformationinGridEnvironments
withNetworkWeatherService,"Proc.ofthe2ndIEEEInter
nationalSymposiumonCluster
Computing and theGrid, May 2002. [34]RTewari, . M.Dahlin,H.Vin,andJKay, . "DesignConsideration
fsorDistributedCachingonthe
Internet," Proc.ofIEEEInternationalConferenceoD n istributed ComputingSystems [35]DThain, . J.Bent,A.Arpaci-Dusseau,R.Arpaci-Dusseau,andMLiv . Creating Communitiesfor Grid I/O,”
1999. , ny,“Gatheringathe t Well:
Proc. Supercomputing 2001. ,
[36] D. Thain, J. Basney, S. Son, and MLivny, . “The Kangaroo Appr
oach tD o ata Movement on the Grid,”
Proc. ofthe10th IEEESymposium on High PerformanceDistributed Computing,2001. [37]B.Tierney,R.Aydt,D. Gunter, W. Smith, V. Taylor, R.Wolski,and
M.Swany,"A GridMonitoring
ServiceArhitecture,"GlobalGrid Forum WhilePaper. [38]B.Tierney,W.Johnston,B.Crowley,G.Hoo,C.Brooks,andD.Gunter
,"TheNetLogger
MethodologyforHIghPerformanceDistributedSystemsPerformance
Analysis,"Proc.ofIEEEHigh
PerformanceDistributed Computing,July,1998. [39]SVazhkudai, . S.Tuecke,andI.Foster,“ReplicaSelectioninthe
GlobalDataGrid,”
InternationalWorkshop oD n ata Modelsand Databaseson ClustersandtheGrid [40]MW . ahl,TH . owes,andS.Kille,LightweightDirectory
Proc.ofthe
2001. ,
AccessProtocol,RFC2251,Internet
Engineering Task Force, 1997. [41]B.White,A.S.Grimshaw,andANguyen-Tuong, . “Grid-BasedFileAc Proc.ofHigh PerformanceDistributed Computing
cess:TheLegionI/OModel,”
2000. ,
[42]HYu . andAVahdat, . "DesignandEvaluationoaC f onit-basedContinuou Replicated Services. "
C s onsistencyModelfor
ACMTransactionsonComputerSystems(TOCS)
August , 2002.
[43]Y.Zhu,H.Jiang,X.Qin,D.Feng,andDSwanson, . "ImprovedReadPer CostEfficient,Fault-TolerantParallelVirtualFileSys
formanceinCEFT-PVFS: tem,"inproceedingof
Workshopon ParallelI/O in ClusterComputing and ComputationalGrids [44]J.Al-Jaroodi,N.Mohamed,H.Jiang,andDSwanson, . "AnAgent-Based Java oH n eterogeneous Clusters",
IEEE/ACMCCGRID May , 2003,Japan InfrastructureforParallel
In Proc. of 4th IEEE International Conference oC n luster Computing
(CLUSTER2002),Chicago,Illinois, September 2002. [45]GriPhyN Project:http://www.griphyn.org
23
