File-Access Patterns of Data-Intensive Workflow Applications and ...

File-Access Patterns of Data-Intensive Workflow Applications and their Implications to Distributed Filesystems Takeshi Shibata

SungJun Choi

Kenjiro Taura

University of Tokyo Department of Information and Communication Engineering Graduate School of Information Science and Technology



[email protected]

[email protected]

[email protected]

ABSTRACT This paper studies five real-world data intensive workflow applications in the fields of natural language processing, astronomy image analysis, and web data analysis. Data intensive workflows are increasingly becoming important applications for cluster and Grid environments. They open new challenges to various components of workflow execution environments including job dispatchers, schedulers, file systems, and file staging tools. The keys to achieving high performance are efficient data sharing among executing hosts and locality-aware scheduling that reduces the amount of data transfer. While much work has been done on scheduling workflows, many of them use synthetic or random workload. As such, their impacts on real workloads are largely unknown. Understanding characteristics of real-world workflow applications is a required step to promote research in this area. To this end, we analyse real-world workflow applications focusing on their file access patterns and summarize their implications to schedulers and file system/staging designs.

Keywords workflow applications, distributed filesystems

1. INTRODUCTION Workflow facilitates integration of individually developed executables, making parallel processing readily accessible to domain experts. Thus it has become an important discipline in various fields including natural science, engineering, and information processing. Many systems for executing workflows have been developed [1, 2, 3]. More recently,

programming paradigms and systems specifically designed for large data processing such as MapReduce [4], Hadoop1 , and Dryad [5]2 made it popular to utilize parallel processing without an involved effort of parallel programming. An obvious common goal of these systems is efficient execution of workflows. To this end, there have been efforts on various components of workflow engines including scheduling algorithms [6, 7, 8, 9, 10, 11, 12], data transfers [13, 14], and fast dispatchers [15]. There are efforts focusing on scheduling with data transfer costs taken into account [16, 17, 18, 19] A good survey on scheduling algorithm is in [20]. Despite their importance, practical evaluation of workflow systems have been rare and remain difficult, mainly due to lack of commonly accessible benchmarks. Even with a real application, translating the result on a particular platform into a generally acceptable observation on workflow systems is difficult because performance of a workflow depends on so many components of the environment such as nodes, networks, file systems, and so on. This is particularly so because workflows typically consist of many sequential executables each of which may have unknown sensitivities to their environments. Most existing studies on scheduling algorithms therefore have been based on simulation with synthetic workloads such as randomly generated task graphs. Bharathi et al. [21] is one of the few studies that systematically characterizes several real workflow applications. The present paper shares the same spirit as theirs, but pays a special attention to IO (file access) behaviors of applications. A single workflow generally consists of a set of tasks each of which may communicate with (i.e. depends on or is depended upon) another task in the workflow. Since a task is typically a sequential (single node) application, a data transfer among tasks is generally handled by the workflow system. Data may be implicitly transferred via a shared file system or explicitly moved by a staging subsystem. Either case, they generally go through a secondary storage to ensure a certain degree of fault tolerance—that a workflow 1 2

http://hadoop.apache.org/ http://research.microsoft.com/en-us/projects/Dryad/

can restart from a point where a job failed. As such, understanding IO behaviours of real workflow applications is very important for implementing efficient workflow systems. It is particularly so for data-intensive workflows, which are becoming major targets of workflow systems. To this end, this paper shows the result of our study on five real workflow applications in the fields of natural language processing, web data analysis, and astronomy image analysis. With this study, we like to provide data points by which we can investigate important design questions for implementing workflow and related subsystems such as file systems. Relevant questions include: • How effective is it to place jobs according to file affinity, and what is the tradeoff between the complexity and benefit of scheduling algorithms? • What are advantages and disadvantages of using shared file systems (such nas NFS[22], Lustre3 , GPFS [23]4 , and Ceph[24]) over using explicit staging? • How much we can gain from staging systems that transfer files directly between compute hosts rather than indirectly through a central home node? • Likewise, how much we can gain from user-level file systems such as Gfarm [25] or GMount [26] that can use any compute host as a storage server?

2. RELATED WORK 2.1 Workflow Studies Bharathi et al. [21] is the closest to the present work. It studies five real workflow applications. It analyses dependency graphs of each workflow and identifies common communication and dependency patterns such as pipeline, data distribution, aggregation, and redistribution. It also shows statistics on job granularities. In addition to dependency patterns, we also study file access behaviors and discuss their implications to file system or staging designs.

2.2 Executable Workflows Montage [27] is one of the few workflow applications that can be readily downloaded and executed by others. Both Bharathi et al. [21] and our work include this application as a target of their studies. We will provide four more executable applications with annotations that can be shared by researchers in this community.

2.3 Workflow Collections For publicly accessible collection of workflows, Bharathi et al. also provides WorkflowGenerator web site5 , which generates DAGs whose nodes are tasks annotated with runtime and files used. Files are annotated with their sizes, so dependencies and communication volumes can be reconstructed from the DAGs. We are preparing a web site that provides similar data for applications studied in this paper. In addition, we will provide more comprehensive traces of file access behaviors, as well as their executables.

myExperiment6 is a repository of workflow descriptions, currently providing more than 750 workflows. Each workflow is given as a description for a particular system (mostly for Taverna [3]), and not annotated with profile information such as runtimes and file sizes. The number of workflows we are going to provide is much smaller, but we are trying to provide them in a form executable by others.

2.4

Workflow Scheduling Techniques and Distributed File Systems

There have been efforts on various components of workflow engines including scheduling algorithms [6, 7, 8, 9, 10, 11, 12]. However, most of those methods require the complete form of DAGs for workflows and completion time, input/output file size for each jobs, etc. It is difficult to apply scheduling techniques such as HEFT without those information. Especially, The complete form of DAGs requires workflow descriptions to have explicit declarations of input/output files for each job. Because the workflow descriptions we examined in this paper has no explicit form of input/output files, some systems which help dispatched jobs find required files and register output files is required. Distributed file systems are able to be used for that purpose.

3.

WORKFLOW EXECUTION AND FILE ACCESS LOGGING 3.1 Workflow Engine Workflows described in this paper are described in Makefiles and a parallel and distributed make engine called GXP Make7 was used to execute them. Make is a convenient tool for describing jobs with dependencies and can naturally be used for describing executing workflows. Except for Montage, all workflows are written in Makefile by their original authors. For Montage, one of the tools for it generates an abstract workflow in an XML format called DAX, used Pegasus. Output DAX files are converted to Makefiles. Every job description by the tag hjobi is converted to a ”rule” of makefile; commands of a rule are extracted from hjobi tags, whereas dependency (prerequisite) and target filenames from husesi tags with arguments ’link = ”input”’ and ’link = ”output”’, respectively. GXP Make runs on top of an existing Make implementation GNU Make, which supports parallel execution for a single multiprocessor node. GXP Make extends this function to parallel execution in distributed environments by intercepting its child process invocations. Processes forked by the underlying GNU make enter a queue managed by GXP make; It then dispatches jobs to available workers that have been acquired by any underlying resource manager such as TORQUE, SGE, SSH, etc. Input and output files of tasks are stored in a single network file system, such as NFS, Lustre, GFarm, etc.

3

The choice of this particular workflow system is largely orthogonal to the results presented in this paper. It is not difficult to automatically generate descriptions for other work-

4

6

http://www.sun.com/software/products/lustre/ http://www-03.ibm.com/systems/clusters/software/gpfs/ 5 http://vtcpc.isi.edu/pegasus/index.php/WorkflowGenerator

7

http://www.myexperiment.org/ http://www.logos.ic.i.u-tokyo.ac.jp/gxp/

2. The distance of dispatched command lines from normal commands are measured by edit distance. Each dispatched command line is assigned to a category which is represented by the nearest normal command.

flow system such as DAGMan, given file access logging methods presented in the next section.

3.2 File Access Logging In data intensive applications, it is important to examine how much data are transferred over the network, when and where these transfers happen, and where bottlenecks are. When data are transferred mainly as files through a distributed file system, the detail of file access is required to extract such information. In order to record file access operations, we developed a logging tool which is specialized for GXP Make using a tool called Filesystem in Userspace (FUSE). Table 1 shows the recorded data for each access.

For instance, if the following block is in a makefile : DIR := s o m e d i r $ (DIR)/%. i n p u t : $ (DIR)/%. output cmd $< > $@ then the corresponding normal command is cmd s o m e d i r /%. i n p u t > s o m e d i r /%. output .

Table 1: job id operation data id date time size

logged data par a file access operation hostname:processId read, write, open, create, getattr, mkdir, etc. filename execution time of each operation in nsec. from the epoc blocking time during each operation size for each read and write

A file system using our logging tool provides a mount point under which the target directory is mirrored. When a process accesses a file or a directory under the mount point, one of our registered callback routines is called by FUSE, in which the access is logged along with the client process id.

Although another FUSE-based tool ParaTrac [28] offers similar results to ours, our tool is implemented independently for different purpose. While ParaTrac is a general purposes profiling tool, this profiling tool is specialized for GXP Make workflow system and has several different functions from ParaTrac. For example, while our tool can not produce fine-grade analysis such as statistics for time when ’read’ operations occur in processes, it is able to divide jobs into categories for given makefiles and show their statistics, extract user cpu time of each job and show critical path of the workflow, produce workflow DAGs in HTML files with information of workflow execution, and so on.

4. We need to translate the process id to the identity of a job (note that a job may consist of multiple processes). GXP Make gives a unique number to each job through an environment variable (GXP_WORK_IDX). When our logging tool receives a callback, it records GXP_WORK_IDX of the client process if it is the first callback from that process.

WORKFLOW APPLICATIONS

In this section, details and profiling result for the five workflow applications 8 are given. They are: • Medline to Medie Indexing • Case Frames Construction

When the file system is unmounted after logging all file access of a workflow, access logs by all hosts are collected to form a database. Dependencies among dispatched jobs and accessed files are inferred from those files. Dependencies are inferred as follows. (1) If a job X accessed a file A with any file operation, X is related to A. (2) If X created A, then A depends on X (X → A), otherwise X depends on A (A → X). After the database is created, the workflow DAG is generated as a clickable HTML file with information for the workflow execution. Statistics for file-access patterns of each job and each category for jobs are also extracted from the database. A category for jobs is a set of jobs which are derived from the same line in the makefile, or the same command without input file. All dispatched command lines, or jobs, are divided into categories by the following steps.

1. Strings which is called normal commands are extracted from the given makefile by shallow pursing. Each normal command is basically corresponding to single line of the makefile.

• Similar Pages Extraction • Supernovae Explosion Detection • Montage The first two are applications in natural language processing; Similar Pages Extraction is a data mining application applied to a large collection of web pages; The last two are image analysis applications in astronomy. The experimental environment is a single cluster composed of 15 nodes (120 CPU cores) with a single NFS server. They are connected by Gigabit Ethernet with bonding (2Gbps/node). We use GXP make as the workflow engine. Details of file-access patterns are logged and analysed by means of the previous section. Following subsections describe each application with more details. 8

https://www.intrigger.jp/wiki/index.php/Applications

4.1 MEDLINE to MEDIE MEDLINE to MEDIE Indexing (hereafter M2M) workflow creates indexing database for a running search engine called MEDIE.9 It indexes all registered abstracts in the MEDLINE, the world largest biomedical text database. It indexes and retrieves sentences with their semantic subjects, verbs, and objects, enabling queries such as “what does p53 activate?” To this end, its indexing involves deep natural language processing including parsing, requiring a large amount of CPU times and intermediate data storage. M2M takes a compressed XML text files as its input, each containing a number of biomedical papers in MEDLINE. Most tasks in M2M input and output compressed XML files. Figure 1 shows a sub-workflow of M2M for a single input file. A circle is a single task, and a square represents a file created or read by tasks. An arrow from a task to a file indicates that the task created the file, and one from a file to a task that the task opened the file for reading. The most time consuming part is parsing done by a parser called Enju. This is parallelized within each input file by splitting the intermediate file right before the parsing into small chunks. This can be seen as a number of horizontally aligned boxes around the middle of the graph. F258

J43

F259

J20

J42

F152

F257

800000

meta data read write stime utime

700000 600000 500000

]c se[ 400000 e im T 300000 200000 100000 0 1

2

3

4 Ranking

5

6

7

(a) Time detail of each job

utime stime write read metadata

F260

J45

F147

J44

J17

F146

J29

F153

J56

F64

J21

J28

F158

J23

F156

F157

J22

J16

J50

F171

F275

(b) Time detail for whole workflow

F155

F90

F277

J57

F272

F273

J51

F88

J10

J32

J11

F177

F91

F87

F201

F162

F98

F100

F103

F104

F105

F71

F106

F107

F108

F109

F72

F110

F76

F77

F78

F1

J68

J63

J66

J61

J64

J0

J1

F79

F80

F84

F85

F86

F2

F4

J52

F3

F73

J62

F68

F74

J8

F81

F111

F112

F75

F8

F10

F6

J65

J67

J3

J4

J2

F82

F83

F9

F11

F7

J9

F261

F70

J46

F262

Figure 2: MEDLINE to MEDIE :Time detail of each job and of the whole workflow

J12

F113

J39

J24

F149

F161

J18

J25

F164

F54

J35

J58

F60

J33

F166

J34

F58

F140

J26

J5

F282

4. Medie-ner-disamb.: named entity recognition for biomedical terms, such as protein names, and removing ambiguity from them.

J13

F56

F138

J40

F148

J14

F203

F165

J53

F278

F143

J69

J70

F150

J19

F151

F289

J31

F175

J71

F284

F174

J59

J30

F279

F281

J54

5. Medie-ner-filter: preprocess for Medie-ner-disamb.

F168

J55

F280

J36

J60

J37

F263

F169

F268

F195

J47

J27

F145

F264

J15

J49

J48

F267

F266

J72

F144

6. MakeDB : creation index database for MEDIE.

F290

J38

J73

F65

F291

J6

F66

7. Som-merge : merging intermediate XML files.

J7

F67

J41

F230

F224

F255

F254

F221

F252

F229

F248

F239

F223

F242

F240

F228

F220

F218

F256

F231

F227

F217

F233

F235

F251

F246

F222

F236

F232

F234

F237

F243

F241

F247

F244

F212

F253

F216

F245

F226

F225

F219

F250

F238

F249

Figure 1: MEDLINE to MEDIE Each input file results in 68 independent jobs, including natural language processing, database creation, managing and conversion of files described in XML, etc. Resource consuming tasks are listed below in the decreasing order for execution time. 1. Enju : HPSG parser, which constructs of predicate argument structures. 2. Akane : Protein-protein interaction inference from sentences. 3. Genia-tagger : part-of-speech tagging and shallow parsing for biomedical text. 9

http://www-tsujii.is.s.u-tokyo.ac.jp/medie/

Figure 2(a) shows a timing detail of the fifteen most time consiming tasks for 99 input files (491 MB). In the experiment, Enju is executed in 3584 times and each of the other tasks is executed 99 times (once for each input file). While the most time-consuming task, Enju, is clearly cpu-intensive, the second one, Akane, is very IO intensive. Moreover, it spends a significant amount of time on meta-data accesses. Overall, 46% of the whole time are consumed by file IO and metadata access (Figure 2(b)). Details of metadata accesses are shown in Table 2.

4.2 Similar Pages Extraction Similar Pages is a workflow that takes a large number of texts collected from the web as input and analyses them. It outputs all pairs of ’similar’ sentences, which have Hamming distances smaller than a specified threshold. The key algorithm, called Sachica, is able to enumerate all such pairs

Table 2: detail of file metadata access for M2M getattr readdir open count 1,749,482 99 3,418,394 aggregate time[sec.] 84,266 4 128,671 average time[msec.] 48 40 38 time ratio 0.39 0 0.59 create 21,642 1,199 55 0.01

count aggregate time[sec.] average time[msec.] time ratio

mkdir 496 33 67 0

unlink 4,871 1,069 219 0.01

30000 meta data read write stime utime

25000

20000

]c es [ 15000 e m i T 10000

5000 0 1

2

3

Ranking

rapidly [29]. The dependency DAG of Similar Pages Extraction workflow is shown in Figure 3. It has three kind of tasks:


• Mkbins : gather all input text files and split them into a given number (N ) of files.

utime stime write read

• Sachica1 : find pairs of similar subsequences in a single file.

metadata

• Sachica2 : find pairs of similar subsequences in two files, one from each of them. 4.txt

2.txt

5.txt

3.txt

8.txt

9.txt

0.txt

6.txt

7.txt

1.txt


mkbins

bin.1

sachica1

xresult.1.bz2

log.1

sachica2

xxresult.0.1.bz2

log.0.1

bin.0

sachica2

log.1.2

xxresult.1.2.bz2

bin.2

sachica1

xresult.0.bz2

bin.table.0

bin.table.1

sachica2

log.0

log.0.2

Figure 4: Similar Pages :Time detail of each job and of the whole workflow

bin.table.2

sachica1

xxresult.0.2.bz2

log.2

xresult.2.bz2

Figure 3: Similar Pages Extraction workflow Characteristic of the dependency DAG for Similar Pages is that Sachica1 and Sachica2 have N and N (N − 1)/2 jobs respectively, where N is the number of files Mkbins outputs. Each split file is read N times from Sachica1 and Sachica2. Figure 4 shows an experimental result for about 1 million (total 5 GB) input files with N = 30 (Mkbin:1, Schica1:30, Schica2:435). Mkbin is very file-access intensive, whose cost are mainly IO operation ’read’ and metadata operations ’get attribute’ and ’open’. Overall, file-access cost occupies the half of its execution time and ’read’ is the main reason of the file-access costs.

4.3 Case Frames Construction Case Frames Construction is a workflow to create data structures in natural language processing called case frames [30, 31]. Case frames, which are frequent patterns of predictargument structures, are generated by analysing large mount of text collected from the web. The workflow is mainly constructed from the following jobs in the increasing order with consuming time(Figure 7(a)): 1. parse-comp : a parser for Japanese texts with KNP [30] including named entity recognition.

2. make-caseframe : a job that creates case frames by extracting reliable predicate-argument examples from the parsing result. 3. cf-distribute-rest : a job that adds all case frames to clusters made by cf-clustering. 4. cf-clustering : clustering highly frequent case frames and merging them with similarity. 5. calc-of-similarity : computing similarities for each pair of case frames. The DAG for the workflow with 4 input files is shown in Figure 5. As the figure shows, all files output by the upstream jobs are gathered into one large file, and it is split into files with other size, and finally they are combined into a result file. So parallelism of the workflow has two peaks. Figure 6 shows the number of concurrent jobs with respect to the elapsed time (in seconds) of an execution. In the experiment, 59 % of the whole time is spent for file access, while the ratio of time for each job has large variance (Figure 7).

4.4 Finding Supernovae Finding Supernovae is a workflow to find coordinates where supernovae appears to be from astronomical data. The workflow uses the same software and data with Data Analysis

x000002.org

J16 0.06/0.06

J1 1.77/1.77

J0 2.01/2.01

J2 1.39/1.39

J3 1.64/1.64

all.list

x000001.jmn

x000000.jmn

x000003.jmn

x000002.jmn

J6 63.37/65.14

J7 89.46/91.47

J4 70.35/71.74

J5 65.86/67.5

x000001.org

x000001.knp

x000001.log

x000000.org

x000001.knp0

x000001.log0

x000000.knp

J8 13.23/78.37

x000000.knp0

x000003.org

x000000.log

J11 15.78/107.25

x000001.data

x000000.log0

x000003.knp

x000003.log0

x000003.log

J10 13.71/85.45

x000000.data

x000003.knp0

x000002.knp

x000002.log0

1000000

x000002.log

900000

x000002.knp0

J9 13.67/81.17

x000003.data

x000002.data


800000

J12 0.08/107.33

all.data

700000

J13 0.19/107.52

all.data.orig

]c 600000 es [ 500000 e m i T 400000

J14 0.2/107.72

all.data.sid

J15 0.25/107.97

all-000000.data.sid

all-000002.data.sid

all-000001.data.sid

J19 0.31/108.28

J18 0.31/108.28

J17 0.32/108.29

all-000000.data.basic


J20 0.06/108.35


J21 0.07/108.42

J25 0.12/108.47

J26 0.07/108.41

J29 0.16/108.5

all-000000-cross.log

all-000000-cross-wa.result

all-000000-cross.formatted

all-000002-cross-wa.formatted

all-000000-cross-wa-soto.result



100000

J36 0.26/109.15

all-000002-cross-wa.log





J39 1.02/110.17

all-000002-cross-wa-soto.log



all-000002-cross-wa-soto-others_tmp.log

all-000001-cross-wa-soto-others_tmp.formatted

J45 0.12/110.02

J54 0.14/110.55

all-000002-cross-wa-soto-delwa.formatted


J44 0.13/110.18

J46 0.23/110.25

J57 0.23/110.78


all-000002-cross-wa-soto-delwa-sub_tmp.formatted


J47 0.23/110.41

J49 0.07/110.32

J61 0.06/110.84


all-000002-cross-wa-soto-delwa-sub.formatted


J48 0.13/110.54

J51 0.07/110.39

J63 0.06/110.9


all-000002-cross-wa-soto-delwa-sub-refined.formatted


J53 0.06/110.6

J52 0.23/110.62

J64 0.24/111.14

all-000002-cross-wa-soto-delwa-sub-refined-or.formatted


J55 0.23/110.83

J56 0.13/110.75

J67 0.14/111.28


all-000002-cross-wa-soto-delwa-sub-refined-or-sorted.formatted


J58 0.12/110.95

J59 0.24/110.99

all-000002-cross-wa-soto-delwa-sub-refined-or-sorted-cs.formatted

J60 0.25/111.2

J62 0.24/111.23

J73 0.24/111.76


all-000002-cross-wa-soto-delwa-sub-refined-or-sorted-cs-sm.formatted


J65 0.23/111.43

J66 0.14/111.37

J75 0.13/111.89


all-000002-cross-wa-soto-delwa-sub-refined-or-sorted-cs-sm-postprocessed.formatted

J68 0.14/111.57

J72 0.07/111.44


J71 0.33/111.9

all-000000.cfsim.dat

J74 0.06/111.63

3

5

7

9

11 13 15 17 19 21 23 25 27 29 Ranking

(a) Time detail of jobs

utime

J70 0.24/111.52


all-000000-cross-wa-soto-delwa-sub-refined-or-sorted-cs-sm-postprocessed.log

1



all-000000-cross-wa-soto-delwa-sub-refined-or-sorted-cs.log

0

all-000001-cross-wa-soto.formatted

J50 0.24/110.41



200000

all-000001.data.basic.rest

J43 0.24/109.9


J42 0.24/110.05




J40 0.25/109.81



J41 0.25/109.66


J27 0.16/108.51

all-000001.1st.formatted

J33 0.26/108.89


300000

J30 0.28/108.63

all-000001.1st.log

J38 0.26/109.41


J37 0.25/109.56


J24 0.06/108.41

all-000002.1st.log

J35 0.54/109.15


J34 0.59/109.31



J32 0.27/108.61


J31 0.25/108.72

all-000000.1st.log

J22 0.07/108.35


all-000000suru.data.basic

J28 0.15/108.62


J23 0.06/108.34


all-000002-cross-wa-soto-delwa-sub-refined-or-sorted-cs-sm-postprocessed.log

J69 0.24/111.61

J77 0.06/111.95





stime write all-000001-cross-wa-soto-delwa-sub-refined-or-sorted-cs-sm-postprocessed.log

read

J76 0.22/112.11


metadata

all-000002.formatted



J78 0.06/112.01

cf.formatted

J79 0.22/112.23

cf.knpdict

Figure 5: Case Frames Construction (b) Time detail for whole workflow Figure 7: Details for Case Frames Construction r0002_10_t1.fits

r0002_10_t0.fits

J12 3.32/3.32

r0002_10_t1.fits.det

J14 11.01/14.65

r0002_10_t1.fits.mat

J21 0.24/14.89

r0002_10_t1.fits.coef

J22 5.96/20.85

r0002_10_t1_sft.fits

r0002_10_t1.fits.coef_r

r0002_10_t1.fits.dso

r0002_10_t1_obj.lst

r0000_00_t1.fits

J2 3.64/3.64


r0002_10_t0_obj.lst

r0002_10_t1_flg.fits

J6 0.23/3.87

msk.lst



J15 853.62/857.26

r0002_10_t0_r0002_10_t1_conv.fits


r0002_10_t1.reg

msk.lst

J19 0.12/14.43

r0002_10_t1_sub_msk.fits

r0000_00_t1.fits.coef

J8 1.58/862.22

r0002_10_t1.result

r0000_00_t1.fits.coef_r

J20 7.03/21.46

r0000_00_t0_obj.lst


J17 0.17/3.58

r0000_00_t1.fits.mat

J7 3.38/860.64

J1 3.41/3.41

r0000_00_t1_obj.lst

J13 10.9/14.31

r0002_10_t1_sub.fits

r0000_00_t0.fits

J11 3.17/3.17





J16 756.12/759.53

r0000_00_t0_r0000_00_t1_conv.fits

r0000_00_t1.reg

r0000_00_t1_sub.fits

J18 3.41/762.94

r0000_00_t1_sub_msk.fits

J3 2.82/765.76

r0000_00_t1_sft.fits

r0000_00_t1.result

J9 0.08/862.3

J4 0.09/765.85

r0002_10.result.tmp

r0000_00.result.tmp

J10 0.07/862.37

J5 0.07/765.92

r0002_10.result

r0000_00.result

J0 0.07/862.44

total_result

Figure 6: Parallelism of an execution for Case Frames Construction Challenge on IEEE Cluster/Grid 2008 10 . Each pair of astronomical images which are for the same position but different time is compared with each other and the positions of candidates for supernovae are listed up. Finally the candidate positions are gathered into one file. Figure 8 shows the DAG of the Finding Supernovae for 2 pairs of input images. In this workflow, the most heavy job (imbus3vp3) is relatively compute-intensive while other jobs are not.

4.5 Montage We execute Montage DAX and, take log and generate the dependency graph(Figure 11).

Figure 8: Finding Supernovae 3156 2MASS(J)images as input, where each size of them is about 2MB . As Figure 11(b) shows, almost of the aggregate time is consumed for file access. Concentrating all files to single file server becomes the bottleneck and degrades performance of the workflow. Percentage of cpu usage time to aggregate time becomes larger as the input size (Figure 11(c)), which means that concentrating data access degrades throughput of the file server.

5. Each input file is a astronomic image file, and the workflow gathers them into one picture. In the experiment we use 10

http://www.cluster2008.org/challenge/

COMPARISON FOR DESIGNS OF WORKFLOW SOLVERS

In this section, we investigate various cost for designs of workflow solvers such as staging, centorized file system, schedul-

600000

700000 meta data read write stime utime

600000 500000

]c es 400000 [ e m i T 300000


500000 400000

c]e [s 300000 e im T 200000

200000 100000

100000

0

0 3 p v 3 s u b im

t if h s im

3 t c e t e d

t r is o L t k c s ra a t x M k e s m

3 k s a m

2 h c t a m

2 t fi ly o p

it d iffF oun mD ckgr a mB

k w a

P dd tbl nk Fit l(2) mA jectP Shri mImg mgtb ncat o m o I r C m m mP

Ranking

Ranking

(a) Time detail of jobs


utime

utime

stime

stime

write

write

read

read

metadata

metadata


(b) Time detail for whole workflow 1.00

Figure 9: Details for Finding Supernovae

p2mass-atl as-98120 4n-j0420 044_area .fits

2mass-atla s-981204 n-j04200 44.fits


mProjectPP 2.28/2.28


p2mass-atl as-98120 4n-j0420 044.fits

mDiffFit 0.52/2.8

diff.001.0 04_area. fits

diff.001.0 04.fits

mDiffFit 0.38/2.66

fit.000001 .000004. txt

fit.000001 .000003. txt

diff.001.0 03.fits




mProjectPP 2.3/2.3


mDiffFit 0.43/2.71

fit.000003 .000004. txt


region.hdr






mDiffFit 0.47/2.77

diff.003.0 04.fits

fit.000002 .000004. txt



mDiffFit 0.37/2.67

statfile_n ew.tbl

diff.002.0 04.fits

fit.000002 .000003. txt

diff.002.0 03.fits


no it ar ep o hc ae ro f oit aR

0.90 0.80 0.70 0.50 0.40 0.30 0.20

mConcatFit 0.12/2.92

0.10

fits.tbl

0.00

pimages.tb l

metadata read write stime utime

0.60

28

mBgModel . 0.22/3.14

308

3474

# of input files

correction s.tbl

mBackground 0.49/3.63


c2mass-atl as-98120 4n-j0420 044.fits


c2mass-atl as-98120 4n-j0420 044_area .fits


cimages.tb l



newcimages .tbl

mAdd -e ne 0.39/4.21

mosaic_200 90330_19 4109_309 67.fits

(c) ratio of time for each oparations w.r.t. the number of input files


mImgtbl . 0.19/3.82




Figure 11: Montage :Time detail of each job and of the whole workflow

mosaic_200 90330_19 4109_309 67_area. fits

mShrink mo 0.2/4.41

shrunken_2 0090330_ 194109_3 0967.fit s

mJPEG -ct 0.3/4.71

shrunken_2 0090330_ 194109_3 0967.jpg

Figure 10: montage workflow

ing, based on logged data for workflows in the previous section.

5.1 Centralization Costs

A staging is a process to transfer required files to nodes in advance. Many staging methods send all updated and created files to central servers when jobs finish. Since intermediate files are not sent directly from the node who output them to other nodes in those methods, called centralized staging, bandwidth and latency to access for central servers frequently becomes a bottleneck especially for data intensive workflows. Many centralized file systems, such as NFS and LUSTRE, also have the same problem, whereas distributed file systems, such as Gmount and Gfarm, can be more effective since they locate files on local disks of nodes and send them

if necessary. While input and output files of a workflow are usually required to be located on file system, intermediate files, which are both read and written by jobs of the workflow, are not. We assume that files which are only read during the workflow execution are the input files and that files which are only written are the output files. Table 3 shows input, intermediate and output aggregate file size and the intermediate rate ( intermediate/(input + intermediate + output) ) for each workflow in the previous section. It is possible to avoid 40-88% of file transfer to central servers on centralized file systems and staging systems that can be become a bottleneck for the whole workflow performance if intermediate files does not send to the central servers.

Table 4: local read for locality-aware scheduling read size [GB] input local remote local read rate m2m 0.51 69.56 119.35 0.36 similarpages 2.49 0.17 74.61 0.002 caseframes 0.73 46.35 0.99 0.96 supernovae 233.73 37.50 67.54 0.11 montage 9.13 56.93 235.86 0.18

access latency to the metadata server on the file system with keeping consistency. Thus metadata operation cost should be able to reduce much less if staging is used in stead of a single file system, since metadata access is only in local nodes and keeping consistency with other nodes is not required with stating methods.

Table 3: ratio for the aggregate size of intermediate files input interm. output [GB] interm. rate m2m 0.51 97.43 13.25 0.88 similarpages 2.49 2.49 0.97 0.42 caseframes 0.73 46.74 58.51 0.44 supernovae 77.91 103.48 78.26 0.40 7.26 72.84 21.44 0.72 montage

Table 5: the ratio of metadata access costs ratio for whole ratio for file-access m2m 0.12 0.21 similarpages 0.13 0.18 caseframes 0.05 0.09 supernovae 0.04 0.10 montage 0.32 0.32

5.2 Locality-aware Scheduling When a scheduler for workflows decides the compute nodes where each job are dispatched, it is important to consider the cost to access files located on remote nodes. Jobs are preferred to be lunched at a compute node where as many input files are located on the local disk or memory. In many scheduling methods like HEFT [7], the cost to access files on remote nodes are considered when jobs are assigned to compute nodes, as well as predicted computation time for jobs, length of critical path, etc. Let us define that a job Y depends on a job X if some output file of X is input file of Y . If a job X has some jobs that depend on only X, it is very likely that, at least, one of jobs which depends on X is assigned to the same node with X with any criterion for assignments of jobs. In order to measure how much size of files are able to read from local in locality-aware scheduling methods, for each job X, a job Y which depends on X are randomly selected among jobs depends on X, and X and Y are assigned the same node. Talbe 4 shows the result of the assignment for each workflow. While Case Frames Construction has high local read rate local/(local+input+remote), Similar Pages Extraction does not. The difference of them mainly arise from the difference of the average number of branches in the DAGs of the workflows. If a workflow has the small averaged number of branches for a job in the DAG, considering locality awareness became more important. As Figure 3 and 5 shows, most jobs of Case Frames Construction have one branch while those of Similar Pages Extraction have 30 branches.

6.

CONCLUSION

In Section 5, advantages of modifying file systems, staging methods and scheduling methods are estimated by analysing dependency graphs generated from logs. Since those workflows have no explicit declaration of input and output files for each job, dependencies between files and jobs are created by analysing file access logs. Three key insights for performance are drawn from the experiments. First, the total load on the central file server decreases by 40 − 88% if intermediate files are sent not going through the central file servers, but directly from the producer to the consumer. Secondary, if locality-aware scheduling methods are applied, their effectiveneses highly depend on the shapes of DAGs (the reductions are from 0.002−0.96). In particular, it largely depends on the number of children per job. Lastly, 9 − 32% of whole file access cost can be eliminated when a staging is used instead of a file system, because a large part of file access costs comes from the metadata operation latency between clients and servers. Our future direction includes using profiled information to statistically infer various information about jobs such as names of input files, names of output files, size of output files, and the execution time of the job from the command line. If many of them are successfully estimated, it is possible to build a framework in which jobs can be described without explicit staging directives, files transparently shared across nodes, yet executed very efficiently.

5.3 Costs for Metadata Operations

Acknowledgment

Metadata operations often be non-negligible costs on a single file system. Table 5 shows the experimental result in the previous section shows. The reason for the cost of metadata operations hardly depends on bandwidth limitation but on

This work was supported by Grant-in-Aid for Specially Promoted Research (MEXT, Japan) and the MEXT Grant-inAid for Scienti?c Research on Priority Areas project “ New IT Infrastructure for the Information-explosion Era ”

7. REFERENCES [1] D. Thain, T. Tannenbaum, and M. Livny, “Distributed computing in practice: the condor experience.” Concurrency - Practice and Experience, vol. 17, no. 2-4, pp. 323–356, 2005. [2] E. Deelman, G. Singh, M.-H. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, G. B. Berriman, J. Good, A. Laity, J. C. Jacob, and D. S. Katz., “Pegasus: a framework for mapping complex scientific workflows onto distributed systems,” Scientific Programming Journal, vol. 13, no. 3, pp. 219–237, 2005. [3] T. Oinn, M. Addis, J. Ferris, D. Marvin, M. Senger, M. Greenwood, T. Carver, K. Glover, M. R. Pocock, A. Wipat, and P. Li, “Taverna: a tool for the composition and enactment of bioinformatics workflows,” Bioinformatics, vol. 20, no. 17, pp. 3045–3054, 2004. [4] J. Dean and S. Ghemawat, “Mapreduce: Simplified data processing on large clusters,” in Proceedings of the 6th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2004, p. 137 窶 50. [5] M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, “Dryad: Distributed data-parallel programs from sequential building blocks,” in Proceedings of the 2007 Eurosys Conference, 2007. [6] A. Mandal, K. Kennedy, C. Koelbel, G. Marin, J. Mellor-Crummey, B. Liu, and L. Johnsson, “Scheduling strategies for mapping application workflows onto the grid,” in High Performance Distributed Computing, 2005. HPDC-14. Proceedings. 14th IEEE International Symposium on, July 2005, pp. 125–134. [7] H. Topcuoglu, S. Hariri, and M.-Y. Wu, “Performance-effective and low-complexity task scheduling for heterogeneous computing,” Parallel and Distributed Systems, IEEE Transactions on, vol. 13, no. 3, pp. 260–274, Mar 2002. [8] Z. Yu and W. Shi, “An adaptive rescheduling strategy for grid workflow applications,” in Parallel and Distributed Processing Symposium, 2007. IPDPS 2007. IEEE International, March 2007, pp. 1–8. [9] E. Ilavarasan and P. Thambidurai, “Low complexity performance effective task scheduling algorithm for heterogeneous computing environments,” Journal of Computer Science, vol. 3, no. 2, pp. 94–103, 2007. [10] R. Sakellariou and H. Zhao, “A hybrid heuristic for dag scheduling on heterogeneous systems,” in Parallel and Distributed Processing Symposium, 2004. Proceedings. 18th International, April 2004, pp. 111–. [11] W. Guo, W. Sun, W. Hu, and Y. Jin, “Resource allocation strategies for data-intensive workflow-based applications in optical grids,” in Communication systems, 2006. ICCS 2006. 10th IEEE Singapore International Conference on, Oct. 2006, pp. 1–5. [12] J. Blythe, S. Jain, E. Deelman, Y. Gil, K. Vahi, A. Mandal, and K. Kennedy, “Task scheduling strategies for workflow-based applications in grids,” Cluster Computing and the Grid, IEEE International Symposium on, vol. 2, pp. 759–767, 2005. [13] H. Abbasi, M. Wolf, G. Eisenhauer, S. Klasky, K. Schwan, and F. Zheng, “Datastager: scalable data

[14]

[15]

[16]

[17]

[18]

[19]

[20]

[21]

[22]

[23]

[24]

[25]

staging services for petascale applications,” in HPDC ’09: Proceedings of the 18th ACM international symposium on High performance distributed computing. New York, NY, USA: ACM, 2009, pp. 39–48. T. Kosar and M. Livny, “Stork: making data placement a first class citizen in the grid,” in Distributed Computing Systems, 2004. Proceedings. 24th International Conference on, 2004, pp. 342–349. I. Raicu, Y. Zhao, C. Dumitrescu, I. Foster, and M. Wilde, “Falkon: a fast and light-weight task execution framework,” in SC ’07: Proceedings of the 2007 ACM/IEEE conference on Supercomputing. New York, NY, USA: ACM, 2007, pp. 1–12. G. Khanna, N. Vydyanathan, U. Catalyurek, T. Kurc, S. Krishnamoorthy, P. Sadayappan, and J. Saltz, “Task scheduling and file replication for data-intensive jobs with batch-shared i/o,” in High Performance Distributed Computing, 2006 15th IEEE International Symposium on, 0-0 2006, pp. 241–252. I. Raicu, I. T. Foster, Y. Zhao, P. Little, C. M. Moretti, A. Chaudhary, and D. Thain, “The quest for scalable support of data-intensive workloads in distributed systems,” in HPDC ’09: Proceedings of the 18th ACM international symposium on High performance distributed computing. New York, NY, USA: ACM, 2009, pp. 207–216. K. Ranganathan and I. Foster, “Decoupling computation and data scheduling in distributed data-intensive applications,” in High Performance Distributed Computing, 2002. HPDC-11 2002. Proceedings. 11th IEEE International Symposium on, 2002, pp. 352–358. T. Kosar, “A new paradigm in data intensive computing: Stork and the data-aware schedulers,” in Challenges of Large Applications in Distributed Environments, 2006 IEEE, 0-0 2006, pp. 5–12. F. Dong and S. G. Akl, “Scheduling algorithms for grid computing: State of the art and open problems,” School of Computing, Queen’s University, Tech. Rep., January 2006. S. Bharathi, A. Chervenak, E. Deelman, G. Mehta, M.-H. Su, and K. Vahi, “Characterization of scientific workflows,” in Workflows in Support of Large-Scale Science, 2008. WORKS 2008. Third Workshop on, Nov. 2008, pp. 1–10. B. Pawlowski, C. Juszczak, P. Staubach, C. Smith, D. Lebel, and D. Hitz, “Nfs version 3 - design and implementation,” in In Proceedings of the Summer USENIX Conference, 1994, pp. 137–152. F. Schmuck and R. Haskin, “Gpfs: A shared-disk file system for large computing clusters,” in In Proceedings of the 2002 Conference on File and Storage Technologies (FAST, 2002, pp. 231–244. S. A. Weil, S. A. Brandt, E. L. Miller, D. D. E. Long, and C. Maltzahn, “Ceph: A scalable, high-performance distributed file system,” in Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2006, p. 307 窶 20. O. Tatebe, Y. Morita, S. Matsuoka, N. Soda, and S. Sekiguchi, “Grid datafarm architecture for petascale data intensive computing,” in Cluster Computing and

[26]

[27] [28]

[29]

[30]

[31]

the Grid, 2002. 2nd IEEE/ACM International Symposium on, May 2002, pp. 102–102. N. Dun, K. Taura, and A. Yonezawa, “Gmount: An ad hoc and locality-aware distributed file system by using ssh and fuse,” in 9th IEEE/ACM International Symposium on Cluster Computing and the Grid, 2009, pp. 188–195. “Montage: An astronomical image mosaic engine,” http://montage.ipac.caltech.edu/. N. Dun, K. Taura, and A. Yonezawa, “Paratrac: A fine-grained profiler for data-intensive workflows,” in to appear; 19th IEEE Symposium on High Performance Distributed Computing, 2010. T. Uno, “A new approach for speeding up enumeration algorithms,” in Proceeding of ISAAC98(Lecture Note in Computer Science 1533, Springer-Verlag, Algorithms and Computation), 1998, pp. 287–296. N. K. Daisuke Kawahara and S. Kurohashi, “Japanese case structure analysis by unsupervised construction of a case frame dictionary,” in International Conference on Computational Linguistics (COLING2000), 2000, pp. 432–438. R. Sasano, D. Kawahara, and S. Kurohashi, “The effect of corpus size on case frame acquisition for discourse analysis,” in the North American Chapter of the Association for Computational Linguistics Human Language Technologies (NAACL-HLT2009), 2009, pp. 521–529.

File-Access Patterns of Data-Intensive Workflow Applications and ...

File-Access Patterns of Data-Intensive Workflow Applications and ...

Suggest Documents

Workflow Patterns in Orc

workflow factors and patterns of use

Workflow-based applications - CiteSeerX

Observing health professionals' workflow patterns

LNCS 4001 - Workflow Exception Patterns

Workflow-based applications - Semantic Scholar

Workflow Applications to Research Agenda

Workflow Completion Patterns - Eindhoven University of Technology

Patterns of Workflow Management Facility - CiteSeerX

Pattern Based Analysis of BPEL4WS - Workflow Patterns

UML Activity Diagrams as a Workflow ... - Workflow Patterns

Exception Handling in Workflow-Driven Web ... - Workflow Patterns

Applications of Web-Based Workflow - CiteSeerX

Overhead Analysis of Grid Workflow Applications - Distributed ...

The Implementation Procedure of Organizational Workflow Applications (

an interactive composition of workflow applications

Applications of Web-Based Workflow - CiteSeerX

Exception Handling Patterns in Process-Aware ... - Workflow Patterns

Multi-criteria Mapping and Scheduling of Workflow Applications ... - graal

Workflow Patterns in Orc - Orc Language

Transactional Patterns: Combining Workflow Flexibility ... - CiteSeerX

newYAWL: Achieving Comprehensive Patterns Support in Workflow

newYAWL: Achieving Comprehensive Patterns Support in Workflow ...

Transactional Patterns: Combining Workflow Flexibility ... - CiteSeerX