Towards Workflow Sharing and Reuse in the ... - Semantic Scholar

Towards Workflow Sharing and Reuse in the ASKALON Grid Environment Jun Qin1 , Francisco Hern´andez2 , Erik Elmroth2 , and Thomas Fahringer1 1

2

Institute of Computer Science, University of Innsbruck Technikerstr. 21a, 6020 Innsbruck, Austria {jerry,tf}@dps.uibk.ac.at Department of Computing Science and HPC2N, Ume˚a University SE-901 87 Ume˚a, Sweden {hernandf,elmroth}@cs.umu.se

Abstract. As more Grid workflow applications are built, workflow sharing and reuse becomes important, because (i) building a Grid workflow application from scratch is still a challenging task for domain scientists, and (ii) reusing of established and validated workflows not only reduces workflow authoring time but also improves the quality of workflows. In this paper, we present a Grid workflow sharing and reuse framework and its implementation in the ASKALON Grid environment. The framework introduces the ASKALON Workflow Hosting Environment (AWHE) for workflow sharing among different research groups. Extensions of the Abstract Grid Workflow Language (AGWL) for simple reuse of Grid workflow components, sub-workflows, and workflows are also presented. We illustrate the applicability of our framework using two real-world Grid workflows from different scientific domains.

1 Introduction Scientific workflow systems [1–5] are increasingly being used by scientists to construct complex scientific experiments and to execute them on distributed Grid resources. Much work in Grid workflows has focused on improving performance by optimizing the use of computational resources and network bandwidth. These optimizations are typically performed by middleware services such as meta-schedulers and resource managers for scientific workflow systems. As the gain in productivity obtained by workflows is more noticeable, scientists have started sharing and reusing them. For example, a domain scientist in one research group wants to run a workflow constructed by another group and to obtain results useful for his own research, but without knowing how to construct a Grid workflow application. In other cases domain scientists want to invoke a workflow multiple times (e.g., based on a for loop) within their own workflows. However, who creates the invoked workflow is not important, the workflow may have been composed by the same scientist or some other scientists, and it may have been defined inside or outside the workflow being composed. As presented in these examples, workflow sharing and reuse are important and must be supported by modern Grid workflow systems. ASKALON [5] is a Grid environment with the main goal of providing an invisible Grid for application developers. The Abstract Grid Workflow Language (AGWL) [6] is an XML-based language for describing Grid workflow applications at a high level of abstraction. The AGWL is the main interface to ASKALON. GLARE [7], as part of the ASKALON resource manager service, is a distributed activity registry service.

It provides dynamic registration, automatic deployment and on-demand provision of workflow components that can be used to build Grid workflow applications. Details about ASKALON, AGWL, and GLARE can be found in our previous work [5–7]. In this paper, we introduce a novel approach for workflow sharing and reuse that is part of the ASKALON Grid environment. The approach introduces the ASKALON Workflow Hosting Environment (AWHE) for workflow sharing among different research groups, and extensions to the AGWL that enables the reuse of Grid workflows. With workflow sharing and reuse, domain scientists are not required to build every workflow from scratch. The workflow authoring time is significantly reduced and the quality of workflows is also improved by reusing established and validated workflows. The remainder of this paper is organized as follows. Section 2 and Section 3 discuss our Grid workflow sharing and reuse framework in detail. Section 4 presents the implementation of the framework. We apply our approach to two real-world Grid workflow applications in Section 5. Section 6 presents important related work. The paper ends with a short conclusion.

2 Workflow Sharing – the ASKALON Workflow Hosting Environment (AWHE) Building large workflows from scratch can be a lengthy process, sometimes taking months of time. Some domain scientists use workflows only for obtaining results for their research and are not interested in how workflows are built. Manually keeping track of the relationships among different versions of a workflow is a challenging task. Moreover, sharing workflows via email or file copying is error prone. Based on these observations, we have developed AWHE to share Grid workflows among research groups. The AWHE uses a backend database to store and retrieve Grid workflows, as well as their corresponding metadata (e.g., domain, author, version). The AWHE works with the ASKALON workflow composition tool [8] and provides functionalities for users to: – publish workflows directly in the workflow composition tool. – associate workflow metadata (versions can be generated automatically based on the latest version in the database). – search workflows by their metadata. – show views of workflow graphs, including graphs of its sub-workflows – run workflows with a single click. – mark a workflow as deprecated for discouraging their use because a better alternative exists. Domain users can run workflows directly from the AWHE without modifications or specify their own input data and locations for their output data. However, for this to happen, users must be capable of reading the input data (specified via the source attribute of the input data ports) as well as being able to write the output data to the locations specified in the output data ports (via the saveto attribute). Furthermore, changes to back-end resources and the location of binary executables are transparent to users as they are automatically performed by the ASKALON runtime system. The use of the AWHE enables a categorization of ASKALON users into three classes. Workflow Executors execute existing, validated workflows by means of the

AWHE. Workflow Designers, compose workflows by dragging and dropping available activity types, sub-workflows, and workflows into new workflows. Before the workflow is published they can preset, to default values, the input and output data ports information or they can leave it empty for (potential) users to fill. Application Providers deploy binary executables and Web/Grid services into the Grid and associate them with activity types via GLARE.

3 Workflow Reuse In ASKALON, workflow reuse is supported not only for workflow components but also for sub-workflows and entire workflows. AGWL workflows consist of activities, control flow and data flow constructs, and sub-workflows. Each atomic activity in the AGWL has a type attribute (see Ex. 1) which refers to an activity type registered in GLARE. An activity type is an abstract description of a group of activity deployments (i.e., concrete implementations of computational entities deployed in the Grid) that have the same functionality but probably different performance behaviors. Activity types shield the implementation details of activity deployments from AGWL programmers. As there are no implementation details specified, activity types are reusable workflow components. Example 2. Invoking a sub-workflow

Example 1. Activity and Activity Type 1 2 3 4 5 6 7 8

... ...

1 2 3 4 5 6 7 8 9 10



Similar to procedures in high-level programming languages, sub-workflows are regions of code (defined by the tag ) in AGWL workflows. The code regions are typically invoked multiple times in the enclosing workflow. In order to support both sub-workflow and entire workflow reuse, we extended the semantics of the type attribute of atomic activities in the AGWL, so that it can refer to activity types, sub-workflows or workflows. If an activity has a type referring to a sub-workflow or a workflow, executing this activity invokes the corresponding sub-workflow or workflow. The possible values for the type attribute of atomic activities are: – atrp://domain/activitytype, referring to an activity type registered in the activity type repository, i.e., GLARE. – subw://subworkflow, referring to a sub-workflow that is specified as part of the workflow containing the atomic activity. – awhe://domain/workflow/version, referring to a workflow published in the AWHE. A workflow in the AWHE can be determined by its domain, its name, and its version. Thus, ASKALON provides a simple and consistent way for workflow component, subworkflow, and entire workflow reuse. In order to avoid name conflicts, we use different

prefixes for these three kinds of reuse. Ex. 2 illustrates this idea by showing an example of sub-workflow reuse: the workflow w defines a sub-workflow subW (line 2). which is invoked at line 5 by the activity a. 3.1

Sub-Workflow Reuse

Sub-workflows can be invoked when executing atomic activities by referencing the sub-workflows in the type attribute of the atomic activity. This enables recursive subworkflow invocations for cases where the atomic activity is defined in a sub-workflow. As in high level programming languages (e.g., Java or C++) not well designed recursive sub-workflow invocations can provoke infinite recursion. To address this problem, incorrect recursive sub-workflow invocations are detected by traversing the workflow models while building the workflows and before submitting them for execution on Grid resources. The purpose is to give feedback as early as possible and to make sure that no Grid resources are employed executing incorrect workflows. Our approach for detecting incorrect sub-workflow invocations is described in Algorithm 1. Method 1 is the entry point of the algorithm and for each sub-workflow invoked from the main workflow, it detects recursive sub-workflow invocations (Method 2) and, when found, it evaluates whether the recursive sub-workflow invocation found is correct (Method 3). Specifically, Method 2 detects recursive invocations by recursively checking all possible sub-workflow invocation paths. Method 3 evaluates the correctness of a recursive sub-workflow invocation path by checking whether there are data processing steps (i.e., activities) between two invocations of the same sub-workflow. If there are no activities in between, the sub-workflow invocation path is considered to be incorrect because it will provoke infinite recursion. Fig. 1 illustrates three examples of incorrect recursive sub-workflow invocations. In Fig. 1(a), subA will cause infinite recursion because the first activity in subA invokes the sub-workflow itself. This is also the case with subB (Fig. 1(b)) if the then branch is executed at runtime. Finally, Fig. 1(c) shows a case of incorrect recursive sub-workflow invocations involving different sub-workflows invoking each other. subB

subA

subw://subA

then

else

subC

subD

subw://subD

subw://subC

...

...

...

subw://subB

... ...

(a) subA invoking subA

(b) subB invoking subB

(c) subC and subD invoking each other

Fig. 1. Examples of Incorrect Recursive Sub-Workflow Invocations

Algorithm 1 Find Incorrect Recursive Sub-Workflow Invocation Method 1: detectIncorrectRecursiveInvocation() Input: W : the workflow to detect incorrect recursive sub-workflow invocation. Output: Λ: the sub-workflow invocation path with the last element being the sub-workflow being incorrectly recursively invoked; Λ = ∅ means no incorrect recursive sub-workflow invocation found. Initialize the sub-workflow invocation path Λ = ∅ for all sub-workflow sub invoked in the workflow W do detectRecursiveSubWorkflowInvocation(Λ, sub) if Λ 6= ∅ ∧ ¬ isCorrectRecursiveInvocation(Λ) then return Λ end if end for return ∅ Method 2: detectRecursiveSubWorkflowInvocation() Input: Λ: the sub-workflow invocation path; sub: the sub-workflow to be invoked next Output: true, if a recursive sub-workflow invocation is found; false, otherwise if contains(Λ, sub) then append(Λ, sub) return true end if append(Λ, sub) for all activity a in the sub-workflow sub such that a has a type referring to a sub-workflow s do b = detectRecursiveSubWorkflowInvocation(Λ, s) {recursively call Method 2} if b then return true end if end for removeLastElement(Λ) return false Method 3: isCorrectRecursiveInvocation() Input: Λ: the sub-workflow invocation path with the last element being the sub-workflow being recursively invoked if there are activities between two invocations of the last sub-workflow then return true else return false end if

3.2

Entire Workflow Reuse

As presented in Section 2, to facilitate workflow execution, workflow designers may publish their workflows into the AWHE with predetermined or commonly used input data and output data. However, reusing workflows frequently requires that the reused workflow process the data produced by the activities in the new workflow. We address

this problem by generating workflow templates of the reused workflows (i.e., removing the values of the source and saveto attributes from the workflow input and output data ports) and then use the templates in the new workflows. Workflow designers can assign new values to the workflow input data by connecting the data ports of the templates and of the activities in the new workflows, with data flow links. We support two kinds of AGWL code generation mechanisms related to workflow reuse: (1) embedding the AGWL code of the reused workflows into the new workflow so that the workflow engine can execute the new workflow directly without contacting with AWHE, and (2) using the references of the reused workflows in the new workflow without any embedded AGWL code. The latter is the so-called late binding mechanism. The workflow engine can obtain the code of the reused workflows from the AWHE when needed. This is also the distinction between sub-workflow reuse and workflow reuse as the AGWL code for sub-workflows is always embedded while the code of external workflows is not. Moreover, the ASKALON Grid workflow composition tool also has the capability of publishing sub-workflows to the AWHE, as single workflows for reusing them in other workflows.

4 Implementation Our Grid workflow sharing and reuse framework has been implemented as part of the ASKALON Grid environment [5] and integrated with the ASKALON Grid workflow composition tool [8]. The AWHE uses a backend PostgreSQL database to store AGWL representations of Grid workflows. Two menu items “Open from AWHE...” and “Publish into AWHE...” have been added to the GUI of the ASKALON workflow composition tool for domain users to load and publish workflows. The AWHE can run as a standalone application. Thus, the workflow executors can search for workflows based on workflow metadata (e.g., domain, author, version), load views of workflow graphs, and run workflows without using the ASKALON workflow composition tool. In the ASKALON Grid workflow composition tool, the main workflow and subworkflows are represented as separate UML Activity Diagrams (Fig. 2). A workflow model consists of a main Activity Diagram and some sub-workflow Activity Diagrams. Each Activity Diagram has its own InitialNode and FinalNode representing the (sub)workflow input and output respectively. By clicking the “AGWL” tab associated with an Activity Diagram, the built-in workflow traverser can traverse the corresponding (sub-)workflow. The workflow traverser can also assemble all Activity Diagrams into a complete workflow before the workflow is submitted for execution on Grid resources. Three plugins (Fig. 2) facilitate the reuse of workflows in the ASKALON Grid workflow composition tool. The ActivityTypes plugin lists the activity types registered in GLARE. The SubWorkflows plugin lists the sub-workflows currently specified in the main workflow. And the Workflows plugin lists the workflows published in the AWHE. Each plugin provides a quick search filter to avoid scrolling a long list to look for an activity-type/sub-workflow/workflow. The filter of the Workflows plugin supports searching workflows based on their metadata. To create an activity with the type attribute referring to an activity type, a sub-workflow, or a workflow, domain users can drag an item from the corresponding list, and then either drop it directly

into the workflow diagram or drop it into the type TextField of a selected atomic activity (see arrows in Fig. 2). When this is done, the input and output data ports of the created activity are set to the input and output data ports of the dragged activitytype/sub-workflow/workflow automatically. Then the corresponding data flows can be established by connecting the corresponding data ports.

Fig. 2. (Sub-)Workflow Diagrams and Three Plugins

5 Case Studies In this section, we demonstrate our approach by applying it in two real-world Grid workflow applications. 5.1

MeteoAG

MeteoAG [9] is a meteorology simulation based on the numerical model RAMS [10]. The simulations produce precipitation fields of heavy precipitation cases over the western part of Austria, at a spatial and temporal grid, in order to resolve most alpine watersheds and thunderstorms. Fig. 3(a) illustrates the UML Activity Diagram representation of the workflow which calculates multiple simulation cases in parallel. The parallelFor construct pForAkmin in the loop body of the parallelForEach construct pForEachCase consists of activities required for simulating a single case. The activities defined in pForAkmin can be moved out to an independent sub-workflow. This sub-workflow can then be invoked from the main workflow. Fig. 3(b) shows the revised workflow based on this idea. As it can be seen, the workflow is simplified in terms of

control and data flow (black arrows are control flows and blue arrows with small squares are data flows. Control and data flows may be overlapped). Another advantage of this approach is that the extracted sub-workflow can be reused in other workflows, assuming the sub-workflow is published in the AWHE.

simulation_init



pForEachCase InitialNode/casedata





pForAkmin



akmin=1:40:10

rams_makevfile

rams_init

akmin=1:40:10





rams_makevfile

rams_init







simulation_init

raver

revu_compare

else

then

pForEachCase IntitialNode/casedata





case_init



rams_hist

rams_hist

pForTimeStep



revu_compare



raver



pForAkmin

case_init

timestep=1:12:1



simulateOneCase



pForTimeStep

timestep=1:12:1



revu_dump



revu_dump

(a) a single workflow

(b) the main workflow (left) and the sub-workflow simulateOneCase (right)

Fig. 3. Two Representations of the MeteoAG Workflow

5.2

WIEN2k

WIEN2k [11] is a program package for performing electronic structure calculations of solids using density functional theory. The workflow is illustrated in Fig. 4(a) and consists of a doWhile loop (represented by a DecisionNode and a control flow edge flowing back a predecessor, i.e., the activity lapw0). At the end of each loop iteration an output value is evaluated against a threshold to see whether the computation converged. Similar to the MeteoAG workflow, the body of the loop can be extracted as a sub-workflow. The sub-workflow can then be invoked from the doWhile loop in the main workflow (Fig. 4(b)). As a consequence of this “refactoring”, the workflow model is simplified and the modularization of the workflow is improved. This process also facilitates workflow maintenance and evolution as any change to the sub-workflow will be reflected on any activity, sub-workflow, or workflow that has a reference to the sub-workflow.

6 Related work Altintas et al. [12] developed a framework that supports the design and reuse of Grid workflows in the Kepler scientific workflow system [4]. Their work focused on abstract-





init

sumpara



lapw0





lapw0

sumpara



pfLAPW1

taskNumber=1::1



init

pfLAPW1



lcore



lapw1

taskNumber=1:init/kpoint:1

lcore



lapw1





runOnce

mixer



mixer



lapw2FERMI

lapw2FERMI

pforLAPW2

testconv



pForLAPW2 taskNumber=1::1

taskNumber=1:init/kpoint:1



testconv





lapw2TOT

lapw2TOT

(a) a single workflow

(b) the main workflow (left) and a sub-workflow runOnce (right)

Fig. 4. Two Representations of the WIEN2k Workflow

ing common used workflow components (e.g. for data movement, database querying, etc.) into a set of generic, reusable tasks. Goderis et al. [13, 14] identified seven bottlenecks to scalable workflow reuse and repurposing in e-Science. Their work mainly discussed how reasoning over ontologies, written in the Web Ontology Language (OWL) and the description logic (DL), could help discovering and ranking workflow fragments for reuse and repurposing. Laszewski et al. [15] presented an architecture of a repository service for Grid workflow components. The repository is developed to facilitate the component reuse in Grid workflows. Finally, Cao et al. [16] also only talks about workflow components reuse. While most related work, except [13, 14], is only concerned with workflow component reuse, our work introduces a framework that supports not only reusing workflow components but also reusing sub-workflows and entire workflows. With the activity type repository service GLARE, the AWHE and the extensions of the AGWL, we enable reusability at different levels and in a simple and consistent way.

7 Conclusions As more and more Grid workflows are being designed and built, workflow sharing and reuse becomes important and necessary for enhancing scientists’ productivity. In this work, we presented a framework for workflow sharing and reuse in the ASKALON Grid environment that addresses this problem. The framework provides a simple and consistent way for reusing workflows, supporting workflow components, sub-workflows, and entire workflows. The implementation of the framework is presented and the use of the framework with two real world Grid workflows illustrates the effectiveness of the framework.

References 1. Taylor, I., Wang, I., Shields, M., Majithia, S.: Distributed computing with Triana on the Grid. Concurrency and Computation: Practice and Experience (2005)

2. Deelman, E., Singh, G., Su, M.H., Blythe, J., Gil, Y., Kesselman, C., Mehta, G., Vahi, K., Berriman, G.B., Good, J., Laity, A., Jacob, J.C., Katz, D.: Pegasus: a Framework for Mapping Complex Scientific Workflows onto Distributed Systems. Scientific Programming Journal 13(2) (November 2005) 3. Oinn, T., Addis, M., Ferris, J., Marvin, D., Senger, M., Greenwood, M., Carver, T., andMatthew R. Pocock, K.G., Wipat, A., Li, P.: Taverna: A tool for the composition and enactment of bioinformatics workflows. Bioinformatics Journal 20(17) (June 2004) 3045–3054 4. Altintas, I., Berkley, C., Jaeger, E., Jones, M., Lud¨ascher, B., Mock, S.: Kepler: An Extensible System for Design and Execution of Scientific Workflows. In: 16th Intl. Conf. on Scientific and Statistical Database Management (SSDBM’04), Santorini Island, Greece, IEEE Computer Society Press (June 21-23, 2004) 5. Fahringer, T., Prodan, R., Duan, R., Hofer, J., Nadeem, F., Nerieri, F., Podlipnig, S., Qin, J., Siddiqui, M., Truong, H.L., Villazon, A., Wieczorek, M.: ASKALON: A Development and Grid Computing Environment for Scientific Workflows. In: Workflows for eScience, Scientific Workflows for Grids. Springer Verlag (2007) 6. Fahringer, T., Qin, J., Hainzer, S.: Specification of Grid Workflow Applications with AGWL: An Abstract Grid Workflow Language. In: Proceedings of IEEE International Symposium on Cluster Computing and the Grid 2005 (CCGrid 2005), Cardiff, UK, IEEE Computer Society Press (May 9-12, 2005) 7. Siddiqui, M., Villazon, A., Hofer, J., Fahringer, T.: GLARE: A Grid Activity Registration, Deployment and Provisioning Framework. In: SC’05: Proceedings of the 2005 ACM/IEEE conference on Supercomputing, Washington, DC, USA, IEEE Computer Society (2005) 52 8. Qin, J., Fahringer, T., Pllana, S.: UML Based Grid Workflow Modeling under ASKALON. In: Proceedings of 6th Austrian-Hungarian Workshop on Distributed and Parallel Systems, Innsbruck, Austria, Springer-Verlag (September 21-23 2006) 9. Sch¨uller, F., Qin, J., Nadeem, F., Prodan, R., Fahringer, T., Mayr, G.: Performance, Scalability and Quality of the Meteorological Grid Workflow MeteoAG. In: Proceedings of 2nd Austrian Grid Symposium, Innsbruck, Austria, OCG Verlag (September 21-23, 2006) 10. Cotton, W.R., Pielke, R.A., Walko, R.L., Liston, G.E., Tremback, C.J., Jiang, H., McAnelly, R.L., Harrington, J.Y., Nicholls, M.E., Carrio, G.G., McFadden, J.P.: RAMS 2001: Current status and future directions. Meteorology and Atmospheric Physics 82 (2003) 5–29 11. Blaha, P., Schwarz, K., Madsen, G., Kvasnicka, D., Luitz, J.: WIEN2k: An Augmented Plane Wave plus Local Orbitals Program for Calculating Crystal Properties (2001) 12. Altintas, I., Birnbaum, A., Baldridge, K.K., Sudholt, W., Miller, M., Amoreira, C., Potier, Y., Ludaescher, B.: A Framework for the Design and Reuse of Grid Workflows. In: Proceedings of Scientific Applications of Grid Computing. (2005) 13. Goderis, A., Sattler, U., Lord, P., Goble, C.: Seven bottlenecks to workflow reuse and repurposing. In: Fourth International Semantic Web Conference (ISWC 2005). Volume 3792., Galway, Ireland (2005) 323–337 14. Goderis, A., Sattler, U., Goble, C.: Applying descriptions logics for workflow reuse and repurposing. In: International Description Logics Workshop, Edinburgh, Scotland (2005) 15. von Laszewski, G., Kodeboyina, D.: A Repository Service for Grid Workflow Components. In: International Conference on Autonomic and Autonomous Systems International Conference on Networking and Services, Papeete, Tahiti, French Polynesia, IEEE (2005) 16. Cao, J., Mou, Y., Wang, J., Zhang, S., Li, M.: A Dynamic Grid Workflow Model Based On Workflow Component Reuse. In: Proceedings of Grid and Cooperative Computing. (2005)