Rapid component-based development

Rapid component-based development for Condor job submission, Jae-Woo Lee, David Braun, and Carol X. Song

I

Rapid component-based development for Condor j ob submission Jae-Woo Lee, David Braun, and Carol X. Song Rosen Center for Advanced Computing Purdue University, West Lafayette, IN 47906 {jaewoolee, dbraun, carolxsong}@purdue.edu

Abstract- Many scientific, engineering and business applications benefit from the ability to submit a large number of computational jobs to a Condor pool simultaneously. However, crafting a correct Condor submission script is not trivial, especially for the less computer command line-savvy users. Therefore, developers often wish to create a job submission page in a web client application for the Condor system to provide an easy way for users to run their computation jobs. In doing so, the developers must understand the Condor ClassAd mechanism and provide a proper transformation if the application needs to support user or application defined ClassAds. This paper describes a rapid and easy way of creating a job submission page using our visual web components. By integrating these components into a popular web application Integrated Development Environment (IDE), the developers can easily compose a Condor job submission page by dragging and dropping the components. This will significantly reduce the time spent on creating the job submission application.

Index Terms- Programming environments, Graphical interfaces, Condor job submission

I.

Condor is

a

user

INTRODUCTION

widely used high throughput grid computing

system [1,2]. At Purdue, its Condor pool had nearly 10 million

jobs from campus and national users, including TeraGrid users, in 2007 [15], and we see the usage growing in the first 9 months in 2008, 11 million jobs were executed in the Purdue Condor. One of the issues often encountered by the users is to be able to match jobs to machines in the pool. The ClassAd mechanism in Condor provides a flexible and expressive framework for matching resource requests (jobs) with resource offers (machines) [1]. However, it is not always easy to compose the job submission script with the correct ClassAds. Condor pool administrators may define different ClassAds for each pool to properly characterize the capabilities of the computers in the pool, allowing better resource matching and resulting in productivity gains for end users. We have observed that, partly due to the complexities of ClassAds and lack of -

Jae-Woo Lee, David Braun, and Carol X. Song, "Rapid component-based development for Condor job submission", Proceedings of Grid Computing Environments 2008.

knowledge of the correct usage, the end users often have errors in their Condor submit script. To make it easy for end users to send jobs to Condor pools, developers create customized web applications that hide the complexities of syntax. In doing so, the developers must understand the ClassAd mechanism and provide a proper transformation if the application needs to support user- or application-defined ClassAds. If it is a case of testing the functionality of a server, creating a simple job submission page can be a tedious and time consuming task. Frequent changes in development testing pages can be time consuming. This paper describes a set of components and methods that enable rapid development of a web application for Condor job submission through support of the ClassAd mechanism and Condor system. We designed and implemented visual Java Server Faces (JSF) custom components representing the ClassAd so that web application developers can easily develop their Condor job submission client applications simply by dragging and dropping our components on a web page designer and by setting the look and feel on the property window without much knowledge of ClassAd and/or Condor. Our component also provides flexibility by supporting the end user defined User ClassAd, and the web developer defined Application ClassAd, and by supporting a transformation from the Application ClassAd to the Submit ClassAd. The component can display the resource information via a table-style visual component. The expression editor and validator provide the ClassAd expression checking, which ensures the correctness of user input before submitting Condor jobs. We currently implemented ClassAd visual components and transformation APIs but the expression editor and validator is under development. In this paper, we focus on the client-side Condor job submission application development using our visual components and the end user and the client application developer defined ClassAd supports. II. TECHNOLOGIES AND RELATED WORKS

The Condor web components developed by this team consist of custom visual Java Server Faces (JSF) components and Java APIs. JSF is an application framework defined in JSR 127 [3,4] which is used to create web-based user interfaces. JSF contains

Authorized licensed use limited to: IEEE Xplore. Downloaded on February 10, 2009 at 11:12 from IEEE Xplore. Restrictions apply.

Rapid component-based development for Condor job submission, Jae-Woo Lee, David Braun, and Carol X. Song APIs for representing User Interface (UI) components, managing their state, handling events and input validation, and defining page navigation [5]. It also provides a JavaServer Pages (JSP) custom tag library for expressing a JavaServer Faces interface within a JSP page [5]. Our visual JSF components can be directly integrated into a popular Java Web IDE, NetBeans, especially the Visual Web Framework (VWF). The NetBeans VWF supports rapid development of web applications using a "What You See Is What You Get" (WYSIWYG) designer with UI components representing various HTML elements. The developers add our components to the component libraries of VWF in NetBeans and use them through the palette of UI components in the web page designer. There are other related works which provide rapid web application development. Grid Tag Libraries and Beans (GTLAB) [6] provide reusable components for rapid portlet development for using Grid services. GTLAB is used as a library for creating new portal and portlet application in Open Grid Computing Environments (OGCE) Portal and Gateway Toolkit [7]. It supports abundant grid tag libraries based on JSF for using Condor Web service, Birdbath. For example, its JSP custom tag can hide all the complexities for Condor job submission. GTLAB was also applied to build portlets for QuakeSim portal which simulates the earthquake process [8,9,10]. Rapid system [11] automatically generates a job submission portlet compatible with JSR 168 based on a job submission XML document. However, neither GTLAB nor Rapid supports a visual way to develop job submissions. Our contribution is introducing a new technology to wrap the complexity of the Condor job submission through the visual web components. Moreover, we provide flexibility on job submission pages by allowing user and developer defined ClassAds and transformation to the Submit ClassAds. The components that we are developing are based upon the JSF standard and we expect that they will be used to develop portlets and Condor based web JSF web

applications.

III. CONDOR WEB COMPONENT

Our Condor web components represent a ClassAd in the Condor system. A ClassAd is a set of uniquely named expressions and each named expression is called an attribute [12,13]. The web components are built on top of a wrapper API that provides a layer of isolation and interpretation to the server side Condor API. They allow the developers to build applications which interact with the wrapper API without the developer needing to know the server side implementation. In addition, the wrapper API provides a mechanism to associate the ClassAd definition meta data. This information is used to determine the kind of editor to use or provide tool tip help. We categorize these definitions into five types: User, Application, Site, Core, and Submit. This categorization is based on the use case and by who provides the definitions. User ClassAds are defined dynamically by the user at

2

Fig. 1. ClassAd definition editor. Web developers can define Application ClassAds for their web application and generate an XML definition file using our ClassAd definition editor

runtime. The use case for this type of ClassAd is for a more general web application where users can specify their own jobs. Application ClassAds are specified by the web developers and can be specific to the domain of the application. The Site and Core are ClassAds that are defined by the backend Condor implementation. An example of a Core ClassAd would be JobUniverse which is an internal Condor representation for the Universe ClassAd. An example of a Site specific ClassAd might be the name of the cluster on which the job should be specifically directed. A Submit ClassAd is the output ClassAd ofthe client application. This is transmitted to the server side on the end user's job submission, combined with the Site ClassAd and the Core ClassAd on the server side, and transformed finally to the Condor ClassAd for job submission to the Condor Server. The definitions ofthese ClassAds are visible to the developer to either propagate to the user or provide a transformation to generate the required attribute information. Developers construct a set of definitions that will be used within their project and the corresponding transforms that will produce the correct output. Once the definitions are created, the web component can be employed to provide a user interface to the application. We provide a ClassAd definition editor as shown in Fig. 1 so that the web developers can easily define their own Application ClassAd and generate an XML definition file. This

End User

User

Submit

ClassAd

ClassAd

App

Application ClassAd

|t Submit

4

ClassAd

Transformer]

Condor

Bidath ClassAd onr Schedd Web Service Core & Site ClassAd

client server:: Fig. 2. Overall architecture of the component deployment. ClassAdTuple, ClassAdTable, are UI components. Transformer is APIs to execute transformation of different type of ClassAds.

Authorized licensed use limited to: IEEE Xplore. Downloaded on February 10, 2009 at 11:12 from IEEE Xplore. Restrictions apply.

Rapid component-based development for Condor job submission, Jae-Woo Lee, David Braun, and Carol X. Song

file is loaded when the web application starts and provides information about the application defined ClassAd at runtime through linking to our visual web components. A. Overall Architecture Fig. 2 shows the overall architecture of the Condor web components and the flow for different types of ClassAd data. The Condor web components contain three UI components, a tuple style component named ClassAdTuple, a table style component, ClassAdTable and an expression editor, ExpressionEditor. The components also have a library for transforming an Application type to a Submit type. A ClassAdTuple represents an attribute of an Application ClassAd. A set of ClassAdTuples is an Application ClassAd and fed into the Transform API. If the developer has a template set of attributes, s/he can use a ClassAdTable instead of using ClassAdTuples. End users can add user defined attributes in the web page via a ClassAdTable. An ExpressionEditor is used to set an expression for the value of an attribute and to verify the legality of it. When a User ClassAd is submitted by the end user, the submitted ClassAd goes through the Transform API and this component generates a valid Submit ClassAd. Finally, the Condor web application submits this to the Birdbath web service [14]. Birdbath on the server-side combines the Submit information with Core and Site ClassAd and submits them to the Condor job scheduler. In this paper, we only focus on the client side components and related ClassAds. B. ClassAdTuple - Tuple Style ClassAd Component A ClassAdTuple component represents an attribute of an Application ClassAd. It has some properties the developer can specify. A ClassAd name, type and value are some examples of important properties that need to be set. In addition to these basic properties, the tuple can reference a set of ClassAd definitions. They provide the additional metadata to the tuple component such that at run time, the correct editor and tooltip information will be provided. The developer can change the style of the component as well. Using the style property the developers can specify the location of the component on the web page and change the view of the component such as font, size, color, etc. Web developers can easily set property values with the component property sheet provided by the NetBeans UI designer. Additionally the web developer can use the JSF expression language binding facility to retrieve or set data from

managed beans.

C. ClassAdTable - Table Style ClassAd Component A ClassAdTable is a container for ClassAdTuples displayed in table style. The developer can use the ClassAdTable by binding the ClassAd property to the ClassAd object which extends a list class and holds a list of ClassAd objects. The main usage of this component is to provide the developer with a UI component that is suited for displaying dynamically changing list of ClassAds. These lists could be used to develop a template for less application specific uses, user defined

3



Fig. 3. An example of a transformation mapping file. We provide APIs to load and manipulate the mapping information from an XML file. The mapping is from src to target. A transformer property specifies the transformation handler class. A nextTransformer indicates that the mapping includes a sequence of transformation. A hasMacro property means the value of the attribute can include a macro.

ClassAds, or displaying system ClassAd information. When a set of definitions is specified, each tuple will automatically use this definition set to display the appropriate editor and tooltip information. D. ClassAd Transform API A ClassAd Transform API gives developers the flexibility of using their own Application ClassAd or handling user defined ClassAd. To submit a job to a Condor server, the web application should submit a Submit ClassAd which follows exactly the definitions of the server side implementation. But some attributes' names of the Submit ClassAd may not be intuitive to the end users, so the web developer may want to use terminology which is more user-friendly when they define an Application ClassAd. They may also want to use a String type as

the value of the ClassAd attributes instead of numbers. The ClassAd Transform API solves these problems. The web developer defines a mapping of the attributes between an Application type and a Submit type or between a User type and a Submit type. The API can import the mapping information from an XML file but the developers can use any other way to load the mapping data and can add the mapping rules to the Transform API objects. Fig. 3 gives an example of transformation mapping file. A Transform API currently supports three different types of transformation - a Rename transform, a Translate transform and a Macro expansion transform. A Rename transform just replaces an attribute name of an Application type with the corresponding name of a Submit type. A Translate transform changes the String value of an attribute in a ClassAd to the number value of a ClassAd which is used on the Submit type. A Macro expansion transform supports the macro expansion in the value of a ClassAd attribute. This API can also support a sequence of the transforms, for example- first applying Rename transform and later Translate transform. By solving these complex mapping problems, this API relieves web developers

Authorized licensed use limited to: IEEE Xplore. Downloaded on February 10, 2009 at 11:12 from IEEE Xplore. Restrictions apply.

Rapid component-based development for Condor job submission, Jae-Woo Lee, David Braun, and Carol X. Song

4

and end users from the difficulty of creating job submission. E. ClassAd ExpressionEditor and Validator An attribute of a ClassAd is an expression which the Condor system defines. To decide the validity of the expression, the ClassAd should be submitted to the Condor server. The Condor system will return an error when it finds a problem on the

expression of the Condor ClassAd. The main purpose of the ClassAd expression editor and validator is to check the correctness ofthe expression before the job submission. This can improve the accuracy of the 1. Drag and drop ClassAdTuple components to the job submission page and set properties for each ClassAdTuple

3. Drag and drop ExpressionEditor component to the expression editor page and set properties for ExpressionEditor

2. Drag and drop other necessary components like a submit button to complete the submission page

4. Drag and drop other components to complete the expression editor page

5. Set page flow for linking the button on ClassAdTuple to the expression editor page

6. Define Transformation mapping table with an XML mapping file and add codes to load this mapping to the Web application 7. Use the mapping information and transform APIs to transform Application ClassAd to Submit ClassAd on submission source code

Fig. 4. The procedure to develop a job submission page using ClassAdTuple, ExpressionEditor and Transform APIs.

Application ClassAd and save the Condor system from unnecessary expression checking at the same time. The end users can also save time by checking partial expressions which they are editing without job submission. IV. WEB APPLICATION DEVELOPMENT USING COMPONENTS This section talks about how to develop a web application using our components. We provide the components in the form of a complib file. The developers can add the component to the NetBeans IDE by adding this file to the component libraries. Once it is added, the component is available through the palette for the design panel. Fig. 4 shows an example flow for developing a job submission page using our components. In this section, we give three examples of how to use our components and APIs for the Condor web client application. A. Creating an Application ClassAd page As seen in Fig. 5, by adding a few ClassAdTuples and a submit button, the web developer can easily create an Application ClassAd page which is a Condor job submission page. After creating a new web application project in NetBeans, the developer simply drags and drops as many of the ClassAdTuples as the developer wants to add from the palette. Then s/he can specify the properties such as classAdName, classAdValue, and islnputEditable. The ClassAd definition can be bound with existing ClassAdDef object to show the usage and restriction of the attribute as a tooltip. The developer can

also bind the component with an existing ClassAd object which is declared in other source codes. On submission, s/he can extract the value of each ClassAd by invoking the method, getClassAd(. The getClassAd( method returns ClassAd object which the component contains and s/he can put this into the ClassAds objects and submit this information. Developers who do not use IDE tools like NetBeans can still use the component through JSP custom tags from the text editor, or they can use the component directly at the bean files for the JSF application. B. Displaying Server Information and Resources Using the ClassAdTable, developers can create a web page displaying the job status on the server, or other server resource information. Since these ClassAds are dynamic in nature, the application simply has to retrieve a ClassAd list object to display the information. The procedure to develop this web page is similar to creating an Application ClassAd page. The developers can drag and drop a ClassAdTable component from the palette to the designer, set properties through binding to a ClassAd list object and a ClassAd definition object.

C. Custom Editors Developers can also specify a custom ClassAd editor to use. On the right side of the ClassAdTuple there is a small button that when clicked will call an action handler. This handler could be used to build redirect functionality or pop up a custom editor for the user. The users can specify attributes of the ClassAd in this editor and verify if their input data follows the Condor expression specifications before the submission. V. FUTURE WORK AND CONCLUSION In this paper, we introduced a new technology to develop Condor job submission web application by using our visual JSF components. Our components also provide flexibility by supporting the end user defined User ClassAd, and the web developer defined Application ClassAd, and by supporting a transformation from the custom ClassAd to the Submit ClassAd. Our work has demonstrated the potentials of our components for

Authorized licensed use limited to: IEEE Xplore. Downloaded on February 10, 2009 at 11:12 from IEEE Xplore. Restrictions apply.

Rapid component-based development for Condor job submission, Jae-Woo Lee, David Braun, and Carol X. Song

the rapid development of Condor job submission web user interfaces. Currently we implemented the tuple style, the table style visual web component, and the ClassAd transform APIs. We are implementing the ExpressionEditor and the validator now. We will use the security functionality of an additional web service layer in the server side for the security of our components when the job submission pages with our components access to the Condor Birdbath. This layer of protection will allow for user level attribute controls and allow aggregate interactions with more than one Condor Schedd. We also plan to apply the components to one of our projects as a case study. The potential users ofthese tools are part ofthe Condor submission portal that is in development. The initial phase of the portal will allow for dynamically generated web applications pages that make use of our components. Users will be guided through a series of pages to construct a Condor submission. The ClassAd util package and JSF components package will be released under a Condor tools project that will either be hosted on sourceforge or google. REFERENCES [1] What is Condor?,

http://www.cs.wisc.edu/condor/description.htm1

[2] Douglas Thain, Todd Tannenbaum, Miron Livny: Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience 17(2-4): 323-356 (2005) [3] A first look at JavaServer Faces, http://www.javaworld.com/javaworld/jw-I 1-2002/jw-I 129-jsf.html [4] Java Service Request 127: JavaServer Faces,

http://www.jcp.org/en/jsr/detail?id=127

[5] JavaServer Faces Technology Overview,

http://java.sun.com/javaee/javaserverfaces/overview.html

[6] Mehmet A. Nacar, Marlon E. Pierce and Geoffrey C. Fox, "GTLAB: Grid Tag Libraries Supporting Workflows within Science Gateways", Proceedings of the Third International Conference on Semantics, Knowledge and Grid, October 2007 [7] The Open Grid Computing Environments Portal and Gateway Toolkit,

http://www.collab-ogce.org

[8] Mehmet A. Nacar, Marlon E. Pierce, Andrea Donnellan, and Geoffrey C. Fox, "Building QuakeSim Portlets with GTLAB", International Workshop on Grid Computing Environments, November 2007 [9] Andrea Donnellan, John Rundle, Geoffrey Fox Dennis McLeod, Lisa Grant, Terry Tullis, Marlon Pierce, Jay Parker, Greg Lyzenga, "QuakeSim and the Solid Earth Research Virtual Observatory", Special Issue of Pure and Applied Geophysics (PAGEOPH) for Beijing ACES, July 2004 [10] QuakeSim portal, http://quakesim.jpl.nasa.gov [11] Jos Koetsier, Srihathai Prammanee, "Manual: Writing a portlet using the Rapid tool", http://research.nesc.ac.uk/rapid, September 5, 2008 [12] Condor Manual,

http://www.cs.wisc.edu/condor/manual/v7.0

[13] Rajesh Raman, Miron Livny, Marvin H. Solomon: Matchmaking: An extensible framework for distributed resource management. Cluster Computing 2(2): 129-138 (1999) [14] C. Chapman, C. Goonatilake, W. Emmerich, M. Farrellee, T. Tannenbaum, M. Livny, M. Calleja, and M. Dove, "Condor BirdBath: Web Service interfaces to Condor", Proceedings of the 2005 UK e-Science All Hands Meeting, September 2005 [15] P. Smith. Implementing a Central Quill Database in a Large Condor Installation. CondorWeek 2008, Madison, WI, April 30 - May 2, 2008.

http://www.cs.wisc.edu/condor/CondorWeek2008/wednesday_condor.ht ml.

Authorized licensed use limited to: IEEE Xplore. Downloaded on February 10, 2009 at 11:12 from IEEE Xplore. Restrictions apply.

5