The Management of ClassAds in RDBMS and XML Native Storage Systems Youngsang Shin
Master’s Project Report
Spring 2004 Project Advisor Professor Jeffrey F. Naughton
The Management of ClassAds in RDBMS and XML Native Storage Systems Youngsang Shin Computer Sciences Department University of Wisconsin – Madison Madison, WI 53706
[email protected] May 10, 2004
1 Introduction ClassAd is a semi-structured data represented by ClassAd language [6] which has been developed in the Condor project [9]. In Condor, most metadata and logs are denoted in this form of data. Currently this data can be handled through the ClassAd library, which is designed for handling a main-memory resident data structure, and optimized for a relatively small amount of data and several functions. However, as metadata managed in Condor is getting larger and more various complex querying functions are needed, the library should be able to handle efficiently data in a disk, and be augmented with the hard coding for various additional querying functions. To effectively address these problems, RDBMS could be a serious alternative. Furthermore, since this language is semi-structured and easily translated into XML(eXtendible Markup Language) [10], a XML native storage system may be another possible substitute. This project is focused on exploring relational and XML representations for ClassAd data, and studying the comparison between them and the current ClassAd library in Condor. There are two schemes, vertical and horizontal ones to store irregular data in a relational database. Especially in e-commerce applications, a vertical scheme is widely adopted [1]. However, there is a tradeoff between horizontal and vertical schemes in RDBMS due to several factors such as properties of the dataset, and how many attributes are retrieved in the queries [5]. To study the performance of two representations for managing ClassAd data, we present two schemas respectively for them and make a comparison between those two representations. In addition, an XML representation for ClassAd data is stored and queried in a XML native storage system. Finally, we introduce an application, jqmond that manages ClassAd data in RDBMS. jqmond stores the job ClassAds of a job queue managed by schedd in RDBMS, and provides a retrieval service to condor_q utility [12].
2 Vertical Scheme Many e-commerce systems usually adopt a vertical scheme for storing objects in a relational database to address several problems such as schema evolution, sparsity, large number of columns, and performance [1]. When storing ClassAd in RDBMS, the similar problems happen, too: ClassAd does not have any static schema since it is described by the semi-structured, ClassAd language. Therefore, each ClassAd can have different number and/or type attributes. To store this data in a table, the table should be very wide. This kind of schema may cause the similar problems met by e-commerce systems. cid
did attr_name Fig 1. Vertical Schema for ClassAd
1
attr_val
In Fig 1, a 4-ary vertical representation is illustrated. cid stores ClassAd id, and did has DeWey id for keeping a nested structure information [2]. attr_name has the name of attribute, and attr_val has its value. To get the best querying performance, all the attributes in this table are indexed. Although the attributes of ClassAd can be various type values such as integer, float, string, expression, record, and list, all the attributes are only classified to two data types, string and number in this study. Thus, only two tables are created for string and number types; all the other types except a number type are stored in a table for a string type.
3 Horizontal Scheme When there are a small number of attributes in a ClassAd data set, a horizontal scheme might work very efficiently although it has a performance problem with schema evolution. Therefore, it is beneficial to observe the tradeoff between vertical and horizontal schemes for storing ClassAds. In this section, a horizontal scheme is presented to store ClassAd in a table. cid
attribute1 attribute2 Fig 2. Horizontal Schema for ClassAd
…
In Fig 2, the first attribute, cid is used to store ClassAd id and DeWey id together. After that, attributes as many as required for representing all the attributes of stored ClassAds are needed. For performance purpose, cid and other frequently queried attributes are indexed.
4 XML implementation Since a generic XML syntax is already defined for the ClassAd language, in this study it is used to store ClassAd without any modification in a XML native storage system. However, this syntax has an inefficiency problem for indexing. For example, the current generic XML representation of an example ClassAd (a) is (b) in Fig 3. [ MyType = “Machine”; TargetType = “Job”; Name=”a.b.c.d”; Rank = 0.0; Disk = 998776; ]
(a) Native Syntax
Machine Job 0.0 998776
Machine Job a.b.c.d 0.0 998776
(b) XML Syntax
(c) Alternative XML Syntax
Fig 3. XML syntax for ClassAd language Because all the attributes are just node a in this case, only one index can be built just for node a. However, if ClassAd is represented like (c), an index could be built for each attribute. It may significantly affect on
2
performance. This problem could be addressed by a translation from the current generic form to an application specific representation. XSLT [11] is one of methods that can be used to do it.
5 Experiment In this section, we present the performance study of RDBMS, XML native storage, and ClassAd library implementations. The experiment was performed under an environment with Pentium III 700Mhz, 256Mbytes RAM, RedHat Linux 7.2. For RDBMS and XML native storage systems, we adopted open source systems, PostgreSQL 7.4, and Berkeley DB XML 1.1.0 respectively, which includes Xerces 2.3 and Libpathan 1.2 R1. In PostgreSQL, a buffer size was set to 512k bytes. Sample ClassAds, machine descriptions were obtained as a snapshot in the CS Condor pool – there were 1035 CPUs listed, and the average number of attributes of each ClassAd was 95. We performed two tests for querying and updating in RDBMS, a XML native storage system, and the ClassAd collection library. For querying test, three frequent queries were asked. The first one was aggregation, COUNT. It actually has two conjunctive predicates. Second query is just one which has a predicate and no projection. The last one is more complex: it has four conjunctive predicates, and a projection which asks one attribute. All the attributes queried were indexed. Query 1
Aggregation: Find the number of available machines: predicates State == “Unclaimed” && Activity == “Idle”
Query 2
Simple Selection: Find ClassAd whose Name is
[email protected]
Query 3
Complex Selection and Projection: Find the Name attribute of ClassAds which meet conjunctive predicates: Arch == “INTEL” && OpSys = “LINUX” && Disk >= 10 && Memory >= 0.009765625 Fig 4. Queries Tested
For updating test, the value update of an attribute, and an attribute and a ClassAd object insertions were tested. Most current applications using the ClassAd collection library find a ClassAd object with a name by hashing. However, for a fair comparison in this study, we do not use hashing to find ClassAd objects to be updated, but find them with some predicates. This causes some deficiency here. To do this in the ClassAd collection library, we must create a sub-view, and query against it. In this study, all the updated attributes were indexed, and a sub-view was created in order to reflect the performance loss due to them on the result. The query test results are shown in Fig 5 and 6. Fig 5 is for the horizontal and vertical schemes, and the ClassAd library, and Fig 6 is for the XML representation. As shown in Fig 6, the times of executing the queries against the XML representation seem to be much worse. This result may be contributed by the immature of XML native storage systems and the current generic XML syntax for the ClassAd language that is not efficient for indexing purpose as illustrated.
3
Horizontal
Vertical
ClassAd lib 45000
400
40000
350
35000
Execution Time (ms)
Execution Time (ms)
300
250
200
150
30000 25000 20000 15000
100
10000
50
5000 0
0
Q1
Q2
Q1
Q3
Fig 5. Querying Test Result of RDBMS and ClassAd library
Q2
Q3
Fig 6. Querying Test Result of XML Native Storage System
350
Execution Time (ms)
300
250
200
Get Result Create Sub-View
150
100
50
0
Q1
Q2
Q3
Fig 7. Querying Time in the ClassAd Collection Library Fig 5 shows the performance results of the three queries in RDBMS and the ClassAd library. For the first query, the horizontal scheme showed the best performance because the first query was aggregation, COUNT, and the number of selected rows was the least in that scheme. In this case, the ClassAd library does not support any aggregation functions, so that the time of executing a selection operation was measured. For the second query, the vertical scheme was best. Since the query had only one predicate and no projection, the horizontal scheme might not have an advantage of projection during a query optimization. The third query had multiple predicates and a projection. Due to projection and multiple predicates, the horizontal scheme gave the best performance. The query execution times of the ClassAd library were the sum of creating a sub-view and querying against it. The current ClassAd collection library does not support on-the-fly querying API. Thus, a sub-view must be created to find some ClassAds that meet some predicates beforehand. In Fig 7, it is shown how long it took to create a sub-view. However, once a sub-view was created, the query execution time against that was so fast as shown since it was materialized when created. Thus, there is a tradeoff. In Fig 8, the results of attribute update test are shown. As shown in Fig 6, the execution times of queries for XML representation were relatively very slow, so its result was excluded in this test. The update was just to update one attribute of ClassAds that meet a predicate. Therefore, the vertical scheme was most fast. In this case, the execution time of ClassAd library was still the sum of creating a sub-view and updating the attributes of the result ClassAds. The updating time itself was actually one third of that of the vertical scheme.
4
Horizontal
140
ClassAd lib
120
Execution Time (ms)
120
Execution Time (ms)
Vertical
140
160
100
80
60
100
80
60
40
40
20
20
0
0
Horizontal
Vertical
Insertion of Attribute
ClassAd lib
Fig 8. Execution Time of Update of Attribute
Insertion of ClassAd
Fig 9. Execution Time of Insertion of Attribute and ClassAd Object
Fig 9 presents the performance result of inserting an attribute and a ClassAd entry. In case of inserting an attribute, the vertical scheme was most fast because its operation was just one row insertion. However, for the horizontal scheme, the schema of table must be changed before an actual insertion happens. Schema Change 200
Attribute Insertion
140
120
Execution Time (ms)
Execution Time (ms)
150
100
100
80
60
40
50
20
0
4 52 8 53 2 53 6 54 0 54 4 54 8 55 2 55 6 56 0 56 4 56 8 57 2 57 6 58 0 58 4 58 8 59 2 59 6 60 0
52
2 51 6
52
4 50
50
25 49 73 97 121 145 169 193 217 241 265 289 313 337 361 385 409 433 457 481 505 529 553 577 601 625 649 673 697 721 745 769 793
51
0 50
1
8
0 0
Iteration #
Iteration #
Fig 10. Execution Time of Attribute Insertions (Schema Change and Attribute Insertion) in the Horizontal Scheme
Fig 11. The times for Schema Change and Attribute Insertion in the Horizontal Scheme
Furthermore, the time needed for insertion increased almost linearly as the number of attributes rose as in Fig 10. In Fig 11, the time to change a schema increased just slightly, but the insertion time rose relatively steeply. When inserting a ClassAd object, the vertical scheme needs much more time since the insertions as many as the number of attributes are needed. Furthermore, since every column was indexed for a fast querying purpose, more time was required to update the indexes also. In case of the ClassAd library, the insertion was very fast because it did not need to create a sub-view, and to write it into a disk
6 jqmond: RDBMS-backed ClassAd Application with a Vertical Scheme In this section, we introduce jqmond (Job Queue MONitor Daemon) which is an application that manages job ClassAds through RDBMS. It fetches job ClassAds in a job queue maintained by schedd, which is a scheduler in a job submitting machine and serves condor_q which is used as a command to query the status of a job queue in Condor. Currently schedd acts many roles including service to condor_q. This RDBMSpowered application can reduce this heavy load of schedd and provide more extended functions such as more complex querying and efficient retrieval against a large number of ClassAds.
5
Fig 12 shows how schedd logs job ClassAds in its job queue, and how jqmond behaves. schedd maintains the job list of the job queue as a collection of job classads in its memory, and every operation performed to this collection is logged into a job_queue.log file, which is regularly compressed by schedd. A job ClassAd removed from the job queue for some reasons such as finished running and any other error, is added to a history file as soon as it is eliminated from the job queue. This history file does not shrink, but only grows.
condor_q
schedd collection of ClassAds
jqmond
Polling
job_queue. log PostgreSQL
history Database
Fig 12. Overview of jqmond jqmond periodically probes a job_queue.log file that is logged by schedd, and stores the acquired job ClassAds into RDBMS. Polling can cause inconsistency between the current job queue and the job queue database at a certain time. However, such probing is the only way to do it since the development of this application started with the assumption that there must not be any change in the current Condor system. At each polling time, jqmond incrementally updates the database with the difference between the last and current job_queue.log files. The ClassAd language implementation applied in the current Condor system is not by the latest specification. It does not include some new features such as nested structure. Therefore, the database schema of jqmond does not consider a field for nested structure information. Furthermore, it does not translate between horizontal and internal vertical representations to avoid unnecessary self join operations unlike other e-commerce applications. In Condor, condor_q utility is used to retrieve the status of jobs in a job queue. When it is used to query the jobs of a job queue in a local machine or in a remote machine, it contacts to schedd. jqmond can provide this service to condor_q instead of schedd. Current schedd naively processes a constraint within a simple loop, but jqmond can handle this query through an efficient RDBMS query processor. When such a
6
constraint is more complex, or the number of job ClassAds is very large, jqmond can fully take an advantage of efficient query optimizing in RDBMS. Current jqmond is fetching only job ClassAds in a local machine. It can be globally scaled up to manage all the job ClassAds in a Condor pool. It also handles only a job_queue.log file at this moment. It will be extended to process a history file. In the appendix, the design detail design of jqmond is shown.
7 Conclusion and Discussion The execution times of RDBMS and XML implementations could not beat those of the current ClassAd library in most cases since the latter treats only a main memory resident data structure. Furthermore, the library is optimized for only specific functions such as sub-view. However, this result is probably applicable only for relatively small sized data and restricted functions. If a volume of data to be managed gets enormous, and/or more various complex querying functions are needed, RDBMS can definitely be an imperative alternative. RDBMS can provide better functionalities, extendibility, and flexibility like the followings. Firstly, the current ClassAd library provides only a sub-view, materialized view for querying against a ClassAd collection. This facility enables a query to be executed very fast once it is computed. However, it takes much more time to compute initially a materialized view than an on-the-fly query execution in RDBMS. Therefore, in the case that a sub-view was not created for a required query in the ClassAd library in advance, RDBMS can give much better performance. Secondly, RDBMS can give a more powerful querying facility. For the ClassAd library, much more effort to implement such extended querying functionalities is needed with hard-coding. Thirdly, RDBMS can efficiently manage a large volume of data in a disk. However, the current ClassAd library handles only a memory-resident data structure, so that it depends on OS cache management when the whole content of a ClassAd collection cannot be hold in a physical main memory. Usually, the buffer management in RDBMS gives a better performance for manipulating a data. The last benefit is that RDBMS is so matured that it can already provide very numerous features that the current ClassAd library does not support or not treat efficiently. For example, a Client-Server version ClassAd library could be easily implemented with RDBMS. Thus, various needed functionalities can be easily realized. In this study, the horizontal schema gave a little better query performance than the vertical scheme because the number of all the attributes was not huge, and the sample data was not actually sparse. Therefore, if the schema of ClassAd in a certain application is static and its schema information can be provided, a horizontal representation will be the best choice in such a limited case. However, the vertical scheme efficiently supports dynamic schema changes and thousands of attributes, and economical storage usage for sparse data; thus, it is more suitable for representing a ClassAd data. More comprehensive study about the tradeoff between these two schemes when storing e-commerce data in RDBMS can be found in [5]. We introduced jqmond, an RDBMS-backed ClassAd application with a vertical scheme. This application will be extended to cover more complex queries, new ClassAd features such as nested structure, and scaled up in Condor. More performance study on these extended functions will show the effect of RDBMS on managing ClassAd data. Furthermore, since XML native storage systems usually have some advantages over path quering against nested structure data, its application might also provide motivating results. In this case, more performance gain can be obtained from a translation from the current generic XML syntax to other forms for the ClassAd language.
7
References [1] Rakesh Agrawal, Amit Somani, Yirong Xu, Storage and Querying of E-Commerce Data, Proceedings of the 27th VLDB Conference, 2001. [2] Igor Tatarinov, Stratis D. Viglas, Kevin Beyer, Jayavel Shanmugasundaram, Eugene Shekita, Chun Zhang, Storing and Querying Ordered XML Using a Relational Database System, Proceedings of ACM SIGMOD Conference, 2002. [3] Jayavel Shanmugasundaram, et al., A General Technique for Querying XML Documents using a Relational Database System, SIGMOD Record, 2001. [4] D. Florescu, D. Kossmann, XML Data Using an RDBMS, IEEE Data Engineering Bulletin, 22(3):27-34, 1999. [5] Rajasekar Krishnamurthy, Jennifer Beckham, Jeffrey Naughton, The Tradeoff between Horizontal and Vertical Representations of Sparse Data Sets, 2003. [6] http://www.cs.wisc.edu/condor/classad/. [7] Steve Sarette, Getting Started with DBXML (C++ Edition), SleepyCat Software, 2003. [8] Rajesh Raman, Matchmaking Framework for Distributed Resource Management, Ph.D. Dissertation, Computer Science Department, University of Wisconsin – Madison, 2000. [9] Condor Project, http://www.cs.wisc.edu/condor [10] XML, http://www.xml.org [11] XSLT, http://www.w3.org/TR/xslt [12] condor_q, http://www.cs.wisc.edu/condor/manual/v6.6/condor_q.html
8
Appendix – Design of jqmond (Job Queue MONitor Daemon) 1 Overview Collection of Job ClassAds
condor_q
schedd Job Queue
jqmond Logging Polling Storing / Updating / Querying job_queue. log
PostgreSQL
history
Database
Fig 1. Behavioral Overview jqmond fetches the job queue and history information maintained by schedd, stores them in RDBMS, and provides querying functions against them, which is issued by condor_q utility in Condor.
2 ClassAd Data: job_queue.log and history files 2.1 job_queue.log -
-
Currently submitted but not finished job ClassAds are in the job queue. Any change, insertion, and deletion in the job queue are logged in this file. Each job in the job queue of schedd has two ads: the proc ad and the cluster ad. All jobs in the same cluster share the same cluster ad. Each job has its own proc ad. Attributes that appear in the cluster ad apply to all jobs in that cluster. Attributes that appear in a proc ad apply only to that job. If the same attribute is in both, the proc ad one dominates. This saves a lot of space in the queue, as most of the attributes are identical within a cluster. Format + Text log file + Command{_Key_Attribute_Value}
9
-
+ Command Types - 101: New ClassAd - 102: Destroy ClassAd - 103: Set Attribute - 104: Delete Attribute - 105: Begin Transaction - 106: End Transaction + Key - Cluster Id.[-1 | 0] + -1 : cluster ad + 0 : proc ad Example 101 103 105 103 101 101 103 103 103 103 103 . . .
0.0 Job Machine 0.0 NextClusterNum 11 0.0 NextClusterNum 12 011.-1 Job Machine 11.0 Job Machine 011.-1 ClusterId 11 011.-1 QDate 1077859390 011.-1 CompletionDate 0 011.-1 User "
[email protected]" 011.-1 Owner "ysshin"
2.2 history -
Job ClassAd is appended to this file as soon as it is removed from the job queue. Format + Text log File + Each ClassAd are sperated by *** Example MyType = "Job" TargetType = "Machine" ClusterId = 1 QDate = 1075353552 CompletionDate = 0 Owner = "ysshin" RemoteWallClockTime = 0.000000 LocalUserCpu = 0.000000 LocalSysCpu = 0.000000 RemoteUserCpu = 0.000000 RemoteSysCpu = 0.000000 ExitStatus = 0 NumCkpts = 0 NumRestarts = 0 NumSystemHolds = 0 CommittedTime = 0 TotalSuspensions = 0 LastSuspensionTime = 0 CumulativeSuspensionTime = 0 ExitBySignal = FALSE CondorVersion = "$CondorVersion: 6.6.1 Jan 26 2004 PRE-RELEASE $" CondorPlatform = "$CondorPlatform: I386-LINUX-RH9 $" RootDir = "/" Iwd = "/scratch/ysshin/CONDOR_WORKSPACE/test" . . .
10
3 How to Collect Data There are two possible ways: polling and sending events
3.1 Polling -
Periodically probing a job_queue.log and a history files. Pros: not necessary to modify the current source code Cons: there might be some inconsistency sometimes, and more overhead due to regularly polling
3.2 Sending Events -
Sending an event to jqmond at certain times, for example, when schedd sends a job list to a collector, or it calls the functions of the ClassAd collection library. Pros: Less overhead, and guarantees consistency between the job ad database and the job queue Cons: Some modification to the current source code is needed. However, it does not change any logic in schedd. It will be just a simple minor addition.
In this project, polling method is used to avoid any modification in the current Condor code.
4 DB Schema Seven tables: ProcAds, ClusterAds, History.
4.1 Process Ads: ProcAds_Str, ProcAds_Num cid pid attr val
variable string variable string variable string variable string or number
Index: pid, cid, attr, val, attr & val SQL CREATE TABLE ProcAds_{Num|Str} ( cid text, pid text, attr text, val {text|double precision} primary key (cid, pid, attr) ) CREATE CREATE CREATE CREATE CREATE
INDEX INDEX INDEX INDEX INDEX
ProcAds_{Num|Str}_i_pid ON ProcAds_{Num|Str} (cid); ProcAds_{Num|Str}_i_cid ON ProcAds_{Num|Str} (pid); ProcAds_{Num|Str}_i_attr ON ProcAds_{Num|Str} (attr); ProcAds_{Num|Str}_i_val ON ProcAds_{Num|Str} (val); ProcAds_{Num|Str}_i_attr_val ON ProcAds_{Num|Str} (attr, val);
4.2 Cluster Ads: ClusterAds_Str, ClusterAds_Num cid attr val
variable string variable string variable string or number
Index: cid, attr, val, attr & val
11
SQL CREATE TABLE ClusterAds_{Num|Str} ( cid text, attr text, val {text|double precision} primary key (cid, attr) ) CREATE CREATE CREATE CREATE
INDEX INDEX INDEX INDEX
ClusterAds_{Num|Str}_i_cid ON ClusterAds_{Num|Str} (cid); ClusterAds_{Num|Str}_i_attr ON ClusterAds_{Num|Str} (attr); ClusterAds_{Num|Str}_i_val ON ClusterAds_{Num|Str} (val); ClusterAds_{Num|Str}_i_attr_val ON ClusterAds_{Num|Str} (attr, val);
4.3 History: History_Str, History_Num cid pid attr val
variable string variable string variable string variable string or number
Index: pid, cid, attr, val, attr & val SQL CREATE TABLE History_{Num|Str} ( cid text, pid text, attr text, val {text|double precision} primary key (cid, pid, attr) ) CREATE CREATE CREATE CREATE CREATE
INDEX INDEX INDEX INDEX INDEX
History_{Num|Str}_i_pid ON History_{Num|Str} (cid); History_{Num|Str}_i_cid ON History_{Num|Str} (pid); History_{Num|Str}_i_attr ON History_{Num|Str} (attr); History_{Num|Str}_i_val ON History_{Num|Str} (val); History_{Num|Str}_i_attr_val ON History_{Num|Str} (attr, val);
4.4 Polling Information: JobQueuePollingInfo last_file_mtime last_file_size last_next_cmd_offset last_cmd_offset last_cmd_type last_cmd_key last_cmd_mytype last_cmd_targettype last_cmd_name last_cmd_value
number (BIGINT) number (BIGINT) number (BIGINT) number (BIGINT) number (SMALLINT) variable string (text) variable string (text) variable string (text) variable string (text) variable string (text)
SQL CREATE TABLE JobQueuePollingInfo ( last_file_mtime BIGINT, last_file_size BIGINT, last_next_cmd_offset BIGINT, last_cmd_offset BIGINT, last_cmd_type SMALLINT, last_cmd_key text, last_cmd_mytype text, last_cmd_targettype text,
12
last_cmd_name last_cmd_value
text, text
)
5 Polling start
Changed
Check the file timestamp change
No Change
Check the file size
Increaed
No Init Timestamp No Change Check Last Command
Get Delta
Same Check Last Command
Different end
Decreased Different Same
Initial Status
Addition Status
Error Status*
Compression Status
Process the Delta
Save a timestamp
Read the whole file & process it
Delete the current DB
*Error Status: Currently Unresolved, but could be refined into more states
Fig 2. Probing of a job_queue.log file
6 Building History A history file is usually very large because it is an archive of job ads removed from a job queue. Therefore, it is not efficient to use the same polling scheme for a job_queue.log file. History relation can be built by inserting the union of process ClassAds and cluster ClassAds when a command 102 (Destroy ClassAd) is met. However, if this method is just adopted, there is no information to track what happens in schedd during jqmond’s crashing time for some reasons. Therefore, some tracking method for a history file should be devised. This is currently not implemented.
13
7 Recovery A recovery scheme follows the polling scheme.
8 Service to condor_q condor_q utility program requests the list of job ClassAds in a job queue to schedd over a socket communication. jqmond provides the same functions through the same protocol. Therefore, jqmond can replace schedd for this function.
14
