Proc. ACM SIGMOD Int. Conf. on Management of Data (SIGMOD 2004), Paris, France, pages 683-694.
Joining Interval Data in Relational Databases Jost Enderle
Matthias Hampel
Thomas Seidl
RWTH Aachen University, Germany Department of Computer Science
[email protected]
University of Munich, Germany Institute for Computer Science
[email protected]
RWTH Aachen University, Germany Department of Computer Science
[email protected]
ABSTRACT The increasing use of temporal and spatial data in presentday relational systems necessitates an efficient support of joins on interval-valued attributes. Standard join algorithms do not support those data types adequately, whereas special approaches for interval joins usually require an augmentation of the internal access methods which is not supported by existing relational systems. To overcome these problems we introduce new join algorithms for interval data. Based on the Relational Interval Tree, these algorithms can easily be implemented on top of any relational database system while providing excellent performance on joining intervals. As experimental results on an Oracle9i server show, the new techniques outperform existing relational methods for joining intervals significantly.
1. INTRODUCTION One of the most important data types needed by temporal and spatial applications is the interval type. Intervals are used, for instance, as transaction time and valid time ranges [34] [32] [8] or as line segments on a space-filling curve [13] [7]. Driven by the ongoing demand for the support of temporal and spatial data in relational databases, the SQL language has recently been extended with corresponding data types and operations. SQL:1999 [21] differentiates between intervals and chronological periods. An interval represents the duration of a period in time (and thus is a scalar value), whereas a chronological period is a compound object specified either as a pair of datetimes (starting and ending point of time) or as a starting datetime and an interval. The only operator currently defined on periods is overlaps which determines whether or not two periods overlap in time. SQL also provides corresponding types and operations for spatial applications within a separate application package [22]. In the following, we will focus our considerations on the data type Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SIGMOD 2004, June 13–18, 2004, Paris, France. Copyright 2004 ACM 1-58113-859-8/04/06 …$5.00.
period (represented by lower and upper points) in conjunction with the join operator overlaps. To create a table R with a key attribute theKey and a row attribute thePeriod representing a period with lower and upper points, we have to write the following DDL statement in SQL:1999: CREATE TABLE R ( theKey INT, thePeriod ROW (lower TIME, upper TIME)) Figure 1. Statement to create a table with period attribute
Given a second table with the same definition, the query in Figure 2 yields the (pairs of) key values of those rows in R and S which have overlapping time periods: SELECT R.theKey, S.theKey FROM R, S WHERE R.thePeriod OVERLAPS S.thePeriod Figure 2. Query with join operator overlaps
The join predicate of Figure 2 can equivalently be formulated as in Figure 3: … WHERE R.thePeriod.lower = S.thePeriod.lower Figure 3. Equivalent formulation of the join predicate
Unfortunately, join algorithms commonly found in relational databases do not support the evaluation of such queries very efficiently. That is because conventional algorithms usually are designed for the evaluation of equijoins on single attributes whereas in our case, multiple interdependent join predicates of the inequality type are involved. For example, a sort-merge join cannot directly be adapted for joining intervals as long as there is no appropriate sort order on the interval values. Index joins can only provide advantages if the index method is specially designed for interval data. In the presence of intervals with very different lengths, using a simple composite index on the two bounding points, either on (lower, upper) or (upper, lower), does not improve access times significantly.
To overcome these problems, several approaches for joining intervals have been presented so far. Some of them are considered in the next section. Unfortunately, these algorithms usually are based either on the augmentation of existing indexes or on the definition of new structures, which means that they provide hardly any support for an integration into existing systems. With the Relational Interval Tree (RI-tree) [25], an efficient access method has been proposed to process interval intersection queries on top of any existing relational database system. Instead of accessing raw disk blocks directly, data objects are managed by common built-in relational indexes following the paradigm of relational indexing [24]. In order to provide efficient interval joins in relational databases, we introduce new join algorithms using the structure of the Relational Interval Tree. Inheriting the easy implementation of the RI-Tree and the superior performance of built-in database indexes, these algorithms have major advantages over former approaches. As experimental results on an Oracle9i server show, the new techniques outperform existing relational methods for joining intervals significantly. The remainder of the paper is organized as follows: Section 2 surveys related work for processing joins on interval data. After recalling the Relational Interval Tree in Section 3, we present our new join algorithms in Section 4. Section 5 describes the results of our experimental evaluation, and the paper is concluded in Section 6.
2. RELATED WORK Several different approaches to provide efficient interval joins already exist in the literature, especially in the field of temporal applications [14]. So most of the presented techniques deal with the problem of temporal joins where two relations are joined on the basis of corresponding time intervals. Spatial join techniques are also applicable when confined to the one-dimensional case. In general, however, they do not behave as well as interval-specific algorithms. Particularly the long durations and high overlaps of intervals in many temporal applications induce severe performance problems. Further applications exist in the field of XML processing where intervals and corresponding operations can be used, e. g., to encode XML instance trees and to evaluate ancestor/descendant relationships in such trees.
2.1 Nested-Loop Algorithms Whereas easy to implement, nested-loop based join algorithms are often not competitive due to quadratic evaluation cost. By imposing certain restrictions on the relations to join, processing times can be improved. Gunadhi and Segev propose a temporal nested-loop based join algorithm called TJ-2 [15] that assumes the inner join re-
lation to be sorted. The sort ordering is based on an ascending primary order on the lower points and an ascending secondary order on the upper points of the temporal intervals. This permits to drop out of the inner loop before having read all the tuples of the inner relation. Rana and Fotouhi [33] also present some nested-loop variants for temporal join processing under the assumption that the smaller join relation fits entirely in memory. For most applications, this restriction is too severe.
2.2 Sort-Merge Algorithms Sort-merge based join algorithms require both relations to be sorted on the join attributes. So a reasonable sort order on the interval data has to be found. Gunadhi and Segev also developed a sort-merge version of their temporal join called TJ-1 [15]. In contrast to TJ-2, both relations are sorted on the lower and upper points of the intervals. Utilizing the additional sorting of the ‘outer’ relation and a special ‘begin-of-file’ pointer for the ‘inner’ relation, less records have to be read within the inner join loop. While a sort-merge join on scalar attributes has to read each of the relation records only once, here some records of the inner relation must be read several times. This results from the fact that intervals cannot be sorted in linear order. TJ-3, a slightly modified version of TJ-1 for append-only databases, is also presented in [15]. For such a database, both relations are sorted on the lower points only. Gunadhi and Segev propose another series of sort-merge algorithms for the evaluation of the temporal equijoin [15]. Here, two tuples from the joining relations qualify for concatenation if the non-time join attributes have the same values and their time intervals overlap. Leung and Muntz developed a set of sort-merge algorithms [26] to support different temporal join predicates such as overlaps and contains. The input relations are considered as sequences of records sorted on the insertion time. Only the tuples at the front of the sequences may be read and merged to compute the join. Pfoser and Jensen provide a further temporal sort-merge join algorithm for append-only databases [31]. Each join relation consists of a non-current and a current partition where tuples of the non-current partition have intervals ending before the current time, while tuples of the current partition have intervals ending at the current time. Updates are assumed to arrive chronologically, so that tuples in non-current partitions are ordered on the intervals’ upper points and tuples in current partitions are ordered on the intervals’ lower points. Using these partitions, the temporal join is computed as the union of the sub-joins on the current and non-current partitions. As mentioned, the approach only supports appendonly databases.
Zhang et al. present the multi-predicate merge join (MPMG join) [40] to support containment queries on XML data in relational database systems. Index data are stored in two relational tables describing occurences of XML elements and text words, respectively. Within these tables, the positions of element occurences are encoded as intervals consisting of the beginning and ending word numbers in the indexed XML document. To evaluate containment queries (e. g. “title contains ‘giraffe’”), SQL SELECT statements with multiple join predicates have to be executed on the the index tables. Using the proposed MPMG join method, such queries are handled quite efficiently. However, the algorithm does not directly support an overlaps join predicate for two intervals.
2.3 Partition-Based Algorithms Partition-based join algorithms first divide the input relations according to their join attribute values. The partitioning is performed so that a given partition of the first relation contains tuples that can only match with tuples in the corresponding partition of the other relation. During the join phase, only tuples in corresponding partitions have to be compared. Finding adequate partitions for interval data is more difficult than finding partitions for scalar values. In general, either a given interval partition has to be compared with several partitions of the other relation, or tuples of one relation have to be replicated in several partitions. Soo, Snodgrass and Jensen introduce a partition-based algorithm for the evaluation of valid-time natural joins [37]. Besides overlapping interval attributes, the joining tuples have to provide matching non-temporal attributes. The algorithm uses a dynamic partitioning approach preventing the replication of tuples in several partitions. The relations are partitioned into non-overlapping intervals, and each tuple is assigned to the last partition that the tuple interval overlaps. During join computation the partitions are processed backwards, and those tuples whose time intervals may intersect more than one partition range are retained in a special join buffer to be combined with tuples in the next partition. So, the partitions are dynamically adjusted during the join computation. Unfortunately, the algorithm performs poorly if the data set contains a lot of tuples with long intervals. In such a case, the number of partitions increases drastically. Furthermore, the approach requires quite sophisticated memory management. Sitzmann and Stuckey [35] propose an extension of this algorithm. Histograms of the temporal data are used to determine partition boundaries that maximize the overall join buffer usage. Because the algorithm regards the number of tuples with long intervals, they can be handled quite efficiently. However, the approach relies on pre-existing temporal histograms. Lu, Ooi and Tan present another partition-based algorithm for computing temporal joins [28]. In this approach, a tuple’s
interval is mapped to a point in a two-dimensional space where the x-value of the point corresponds to the interval’s lower point and the y-value to the length of the interval. Then, the space is partitioned into regions. During join evaluation, a partition of one relation has to be joined with many partitions in the other relation. To speed up the joining phase, the Time Polygon Index [34] is used which is not generally available nor can be directly integrated in object-relational databases. Partition-based algorithms have also been proposed for spatial joins [27] [30] [3]. As mentioned earlier, such spatial algorithms do not behave as well as interval-specific algorithms in general. Our approach also uses partitions to improve the joining performance. Here, the partitions are derived from the internal structure of the Relational Interval Tree.
2.4 Index-Based Algorithms Index-based join evaluation utilizes indexes defined on the join attributes of the input relations to locate joining tuples efficiently. While an index-supported join evaluation is comparatively cheap, the management cost for the index has to be taken into consideration. For the interval join, appropriate ‘interval indices’ are needed whereas in general, relational databases provide indices for scalar attributes only. Though object-relational databases such as the IBM Informix Dynamic Server [20] [6], the Oracle9i Server [29] [38] or the IBM DB2 Universal Database [19] [10] support the logical embedding of custom index types into the database system, they do not facilitate the actual implementation of the access method itself. Modifying or enhancing the database kernel is usually not an option for database developers, as the embedding of block-oriented access methods into concurrency control, recovery services and buffer management causes extensive implementation efforts and maintenance cost [23], at the risk of weakening the reliability of the entire system. The server stability can be preserved by delegating index scans and maintenance to an external process, but this approach requires a custom concurrency control and induces severe performance bottlenecks due to context switches and inter-process communication. Therefore, when considering a certain index type for the support of join computing, we have to check carefully if the index can be integrated into existing systems efficiently. Elmasri, Wuu and Kim propose an index-based algorithm for processing temporal equijoins [12]. The algorithm utilizes a two-level time index [12] where B+-trees are used both to index the non-time attribute(s) in the first level, and to index interval time points in the second. However, the space requirement of the time index is O(n²) for n stored intervals [17]. Due to this redundancy, the time complexity is O(n) for insertion and deletion and O(n²) for interval intersection query processing [1].
Son and Elmasri presented a modified version of the time index that requires less space and used this index to determine the partitioning intervals in a partition-based join algorithm computing the entity join, a derivative of the temporal equijoin [36]. Unfortunately, no experimental results are presented demonstrating the performance of the approach. Zhang, Tsotras and Seeger describe several algorithms based on B+-trees, R*-trees [4], and the multi-version B+-tree (MVBT) [5] for computing the GTE-Join [42]. This sort of join assumes that the key values of the joined tuples lie within a specified range and the time intervals intersect a specified interval. Experiments performed in [42] show that only the MVBT-based algorithms show adequate performance for computing the GTE-Join. Similar results are received for some special cases of the GTE-Join considered in [41]. However, integrating the MVBT-Index in commercial object-relational databases is problematic for the reasons mentioned above. Further join algorithms have been proposed using spatial indexes [16] [9] [18] [2]. In general, these algorithms do not behave as well as approaches using interval-specific indexes. Additionally, they are generally not available in object-relational systems and cannot be integrated by the user efficiently. Of course, the Relational Interval Tree immediately supports index-based temporal join processing in an very obvious way.
3. THE RELATIONAL INTERVAL TREE The RI-tree is a relational storage structure for interval data (lower, upper), built on top of the SQL layer of any RDBS. By design, it follows the concept of Edelsbrunner’s mainmemory interval tree [11] and guarantees the optimal complexity for storage space and for I/O operations when updating or querying large sets of intervals.
3.1 Relational Storage The RI-tree [25] strictly follows the paradigm of relational storage structures [24] since its implementation is purely built on (procedural and declarative) SQL but does not assume any lower level interfaces to the database system. In particular, built-in index structures are used as they are, and no intrusive augmentation or modification of the database kernel is required. On top of its pure relational implementation, the RI-tree is ready for immediate object-relational wrapping. It fits particularly well to extensible indexing frameworks (as already proposed in [39] and provided by current object-relational database systems) which enable developers to extend the set of built-in index structures by custom access methods in order to support user-defined datatypes and predicates. With the RI-tree, all queries and updates on relational storage
a)
John: (4, 23)
Mary: (2, 13)
Bob: (10, 21)
Ann: (21, 30)
root = 16
b) 8 2M
4 2 1
13M
6 3
5
12
10 7
9
4J
21B
10B
23J
14
18
24 20
21A 22
30A
28
26
30
11 13 15 17 19 21 23 25 27 29 31
c) lowerIndex (node, lower, id): 8,2,Mary
16,4,John 16,10,Bob 24,21,Ann
upperIndex (node, upper, id): 8,13,Mary 16,21,Bob 16,23,John 24,30,Ann Figure 4. Example for an RI-tree. a) four sample intervals. b) virtual backbone and registration positions of the intervals. c) resulting relational indexes lowerIndex and upperIndex
structures are processed by pure SQL statements. The robust transaction semantics of the database server is therefore fully preserved.
3.2 Dynamic Data Structure The structure of an RI-tree consists of a binary tree of height h which covers the range [1, 2h−1] of potential interval bounds. It is called the virtual backbone of the RI-tree since it is not materialized but only the root value 2h−1 is stored persistently in a metadata table. Traversals of the virtual backbone are performed purely arithmetically by starting at the root value and proceeding in positive or negative steps of decreasing length 2h−i, thus reaching any desired value of the data space in O(h) CPU time and without causing any I/O operation. For the relational storage of intervals, the node values of the tree are used as artificial keys: Upon insertion of an interval, that node from within the interval that is closest to the root node (when descending the tree) is assigned to the interval. An instance of the RI-tree then consists of two relational indexes which in an extensible indexing environment are preferably managed as index-organized tables. The indexes obey the relational schema lowerIndex (node, lower, id) and upperIndex (node, upper, id) and store the artificial key value node, the bounds lower and upper, respectively, and the id of each interval. An interval is represented by exactly one entry in each of the two indexes, and therefore, O(n/b) disk blocks of size b suffice to store n intervals. For inserting or deleting intervals, the node values are determined arithmetically, and updating the indexes requires O(logbn) I/O operations per interval.
16 8
24
4
12
2 1
6 3
5
10 7
9
20 14
18
28 22
26
30
11 13 15 17 19 21 23 25 27 29 31
Figure 5. Query preparation step for the query interval (11,13) (shaded in light gray): left queries {8,10}; right queries {14,16}; inner query {11-13}
The illustration in Figure 4 provides an example for the RItree. Let us assume the intervals (2,13) for Mary, (4,23) for John, (10,21) for Bob, and (21,30) for Ann (Fig. 4a). The virtual backbone is rooted at 16 and covers the data space from 1 to 31 (Fig. 4b). The intervals are registered at the nodes 8, 16, and 24, respectively. The interval (2,13) for Mary is represented by the entries (8, 2, Mary) in the lowerIndex and (8, 13, Mary) in the upperIndex since 8 is the registration node, and 2 and 13 are the lower and upper bound, resp. (Fig. 4c).
3.3 Intersection Query Processing To minimize barrier crossings between the procedural runtime environment and the declarative SQL layer, an interval intersection query (lower, upper) is processed in two steps. The procedural query preparation step descends the virtual backbone from the root node down to lower and to upper, respectively (Figure 5). The traversal is performed arithmetically without causing any I/O operations, and the visited nodes are collected in two main-memory tables, left queries and right queries both obeying the unary relational schema (node), as follows: nodes to the left of lower may contain intervals which overlap lower and are inserted into left queries. Analogously, nodes to the right of upper may contain intervals which overlap upper and are inserted into right queries. Whereas these nodes are taken from the paths, the set of all nodes between lower and upper belongs to the so-called inner query which is represented by a single range query on the node values. All intervals registered at nodes from the inner query are guaranteed to intersect the query and, therefore, will be reported without any further comparison. The query preparation step is purely based on main memory and requires no I/O operations. In the subsequent declarative query processing step, the transient tables are joined with the relational indexes upperIndex and lowerIndex by a single, three-fold SQL statement (Figure 6). The upper bound of each interval registered at nodes in left queries is compared to lower, and the lower bounds of intervals in right queries are compared to upper. The inner query corresponds to a simple range scan over the intervals with nodes in (lower, upper). The SQL query requires O(h·logbn+r/b) I/Os to report r results from an RI-tree of
SELECT id FROM upperIndex i, :leftQueries q WHERE i.node = q.node AND i.upper >= :lower UNION ALL SELECT id FROM lowerIndex i, :rightQueries q WHERE i.node = q.node AND i.lower
