Consistent TCAM Policy Table Update with Zero

1

Consistent TCAM Policy Table Update with Zero Impact on Data Path Processing Zhijun Wang, Hao Che, Mohan Kumar and Sajal K. Das Center for Research in Wireless Mobility and Networking (CReWMaN) Department of Computer Science and Engineering The University of Texas at Arlington, Arlington, TX 76019, USA Email: {zwang, hche, kumar, das}@cse.uta.edu

Abstract— Using Ternary Content Addressable Memory (TCAM) for general policy filtering has been gaining popularity due to TCAM’s deterministic and fast lookup performance. However, the policy table updating poses significant challenge for efficient use of TCAM for policy filtering. To avoid erroneous and inconsistent rule matching while a policy table is being updated, the traditional wisdom is to lock the policy table during the rule update period. This, however, can negatively impact the data path processing. In this paper, we put forward a Consistent Policy Table Update Algorithm (CPTUA) for TCAM. Instead of attempting to minimize the number of rule movements to reduce the locking time, CPTUA maintains a consistent policy table throughout the table updating process, thus eliminating the need for locking the policy table while ensuring the correctness of the rule matching. Our analysis shows that even for a policy table with 1 million rules, arbitrary number of rules can be updated simultaneously in subseconds in the worst case, provided that 15% of the policy table entries are empty. This implies that with CPTUA, any new rule can be enforced in subseconds for any practical policy table sizes with zero impact on the data path processing. Key W ords: System design

I. I NTRODUCTION As the Internet applications proliferate and the bandwidth continue to expand, network processors in a router need to be able to classify a packet from 40 to a few 100 nanoseconds (ns) to keep up with multi-gigabit line rates. In the past few years, significant research efforts have been made on the design of fast packet classification algorithms for both Longest Prefix Matching (LPM) and general policy/firewall filtering (PF) [1] - [6]. Unfortunately, most of the algorithmic approaches cannot

provide deterministic performance guarantee and keep up with multi-gigabit line rates. An alternative approach, which has been gaining popularity, is the use of a ternary content addressable memory (TCAM) coprocessor to offload the packet classification task from the network processor. TCAMs are fully associative memories. Each cell in a TCAM can take one of three logic states: 0, 1, or don’t care x. The don’t care state allows the TCAM to store wild cards in any location in a rule. Each TCAM lookup takes one clock cycle. A policy table match requires one or multiple number of TCAM lookups, depending on the search key size. Hence, TCAM-based packet classification ensures deterministic and fast lookup performance. To date, the TCAM-based packet classification is the only solution, which can keep up with 10 Gigabit/sec (Gbps) or OC-192 line rate. Despite its unmatched lookup performance, the TCAM-based solution has its own shortcomings. In addition to issues related to TCAM hardware including high power consumption and relatively large footprint, issues related to memory resource management and the TCAM database update are recognized as critical ones. While TCAM hardware issues and resource management issues are being addressed [8]-[14], the issues related to the TCAM database update have not been well addressed, which motivates our work. The primary source of concern for the TCAM database or general policy table update comes from a wide adoption of a class of TCAM coprocessors, known as Ordered TCAM (OTCAM). In an OTCAM, rules in a policy table are arranged in an ordered list with higher priority rules placed in lower memory address locations. When a search key matches multiple rules, the one in the lowest

2

memory address is selected and the corresponding action in an associated memory is returned. In the worst case, adding a new rule in a policy table may require all the existing rules and their corresponding actions to be moved to new memory locations, causing significant interruption of the data path processing (i.e., lookup process). Shah and Gupta [7] proposed two LPM table update management algorithms to minimize the number of route movements per route update in an OTCAM. Its goal was to minimize the LPM table locking time for route updates. One of the proposed algorithms, considered to be optimal in terms of the worst-case number of rule movements per route update, requires at most 16 route movements per route update. As we shall explain in the next section, locking the LPM table for 16 route movements can actually lead to a loss of more than 10 packets at OC-192 line rate. For general policy table update, our conjecture, as shall be explained in the next section, is that the number of rule movements per rule update in the worst-case is O(N ), where N is the total number of rules in the policy table. Consequently, in the presence of multi-gigabit line rates, attempting to lock the LPM or, in general, a policy table for rule updates can significantly impact the data path processing performance. In this paper, we take a different approach to tackle the problem. Instead of attempting to find efficient algorithms to minimize the number of rule movements, and hence the locking time, we propose a Consistent Policy Table Update Algorithm (CPTUA) which eliminates the need for TCAM policy table locking while ensuring the correctness of the rule matching. The idea behind this novel approach is to maintain a consistent policy table during the table updating process to avoid inconsistent and/or erroneous rule matching. Inconsistent rule matching may occur when rules are being re-arranged and are temporarily out of order in a rule updating process. CPTUA ensures that a consistent policy table is maintained for each rule movement. Erroneous rule matching may occur when a rule or its action is being updated. CPTUA avoids erroneous matching by eliminating direct rule overwriting. Instead, a rule overwriting process is decomposed into three steps (1) de-activate the old rule entry; (2) write the new rule; (3) activate the new rule. With these two requirements being met, CPTUA allows the policy table updating process to take place without locking and yet poses zero impact on the data path processing. However, attempting to meet the above two require-

ments tends to increase the number of operations for per rule update and requires more empty memory entries to allow consistent rule movements. Hence, two critical concerns for the proposed solution is that whether a policy rule can be updated within a reasonable amount of time and whether too many empty memory entries are required to do so. Our analysis, however, shows that this is in fact not an issue. Even for a policy table with 1 million rules, the worst-case delay for any rule update is on the order of sub-seconds, provided that 15% or more of the policy table entries are empty. Comparing with the rest of the phases in a policy enforcement process, from a remote policy server to the management plane, and then to the data plane interface, this worst-case delay is negligible. Therefore, the proposed solution successfully addresses a critical issue related to the general policy table update, making OTCAM a favorable choice for high performance packet classification. In this paper, we also show that the technique developed in this paper can be used for the consistent policy table update in a WEIghed TCAM (WEITCAM), in which the rule priorities are determined by relative weight values assigned to different rules. The rest of the paper is organized as follows. Section II describes the TCAM coprocessors. Section III identifies the fundamental difficulty in developing algorithms for fast policy table update. Section IV presents our solution of TCAM policy table update without table locking. Section V gives the worst case performance analysis of CPTUA. Section VI describes how the proposed technique can be used efficiently for LPM table update and general policy table update in WEITCAMs. Finally, Section VII concludes the paper. II. TCAM C OPROCESSOR We briefly describe the TCAM coprocessor in this section. Figure 1 shows a typical TCAM coprocessor used to perform packet classification on behalf of a network processor. The TCAM coprocessor contains self-addressable rules, which map to different memory addresses in an associated memory, normally a SRAM, containing the corresponding actions. A typical rule for packet classification is composed of five tuples: {source IP address, destination IP address, source port, destination port, protocol number}. The rules are either arranged in an order list or weighted, depending on whether it is an OTCAM or WEITCAM. A search key composed of the same set

3

Switch Fabric/other interface

Local CPU Rule Update

Thread 1 Thread 2

...

Search Key Action

Rule n−1 Rule n

Action 2

Action n−1 Action n

TCAM Co−processor

MAC Framer

Frame

Action 1

...

...

Thread m

Network Processor

Rule 1 Rule 2

Line Card

Fig. 1. A network processor and its TCAM Co-processor

of sub-fields, extracted from the header of a packet to be classified, is passed to the TCAM coprocessor for lookup. Depending on the key size, one full key match or multiple partial key matches against the rules in the policy table take place, with each match taking one TCAM clock cycle. The matched rule with the highest priority then results in the corresponding action in the associated memory to be returned to the network processor. The policy table update is via a local CPU/TCAM coprocessor interface. The local CPU resides in the same line card (LC) as the TCAM coprocessor. The user has the choice as whether to lock or not to lock the network processor interface while the TCAM database is being updated. In the latter case, the TCAM table lookup for data packets via the network processor interface is not interrupted. However, by doing so the TCAM coprocessor may return inconsistent and/or erroneous results. The former guarantees that the TCAM table lookup always return correct results but during the database update period, all the threads that need to access the TCAM coprocessor are suspended, impacting the data path processing performance. To quantify the performance impact caused by TCAM database locking, let us consider a network processor that needs to support an aggregated line rate of 10 Gbps. Assume that the minimum packet size is 49 bytes (40 bytes for a minimum sized IP packet plus 9 bytes of PPP header and trailer for a Packet over SONET (POS) interface). In this case, the network processor has to process packets at a maximum rate of about 25 Million packets per second (Mpps) (or 40 ns per packet time). Assume a TCAM memory width or slot size is 64 bits,

and a CPU runs at 66 MHz clock rate (or 15 ns per clock cycle) with a 64-bit bus, the same as the CPU interface for INTEL IXP2800 [16]. Further assume that the rule/key size is 104 bits (i.e., the size of the five-tuple. It takes two 64-bit slots in TCAM) and the action size is 64 bits. To load a rule and the corresponding action into the TCAM coprocessor requires 2x128/64 = 4 TCAM accesses to load the rule and its mask (the mask must be loaded to set the corresponding wildcarded bits in a rule in the TCAM), and 1 access to load the action. So the estimated total number of TCAM accesses for adding a rule is 5. This translates into about 5x15 = 75 ns, or about 75/40 ∼ 2 packet time. Assume 10 rule movements need to be done to add a new rule in a policy table. Then up to 2x10 = 20 packets may get dropped assuming the policy table is locked for per rule update. For an LPM table update, assume that writing a route to a new location takes 3 clock cycles, i.e., 15x3/40 ∼ 1 packet time. As mentioned in the previous section, up to 16 route movements need to be done to add a new route, so there are up to 16 packets may be dropped if the table is locked during the update process. This estimation clearly demonstrates the deficiency of using database lock as a mean to avoid inconsistent and/or erroneous rule matching. III. C OMPLEXITY OF THE OTCAM P OLICY TABLE U PDATE To facilitate further discussion, we introduce two concepts: Connected Rule Group (CRG) and Multiple Match Group (MMG). A CRG is a set of rules that are connected. A policy table is composed of one or multiple CRGs. Figure 2 (a) shows a CRG which includes four rules. A CRG is further composed of one or multiple MMGs. An MMG is a group of rules that have match priority relationship. For example, in the above CRG, rules A → B → C (→ indicates the increasing priority) form an MMG, and A → B → D form another MMG, as shown in Figure 2 (b). Note that a rule can be in multiple MMGs, but can be in one CRG only. Obviously, rules in an MMG must be arranged in an ordered list in OTCAM and rules from different CRGs can be interleaved in arbitrary order. As a special case, for LPM, rules (i.e., routes) cannot partially overlap with one another. All the rules in an MMG must have super-set and sub-set relationships. Hence, the maximum number of rules in an MMG is L, the number of bits in an IP address, e.g., L=32 for IPv4. For the optimal route update algorithm proposed in [7],

4 A B C

D (a) A

A

B

B

D

C (b)

Fig. 2. An Example of CRG and MMG, the priority increases along the arrow direction. Each rule’s coverage is indicated by a line. (a) A CRG; (b) Two MMGs.

the empty slots are always kept at the center of the LPM table, which allows no more than L/2 route movements when a route is added or deleted. Unfortunately, this is not the case for a general policy table, for which the maximum number of rules in an MMG can be potentially very large. This is due to the fact that wildcards can appear at any location in a rule, and two rules can partially overlap with each other and either one may have higher matching priority than the other. Consequently, keeping empty slots at the center of a policy table does not lead to any optimal solution in terms of the worst-case performance. This point is demonstrated in the following example. Figure 3 shows how two MMGs can merge into one MMG by simply adding one rule, which partially overlaps with the rules from both MMGs. The two MMGs A and B are given in Figures 3 (a) and (b) with five-tuple rules A1 → A2 → A3 → A4 and B1 → B2 → B3 → B4 , respectively. In each rule, only source IP address is specified and all other sub-fields are wildcarded. A search key can match rules either in A or rules in B, but not both. A and B have no priority relationship and hence can be placed independent in the policy table. Assume that initially rules are ordered as shown in Figure 3 (c), i.e., the empty rule entries are placed in the center. Now suppose a new rule C, which partially overlaps with all the rules in both A and B as shown in Figure 3 (d), is to be added. Assume rule C has higher match priority than B4 and lower match priority than A1 . After rule C is added, however, A and B merges into one MMG and all the rules in A must be moved to the lower memory addresses than C and all the rules in B must be moved to the higher memory addresses than rule C, which results in all the existing rules to be re-arranged as shown in Figure 3

(d). This example demonstrates that having empty slots in the center of the table does not help as far as general policy tables are concerned. This example also suggests that as long as significant number of rules are partially overlapping with one another, the number of rules in an MMG could be large, even though the maximum number of multiple match rules are small. [15] showed that for an IPv4 source-destination pair based access control list with only 607 rules, the observed number of partial overlaps is 2249. This implies that an MMG could be very large, resulting in potentially large number of rule movements in the worst-case, regardless what algorithm is in use. Our conjecture is that the worst-case number of rule movements for any algorithm is on the order of O(N ) for a general policy table with N rules. A 4 1.1.1.1−x.x.x.x−x.x−x.x−x

B 4 2.2.2.2−x.x.x.x−x.x−x.x−x

A 3 1.1.1.x−x.x.x.x−x.x−x.x−x A 2 1.1.x.x−x.x.x.x−x.x−x.x−x A 1 1.x.x.x−x.x.x.x−x.x−x.x−x

B 2 2.2.x.x−x.x.x.x−x.x−x.x−x B 1 2.x.x.x−x.x.x.x−x.x−x.x−x

B 3 2.2.2.x−x.x.x.x−x.x−x.x−x

(b) MMG B

(a) MMG A

B 4 2.2.2.2−x.x.x.x−x.x−x.x−x B 3 2.2.2.x−x.x.x.x−x.x−x.x−x A 4 1.1.1.1−x.x.x.x−x.x−x.x−x A 3 1.1.1.x−x.x.x.x−x.x−x.x−x

A 4 A 3

1.1.1.1−x.x.x.x−x.x−x.x−x 1.1.1.x−x.x.x.x−x.x−x.x−x

A 2 A 1

1.1.x.x−x.x.x.x−x.x−x.x−x

Empty Slot Empty Slot

1.x.x.x−x.x.x.x−x.x−x.x−x Empty Slot

C

x.x.x.x−x.x.x.x−x.x−x.x−6

B 2 2.2.x.x−x.x.x.x−x.x−x.x−x B 1 2.x.x.x−x.x.x.x−x.x−x.x−x

B4

2.2.2.2−x.x.x.x−x.x−x.x−x

B3

2.2.2.x−x.x.x.x−x.x−x.x−x

A 2 1.1.x.x−x.x.x.x−x.x−x.x−x A 1 1.x.x.x−x.x.x.x−x.x−x.x−x

B2

2.2.x.x−x.x.x.x−x.x−x.x−x

B1

2.x.x.x−x.x.x.x−x.x−x.x−x

(c)

(d)

Fig. 3. A new MMG is formed by combining two MMGs and a new rule. (a) MMG A; (b) MMG B; (c) Original table; (d) Table after rule C is inserted.

IV. TCAM P OLICY TABLE U PDATE W ITHOUT TABLE L OCKING S OLUTION The previous section demonstrated that locking the TCAM policy table for rule update can be harmful, especially in the worst case when the number of rule movements is O(N ). In this section, a novel approach is proposed for rule updating, which eliminates the need for TCAM table locking while ensuring the consistent and error-free rule matching. More specifically, a Consistent Policy Table Update Algorithm (CPTUA) is proposed. The idea behind the CPTUA is to maintain a consistent policy table during the table updating process to avoid inconsistent and/or erroneous rule matching. Maintaining a

5

consistent policy table means that after each rule movement as part of a rule update process, a rule matching results in a matched rule that is the same as the one that would have been matched either before or after the rule is updated. Inconsistent rule matching may occur when rules are being re-arranged and are temporarily out of order in a rule updating process. Erroneous rule matching may occur when a rule or its action is being overwritten. CPTUA avoids erroneous matching by eliminating direct rule overwriting. Instead, a rule overwriting process is decomposed into three steps (1) de-activate the old rule entry; (2) write the new rule; (3) activate the new rule. With these two requirements being met, CPTUA allows the policy table updating process to take place without locking and yet poses zero impact on the data path processing. In the following subsections, we describe our solution in detail. A. Hardware capability The following summarizes the TCAM coprocessor capabilities, which hold true for most of the existing TCAM coprocessors. 1) Each TCAM rule entry has a valid bit associated with it. To activate a rule entry, this valid bit needs to be set, otherwise, the rule entry is considered as inactive or empty and it will never be matched. Consequently, to delete a rule is nothing more than resetting the valid bit and adding a rule does not take effect until this valid bit is set. 2) After a rule is matched, resetting the valid bit has no effect on the action return process. In other words, deleting a rule cannot stop the return of the action for that rule to the network processor if a match for that rule occurs prior to the deletion operation. 3) Resetting the valid bit for a best matched rule between two successive partial key matches causes the rule not being matched. Instead, the second best matched rule is matched as a result.

gets a match is not the best matched rule. Inconsistent rule matching may occur when a key matching takes place in the middle of a rule update process, which does not guarantee table consistency until the process finishes. In what follows, we identify the conditions for avoiding the erroneous and inconsistent rule matching. Erroneous rule matching can be avoided if the following condition is met: no update related operations can be performed on a rule and/or its corresponding action if the valid bit of that rule is set, with the exception for delete operation, i.e., resetting the valid bit. To meet this condition, all that needs to be done is to avoid overwriting an existing rule with its valid bit set. To this end, the overwriting operations need to be decomposed into three separate processes. They are: 1) A delete process, which involves only a single operation to reset the valid bit of the existing rule. 2) A write process, which involves multiple operations to add a new rule and its corresponding action. 3) Setting the valid bit for the new rule. Based on the capabilities 2) and 3) in the previous subsection, 1) cannot cause any inconsistent or erroneous rule matching. The capability 1) in the previous section ensures that 2) also meets the condition. Finally, 3) obviously meets the condition. Inconsistent rule matching can be avoided if the following condition is met: After any rule is added, deleted, or moved, as part of a rule update process, a key matching always results in a best matched rule, which is identical to the best matched rule either right before or after the rule update process. Any policy table update algorithm that meets the above conditions guarantees that the policy table updating process poses zero impact on the data path (or TCAM lookup) process. Using such kind of algorithms for TCAM policy table update therefore eliminates the need for TCAM policy table locking. In the next section, we propose such an algorithm for general TCAM policy table update.

B. Updating without policy table locking There are two possible types of incorrect rule matching during rule updating without policy table locking, i.e., erroneous rule matching and inconsistent rule matching. Erroneous rule matching may occur if a rule gets a match while the rule or its corresponding action is partially updated. Inconsistent rule matching means that the rule that

C. Consistent Policy Table Update for OTCAM In this subsection, CPTUA is introduced. Based on the discussion in the previous subsection, to avoid erroneous matching, writing a rule involves a process with three steps. Now we also define the move process. The move process is composed of a write process to write the

6

rule

...

...

R

rule ...

rule rule

rule

empty

empty

(a)

(b)

rule

empty rule

...

rule

rule

(c)

...

...

rule

R−E

empty

E

empty

empty

E

...

empty

...

R

empty

E

...

E

empty

rule

E

...

R

rule

Hence, given a batch of updates to be performed including one or multiple rule deletions and/or additions, the software in the control card first calculates the final configuration of the policy table. CPTUA then deletes all the rules from the original policy table, which do not appear in the final configuration. Every rule deletion results in a partially updated consistent policy table, according to 1). Then the existing rule orders are re-arranged to the final configuration without adding the new rules but with the corresponding rule entries allocated. This will result in an intermediate configuration, which is equivalent to the configuration before the re-arrangement, i.e., the configuration right after the rules were deleted. This is true because all the rules whose relative orders have been changed due to this re-arrangement must be rules which have no priority relationship. Otherwise, their relative orders are not changed in the final configuration, according to (2). Finally, CPTUA adds the new rules. Note that the rule re-arrangement must follow a given procedure to ensure table consistency as shall be explained shortly. The above procedure ensures consistent table update. The next step is to map this procedure to a TCAM. There are various ways to do so. In the following, we propose a simple approach whose performance can be easily analyzed. Better solutions may be exploited in the future but the proposed version was already found to give satisfactory performance. In this approach, all the empty rule entries are kept at either the top or the bottom of the policy table. Now suppose initially all the rules are placed at the top of the policy table (i.e., in lower memory address) as shown in Fig. 4 (a). First, as explained above, all the rules that do not appear in the final configuration are deleted from the OTCAM policy table. Then the table configuration

rule

...

1) The only possible effect by deleting a rule is to release the priority relationship among the existing rules. Therefore, all the remaining rules in the table are still consistent after each rule is deleted. 2) Adding a new rule cannot change the priority relationship among the existing rules in an MMG.

rule

...

rule to a new location and then a delete process to delete the rule from its old location. For simplicity, in what follows, we simply use write and move to represent a write process and a move process, respectively. The algorithm design is based on the following two observations:

rule

(d)

Fig. 4. An update process is composed of a deletion phase and an iterative procedure. (a) Original table; (b) After some rules are deleted; (c) After E rules are moved into the empty entries on the bottom; (d) After R − E rules are moved to the top

is shown as in Fig. 4 (b). All the rules as well as some empty entries are at the top R rule entries and the bottom E rule entries are empty. The next step is to move E lowest priority rules in increasing priority order (here, increasing priority order means that the lower priority rules are moved before higher priority rules) to the E empty rule entries, their final locations, as shown in Fig. 4 (c). In the case that a particular rule entry in the E rule entries is supposed to be a newly added rule, that rule entry is left empty. Then the rules in the top R rule entries are moved as close towards the top as possible in a decreasing priority order. This creates at most E empty rule entries at the bottom of the top R addresses, as shown in Fig. 4 (d). Note that the structure of the top R rule entries is now identical to the one in Fig. 4 (a). Except for the rule deletions, which have been done at the beginning, the update process for the top R − E

7

rules follow the same procedure as shown in Fig. 4 (c) and (d). This procedure iteratively applies to the top rule entries until all the rules are in their final configuration. Finally, the new rules are added to the pre-allocated empty rule entries. The update process for a policy table with empty rule entries at the bottom is similar except the rule movement order. CPTUA for a policy table with empty entries on the top (bottom) can be formally stated as follows: 1) Calculate the final configuration of the policy table after rules are updated; 2) Delete all the rules which are not in the final configuration; 3) Move the lowest (highest) priority rules into the empty rule entries at the bottom (top) of the table in increasing (decreasing) priority order. If a rule to be written is not a rule in the original policy table, an empty rule entry is kept at that location; 4) Move all the rules except the ones which have been moved to the bottom (top) toward the top (bottom) of the table in decreasing (increasing) priority order; 5) If the table is in the last configuration, go to step 6. Otherwise, go back to step 3. 6) Write all the new rules to their respective empty rule entries. Now, let’s use an example to show how CPTUA works. Assume that a policy table is composed of two MMGs belonging to two different CRGs, i.e., D {D3 → D2 → D1 } and E {E3 → E2 → E1 }, as shown in Fig. 5 (a). Now suppose that a batch of updates includes the deletion of E2 and additions of F and G. Further assume: 1) The deletion of E2 breaks E into two separate single-rule MMGs, i.e., E3 and E1 . 2) The addition of F merges E3 and E1 back into one MMG, i.e., E1 → F → E3 ; 3) The addition of G further merges all the rules into one MMG, i.e., D3 → D2 → D1 → G → E1 → F → E3 . The final configuration is given in Fig. 5 (f). By running CPTUA, four major intermediate steps are identifies, which correspond to four consistent policy table formats, as shown in Fig. 5 (b)-(e). First, E2 is deleted as shown

1 2 3 4 5 6 7 8 9

D1 E1 E2 D2 D3 E3

Empty slots Empty slots Empty slots

D1 1 2 E1 3 Empty slots 4 D2 D3 5 E3 6 Empty slots 7 Empty slots 8 Empty slots 9

(a) 1 2 3 4 5 6 7 8 9

E1 E3

Empty slots Empty slots Empty slots Empty slots

D1 D2 D3

(d)

(b) 1 2 3 4 5 6 7 8 9

Empty slots Empty slots

E3 Empty slots E1 Empty slots D1 D2 D3

(e)

1 2 3 4 5 6 7 8 9

Empty slots

E1 Empty slots Empty slots Empty slots E3 D1 D2 D3

(c) 1 Empty slots 2 Empty slots E3 3 F 4 E1 5 G 6 D1 7 D2 8 D3 9

(f)

Fig. 5. An example of update process of CPTUA. (a) original table; (b) after E2 is deleted; (c) after three lowest rules are moved to the bottom; (d) after remaining rule are moved toward the top; (e) after top rules are moved into their final locations; (f) final configuration after rules F and G are written.

in Fig. 5 (b). The table is partially updated as rule E2 is deleted. In Fig. 5 (c), the three lowest priority rules are moved to the bottom of the table. The rule order relationship is the same as in 5 (b). In Fig. 5 (d), the rules at the top are moved towards the top end. The order does not change in this step. In Fig. 5 (e), the existing rules are moved to the final configuration. The relative order for rules E1 and E3 is reversed, which does not introduce inconsistency since they are independent at this time. Finally, the new rules F and G are added to complete the batch updates, giving the final configuration (Fig. 5(f)). V. W ORST C ASE P ERFORMANCE A NALYSIS While imposing zero impact on the data path processing, in order to maintain consistent and error-free search key matching, CPTUA generally requires more operations for the policy table update than an algorithm that takes advantage of TCAM database locking. However, a salient feature for CPTUA is that the worst-case rule update performance is independent of the number of rules to be updated in a batch. This feature allows CPTUA to enjoy an upper-bounded update delay performance, inde-

8

A. Number of Write and Delete Operations Per Batch Update First, we define the following parameters and measurement metrics: • • • • • •

N : policy table size in the units of the number of rule entries R: the maximum number of rules in the policy table (R < N ) E: the minimum number of empty rule entries the policy table must maintain (E = N − R) α: the minimum percentage of rule entries which are empty (α = E/R) W : worst-case number of write operations per batch update D: worst-case number of delete operations per batch update

Consider a batch update process that adds some rules to the policy table with all empty rule entries at the bottom. If E is no less than R, all the rules can be moved to their final locations in E empty rule entries in increasing priority order. In this case, W = D = R. Now, let’s focus on the following more interesting case, i.e., E < R or α < 100%. In this case, the batch update process is executed following the process described in the previous section. It is an iterative process as shown in Fig. 4 (c)-(d). Each iteration causes at least E lowest priority rules except the new ones to be written into the empty rule entries at the bottom of the table and up to E redundant rules are deleted. Hence, the number of either write or delete operations is E minus the number of new rules which are to be added to empty rule entries. In the worst case, all these E rules are originally at the top of the policy table. This requires that all the remaining rules be moved toward the top of the policy table. This step costs

extra R − iE write operations and delete operations in the i-th iteration, respectively. Hence, the total number of operations for either write or delete is (R − (i − 1)E) in the i-th iteration. The iterative process continues until all the rules are moved to the locations in the final configuration. Finally, the new rules are added to the pre-allocated empty entries after the iterative process. The maximum number of iterations is d 1/αe. Hence, the total number of write operations and delete operations in the worst case is

W =D=

d1/αe

X i=1

(R − (i − 1)E) =

1 1 (1 + d e)R (1) 2 α

The larger the α is, the smaller the number of write and delete operations is required for each batch update. When α ≥ 100%, both W and D reach the minimum value R. If the batch update process also includes the deletion of d rules, the update process first deletes these rules. Hence d extra delete operations are required and 1 1 1 1 (2) D = (1 + )R + d ≤ (3 + )R 2 α 2 α We note that although the above results are derived based on the assumption that all the empty rule entries are at the bottom of the policy table, the results also hold true when all the empty rule entries are at the top of the policy table. This is because the two cases are symmetric and there are no extra operations required for one versus the other. 12

Write Delete

10

Number of Operations (xR)

pendent of rule structures and update patterns. A possible implementation of CPTUA is to allow periodic updates at an update interval equal to the worst-case update time. The update requests are buffered and executed in the next execution period as a batch. With this implementation, the maximum delay for a rule to be updated is twice of the maximum update time. In this section, we first calculate the worst-case number of write and delete operations for each batch update at any given table utilization. Then we give the performance upper bound on the worst-case update delay.

8

6

4

2

0

10

20

30

40

50

60

70

Minimum Percentage of Empty Entry (%)

80

90

100

Fig. 6. The number of write and delete operations versus minimum percentage of empty rule entries in the worst case

Figure 6 plots the functional relationship between D and W versus α. Note that the number of write operations

9

B. Maximum Update Delay

(3)

The maximum delay for each rule update is then equal to 2tu , given that the rules are updated in batch periodically at tu time interval. Equations 1-3 quantitatively characterize the relationship among all the parameters involved. They can be used to guide the OTCAM coprocessor resource provisioning. A designer can adjust any of these parameters to obtain a bounded maximum update delay. For example, if rule enforcement can tolerate a longer delay, the higher TCAM utilization can be achieved. On the other hand, increasing the local CPU write speed can raise the OTCAM utilization without incurring higher rule update delay. Now let’s estimate the worst-case rule update delay by plugging in the parameter values based on the state-ofthe-art technologies. As stated in Section II, IXP2800 CPU interface runs at 66 MHz clock rate (15 ns per clock cycle) with 64-bit bus. We assume that the same CPU interface is available for the TCAM coprocessor. Then each write operation takes 5 clock cycles for a 104-bit rule with 64-bit action, plus 1 clock cycle for activating the rule. Then tw = 15x6 = 90 ns. Each deletion takes 1 clock cycle, i.e., td = 15 ns. Fig. 7 gives the result for the maximum update delay for each rule (i.e., 2 tu ) versus α at R = 1 million rules. Note that the maximum update delay for each rule using CPTUA is a little bit above 2 seconds at α = 5%. The maximum update delay reduces to less than 1 second when α ≥ 15%. Given that the policy enforcement is either controlled manually by network administrators or by a remote policy server, enforcing a rule usually takes seconds to minutes to accomplish. Therefore this maximum

CPTUA Locking Method

2

1.5

1

0.5

0

Assume that it takes tw and td time units for a write operation and delete operation, respectively. Then the maximum update process time can be expressed as: tu = W tw + Dtd

2.5

The maximum delay for rule update (seconds)

is R in the worst case if the policy table is locked for rule updates. CPTUA can achieve the same performance in terms of number of write operations if α ≥ 100%. As α decreases, D and W increase. For instance, for α = 5%, W = 10.5R and D = 11.5R. A delete operation takes a single clock cycle, and a write operation usually takes 5 to 10 clock cycles. Hence, at α = 5%, the maximum update delay by using CPTUA is about 13 times longer than that by using an algorithm based on locking. The following subsection quantifies the maximum update delay.

10

20

30

40

50

60

70

Minimum Percentage of Empty Entry (%)

80

90

100

Fig. 7. The maximum delay versus minimum percentage of empty rule in the worst case

update delay is negligible. Also plotted in Fig. 7 is the maximum update delay for each rule when an algorithm based on locking is used. It is about 0.15 seconds, independent of α. During the lock period (0.075 seconds, half of the maximum update delay), however, up to 1.875 million packets can be dropped assuming that the network processor handles 10 Gbps line rate, resulting in significantly performance degradation. The above analysis clearly demonstrates the viability and importance of CPTUA for TCAM policy table updating. With CPTUA, an OTCAM can then provide true maximum and deterministic throughput performance guarantee for data path processing. VI. C ONSISTENT RULE U PDATE FOR LPM AND WEITCAM BASED P OLICY TABLE Since the LPM table update is a special case of the general policy table update, CPTUA can be directly applied for the LPM table update. However, as we mentioned before, any algorithm that meets the two conditions in Section IV-B does not require table locking while imposing zero impact on data path processing. It can be easily verified that the two algorithms proposed in [7] satisfy the consistency condition specified in Section IV-B. The other condition is met as long as the overwriting follows the three-step procedure specified in Section IV-B. Hence, the two algorithms in [7] can be easily modified to allow route updating without locking the LPM table. The added operations are the deletions to avoid overwriting rules. As the maximum number of route movements is 16 for the algorithms proposed in [7], the maximum number of added

10

delete operations is 16, which amounts to only about 240 ns extra delay. In contrast, locking the LPM table for the movement of 16 routes can cause up to 16 packets to be lost as mentioned in Section II. The technique proposed in this paper works even better for WEITCAMs [17]. For policy table update in a WEITCAM, no extra empty rule entry is required, meaning that the policy table can be fully utilized. Again, given a batch of updates to be performed including one or multiple rule deletions and additions, the rule deletions are performed first, which will not cause any inconsistency. To add a rule, instead of having to move some of the existing rules around as is the case for an OTCAM, the weight values of some of the existing rules may need to be changed. To maintain consistency, the weight values for the existing rules must be updated before the new rules can be added. Since weight value updating requires only one clock cycle, no erroneous search key matching can occur while the weight value is being updated. Therefore, the rule weight sub-field overwriting is allowed. To ensure consistency while the weights are being changed, changing the weights to larger (smaller) values (here, a lager weight indicates a higher priority), it must be executed in decreasing (increasing) priority order. After all the rules in the policy table are set to their final configuration, the new rules can then be written and activated to finalize the configuration. Rule

WT

Rule

WT

J 1

1

J 1

K2 Empty slot

5

K2

3

L

4

K1

4

K1

K3 J 3

6

K3 J 3

2 6

3

(a)

6

Empty slot

Empty slot J 2

1

2

J 2

5

(b)

Fig. 8. A WEITCAM table with 6 rules, 3 weight bits for each rule. (a) Original table; (b) After rule L is inserted.

Let us look at an example as shown in Figure 8. The updating process is to add a rule L into a policy table. Assume that rule L overlaps with rules both in MMG J and MMG K, which are themselves belonging to different CRGs. In MMG J, rule L has higher priority than J1

and lower priority than J2 . In MMG K, rule L has higher priority than K2 and lower priority than K3 . After rule L is added, the possible new weights for these rules are shown in Figure 8 (b). In this case, the weights for J2 and J3 are increased and the weights for J2 and J3 must be updated in decreasing priority order, i.e., first update J3 and then J2 . For K1 and K2 , the weight values are to be reduced and the weights must be updated in increasing priority order, i.e., K1 must be updated before K2 . Finally, the new rule L is added with weight value 4. VII. C ONCLUSIONS In this paper, we proposed a Consistent Policy Table Update Algorithm (CPTUA) for general policy table update in OTCAM. Instead of attempting to minimize the number of rule movements to reduce the locking time, CPTUA maintains policy table consistency after each rule movement, thus eliminating the need for locking the policy table while ensuring the correctness of the rule matching. Thus, using CPTUA for rule update poses zero impact on data path processing. Our analysis showed that even for a policy table with 1 million rules, arbitrary number of rules can be updated simultaneously within subseconds, provided that no less than 15% of the rule entries are empty. This implies that with CPTUA, any new rule can be enforced in subseconds for any practical policy table sizes. Although it is targeted at the policy table update in an ordered TCAM (OTCAM), we demonstrated that the proposed technique can be used to deal with the policy table update in a WEITCAM. R EFERENCES [1] T.V. Lakshman and D. Stidialis. “High speed policy-based packet forwarding using efficient multi-dimensional range matching”, In Proceedings of ACM Sigcomm’98, September 1998. [2] V. Srinivasan, G. Varghese, S. Suri, M. Waldvogel, “Fast and Scalable Layer Four Switching ”, In Proceedings of ACM Sigcomm’98, September 1998. [3] V. Srinivasan, S. Suri, M. Waldvogel, “Packet classification using tuple space search”, In Proceedings of ACM Sigcomm’99, September 1999. [4] P. Gupta and N. McKeown. “Packet classification on multiple fields”, In Proceedings of ACM Sigcomm’99, September, 1999. [5] P. Gupta and N. McKeown. “Packet classification using hierarchical intelligent cuttings”, In Proceedings of the 7th Hot Interconnects (HotI’99), August, 1999. [6] F. Baboescu and G. Varghese. “Scalable packet classification”, In Proceedings of ACM Sigcomm’01, September 2001.

11

[7] D.Shah and P. Gupta, “Fast updating algorithms for TCAMs”, IEEE Micro, pp 36-47, 2001. [8] R. Panigraphy and S. Sharma, “Reducing TCAM Consumption and Increasing Throughput”, In Proceedings of the 10th Hot Interconnects (HotI’02), August, 2002. [9] S. Sharma and R. Panigraphy, “Sorting and Searching Using Ternary CAMs”, In Proceedings of the 10th Hot Interconnects (HotI’02), 2002. [10] F. Zane, G. Narlikar and A. Basu, “CoolCAM: Power-Efficient TCAMs for Forwarding Engines”, In Proceeding of IEEE INFOCOM’03, April, 2003. [11] F. Baboescu, S. Singh and G. Varghese, “Packet Classification for Core Routers: Is there an alternative to CAMs?”, In IEEE INFOCOM’03, April, 2003. [12] N. F. Huang, W.E. Chen, and C.Y. Lou, and J.M. Chen “Design of Multi-field IPv6 Packet Classifiers Using Ternary CAMs”, In Proceedings of IEEE GLOBECOM 2001, November 2001. [13] H. Liu, “Routing Table Compaction in Ternary CAM”, IEEE Micro, 22(1), pp58-64, 2002. [14] J. V. Lunteren and A. P. J. Engbersen, “Multi-field Packet Classification Using Ternary CAM”, Electronics Letters, 38(1), pp2123, 2002. [15] M.E.Kounavis, A.Kumar, H. Vin, R. Yavatkar and A. T. Campbell, “Directions in Packet Classification for Network Processors”, In Proceedings of Second Workshop on Network Processors (NP2), February, 2003. [16] M. Adiletta, M.R.Bluth, D.Bernstein, G. Wolrich and H. Wilkinson, “The next Generation of Intel IXP Network Processors”, Intel Technology Journal, v6(3), pp6-18, 2002 [17] H. Che, Y. Wang and Z. Wang, ”A Rule Grouping Technique for Weight-Based TCAM Coprocessors”, To appear on The 10th Hot Interconnects (HotI’03), August, 2003.