Solutions to the Computer Networking Challenges of the Distribution ...

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007

1

Solutions to the Computer Networking Challenges of the Distribution Smart Grid Josep M. Selga, Member, IEEE, Agust´ın Zaballos, and Joan Navarro

Abstract—Communication networks play a crucial role in the context of Smart Grids since they enable a broad set of appealing facilities, from both producers and consumers side, to be effectively delivered. So far, several standards (e.g., IEC 618505) have been presented aiming to detail the specific requirements for such networks on the Smart Grid domain. However, the definitions proposed by these standards are confined to a reduced part of the grid: the primary substation. This situation prevents electrical distribution networks from fully exploiting the Smart Grid concept as a whole (i.e., provide truly distributed energy resources). The purpose of this paper is to (1) provide a further insight qualitative analysis of the yet-unexplored communication requirements raised by the electrical distribution Smart Grid, and (2) propose novel solutions targeted at creating link layer networks spanning a given electrical distribution area to allow faster and more reliable communication. Specifically, these solutions are devoted to extend all standards and communication mechanisms already defined for primary substations, to the whole grid by considering the electrical distribution grid as a collection of virtual substations. Furthermore, collected experiences throughout the implementation of this proposal envisage some alternatives to boost the performance of this protocol in terms of reliability and latency. Index Terms—Smart Grid, TRILL, Protocol Architecture, Internetworking.

I. I NTRODUCTION

R

ECENTLY, Information and Communication Technologies (ICT) have become a hot research topic in the context of Smart Grids [1], [2]. The specific characteristics of this new energy-delivery concept have driven several research projects [1], [3] aimed to design an adequate ICT infrastructure to meet the Quality of Service (QoS) expected from the Smart Grid functions [4], [5]. In addition, the broad acceptance of the Smart Grid concept and the related Distributed Energy Resources (DER) in electrical distribution networks claim for the introduction of protective relaying schemes similar to those used in high voltage networks. These protective strategies send messages with stringent reliability demands between different points of the electrical distribution network with delay requirements of a few milliseconds, which force the communication network to react in a similar response time in case of failure. Therefore, the United States Department of Energy analyzed the communication requirements of the smart functions (e.g., Demand Response and DER) and defined the latency requirements and reliability levels for each service [6]. Despite the latest progresses Josep M. Selga, Agust´ın Zaballos, and Joan Navarro are with the Grup de Recerca en Sistemes Distribu¨ıts i Telem`atica, La Salle - Ramon Llull University, 08022 Barcelona, Spain (e-mail: {zaballos,jnavarro,jmselga}@salle.url.edu).

achieved by such research projects, current ICT technologies have still big troubles on meeting these requirements either at link layer [1], [7] or network layer [5], [8]. On the one hand, the broadly used Rapid Spanning Tree Protocol (RSTP) at link layer has recovery times up to a few seconds, which are two orders of magnitude far from the demands posed by the Smart Grid [6]. Although there are newer versions of this protocol that feature much better recovery times, they are usually proprietary solutions. Moreover, RSTP uses a single path and thus is severely failure prone. On the other hand, current optimizations of Interior Gateway Protocol (IGP) at network layer enable convergence times of a few hundreds of ms [9], which are still far from the few tenths of ms claimed by the Smart Grid [6]. To overcome these limitations concerning protection coordination, current Smart Grids establish direct links, typically based on fiber optics, between all devices. Nevertheless, this poorly scalable solution is unable to face the massive introduction of DER and protection systems in electrical distribution networks. Hence, the International Electrotechnical Commission (IEC) has addressed the problem of high recovery communication network delays with the Parallel Redundancy Protocol (PRP) and High-Availability Seamless Redundancy (HSR) [10]. Although these proposals are intended for primary substations rather than the whole distribution network, they do reflect the need for new solutions for the entire Smart Grid. Quite recently, two new protocols have been proposed to solve somewhat similar problems in other technological domains: (1) the Internet Engineering Task Force (IETF) Transparent Interconnection of Lot of Links (TRILL) [11], which proposes the concept of RBridges, and (2) the Shortest Path Bridging (SPB) protocol specified in IEEE 802.1aq [12]. The purpose of this paper is to propose a hybrid combination of both concepts—based on the lessons learnt from the consecution of the European project INTEGRIS [3]—to solve many of the problems posed by the Smart Grid distribution layer concerning the ICT infrastructure. Obtained results show that the herein presented TRILL-inspired solution is able to obtain failover delays that compare favorably with RSTP. In addition, collected qualitative experiences suggest several improvements to still go further in reducing the network recovery delays. The remainder of this paper is organized as follows. Section II highlights the requirements set for the Smart Grid versus current networking practices. Section III examines the communication solutions applicable and proposes a specific working architecture. Section IV explores the alternatives for further improvements to fully cope with the Smart Grid requirements. Finally, Section V concludes the paper.

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007

II. ICT R EQUIREMENTS FOR THE S MART G RID Many of the foreseen Smart Grid services [4] have stringent requirements in terms of availability and delay [6], [13]. These claims are not always considering their extension to the power distribution network and thus are confined to the primary substation. This is mostly due to the difficulties arisen from the distributed, complex, and partly buried nature of the distribution network in addition to its partially meshed physical topology. To bridge this challenging gap, the INTEGRIS project [3] has defined the distribution network requirements shown in Table I according to the analysis performed on IEEE 1646 [4] and IEC 61850 [13] among others. Therefore, the constricting requirements previously defined for primary substations can be relaxed and adapted to the context of distribution networks. This leads to the values indicated in Table I, which shows the functional classes and requirements set in the project [3]. Nevertheless, these harsh objectives are still difficult to achieve in the context of electrical distribution networks considering that, system management, non-critical services, file transfers, and best effort traffic must be supported as well. Thus, the aforesaid network reliability ambitious levels should drive practitioners to design custom ICT networks with built-in redundancy facilities. These approaches should point to hybrid solutions that mesh several technologies similarly to some proposals made for primary substations [10], [14]. The reminder of this section is devoted to extend this idea. TABLE I ICT R EQUIREMENTS A CCORDING TO INTEGRIS [3] Function class

Value signal

Transfer time

Reliability level

Active Protection Functions Command & regulations

Block & trip signal

! 20 ms

O/C Command, Load shedding, Peak shaving Analogical & Digital Television Energy meas, Supply mgmt commands, Alarms Energy meas, other signals

! 2s

Very High (99.999%) High (99.99%)

Monitoring and analysis Advanced Meter & Supply management End to end info. exchange and management

! 2s ! 5m ! 10s

High (99.99%) Low (99%)

! 5m ! 5s

Medium (99.9%)

Indeed, a reasonable objective to pursue [1], [5], [14], [15] is to look for a flexible combination of communication technologies capable of coping with the required and stringent demands of Smart Grids. Following this line of thought, the approach proposed by the INTEGRIS project for communications over the distribution network is to focus on the development of a novel and flexible ICT infrastructure based on mixing heterogeneous Ethernet link layer technologies (Power Line Communications, Wireless, Fiber Optics, etc.) glued together by means of TRILL protocol plus some custom improvements for enhancing QoS and security [2], [7], [15]. The INTEGRIS’ approach interconnects every communications, computing, and storage resource needed in the Smart Grid through a single device called IDEV [2]. Although all IDEVs in the grid are technologically similar, they perform different electrical functions (e.g., efficient inter-

2

networking with other IDEVs by means of TRILL through any kind of links, connection to the WAN network—at least at one point but preferably in more points—to reach the desired reliability level, connection to Wireless Sensor Network (WSN) and similar networks to collect local scattered data possibly through low capacity links, handle security aspects and data replication, and gather data from its immediate surrounding devices such as smart meters/sensors). The following section details the networking facilities of such IDEVs and situates them on the proposed ICT architecture. III. ICT N ETWORKING S OLUTIONS

AND

A RCHITECTURE

Existing solutions such as PRP and HSR [10], targeted to cope with the Smart Grid demands in primary substations, basically aim at duplicating the communication infrastructure (i.e., replicating frames and sending them over two disjoint networks) [13]. These strategies prevent themselves from being deployed over the distribution Smart Grid network due to their excessive cost and lack of scalability. This section discusses a possible ICT networking solution suitable for the distribution network based on the progresses achieved in INTEGRIS. In fact, problems faced by Smart Grid have very similarities with problems in other fields such as Data Center bridging and Metro Ethernet. For instance, in the context of Data Centers, the growth of a full mesh Ethernet switching network leads to an inefficient use of the bandwidth, mainly due to the tree like topology of RSTP and slow recovery times. Therefore, INTEGRIS’ main objective was to create a large Ethernet network to cope with the partially meshed distribution power network, which represents a bigger, in terms of size, challenge compared to a Data Center network. Indeed, TRILL and SPB have resulted on an enhanced alternative to RSTP allowing for optimal bridging of unicast, multicast traffic and larger Ethernet networks using the RBridge concept instead of switches. In addition, both protocols are compatible with current Ethernet switches, allowing the coexistence with legacy Ethernet devices. Hence, the RBridge can be seen as an improved switch identified in the network by a nickname (16 bits long) instead of an IP address. Moreover, RBridges use IS-IS routing protocol with further adaptation for the link layer forwarding behaviour. Once each RBridge knows the path to reach other RBridges and the table of Medium Access Control (MAC) addresses, it is able to forward the frames from the source to the destination. When the frame arrives at the first RBridge (i.e., ingress RBridge), it encapsulates the original Ethernet frame with a new Ethernet header and inserts the TRILL header in the middle of both Ethernet frames with the nickname of the final RBridge (i.e., egress RBridge). Thus, the first Ethernet header contains the information to forward the packet to the next RBridge towards the egress RBridge. This new outer Ethernet header allows the RBridge to change the header in each RBridge crossed and thus enables larger networks than the original Ethernet which maintains the MAC addresses unaltered inside a broadcast domain. As depicted in Fig. 1, every frame is routed to the egress RBridge where it is decapsulated and the original Ethernet frame is forwarded. In

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007

addition, Fig. 1 shows the protocol stack used to transfer a TRILL unicast packet between two legacy Ethernet devices. Actually, an RBridge combines the advantages of bridging and routing [11], which is very convenient for the distribution Smart Grid as it (1) makes no assumptions about physical topology, (2) supports dissimilar link layer Ethernet technologies, (3) coexists with standard bridges, so legacy networks can be upgraded slowly by replacing bridges one at a time, (4) enables optimal paths and effective use of parallel paths within the defined electrical distribution area (not as in bridging), (5) allows interconnection of IP nodes within a distribution area, (6) works for any layer 3 protocol, and (7) does not require network configuration.

3

Fig. 2.

Testbed for TRILL experimentation

Forwarding Detection (BFD) must be used. Remark that nonproprietary TRILL failure detection is based on the non-receipt of hello packages during the hold timer time and it has a minimum expressible value of 1 second [16]. To solve this problem, a promising worthwhile possibility is to explore the combination of TRILL and PRP protocols in order to increase the resiliency of the Smart Grid ICT network as proposed in the following section. IV. I MPROVEMENTS FOR THE S AKE OF

Fig. 1.

Protocol stack between two legacy Ethernet devices

Even though TRILL is a great improvement over RSTP and is capable of managing equal-cost multipaths, it is still unable to meet the required convergence time claimed by the Smart Grid requirements [3], [6] for itself, as shown in the experimentation carried at the several proof-of-concept experiments done in order to test the performance of the TRILL implementation within the INTEGRIS project. On the one hand, the goal of laboratory tests (e.g., Fig. 2 shows one of the deployed scenarios) was to better understand the TRILL implementation and empirically evaluate its capabilities beyond theoretical expectations (note that simulation tools are not suitable because they are unable to handle such a variety of real issues easily). Although the TRILL convergence behavior should be similar to the one of the IS-IS routing protocol (i.e., the theoretical limit for link-state routing protocols to re-route is in link propagation time scales), we experienced that current non-proprietary implementations of TRILL are not anywhere near this point. Despite we tested the TRILL protocol with aggressive timers that brought the convergence time around one second, we observed that it usually created instability issues and increased the traffic overhead as well, making this configuration not recommended for Smart Grids exploitation usage. On the other hand, the field trial experimentation carried out in the distribution grid of A2A Reti Elettriche SpA in the city of Brescia (Italy) [2], [7], [15] has also been used to confirm all the experiments carried on the laboratory. Despite that it is neither possible to submit it onto the same particular situations as in the laboratory, we have observed worse results in terms of response time than it was expected. Nevertheless, we have seen that the proposed system successfully allows layer 2 communication over real-world critical distribution areas through IEC 61850 protocols, which represents a considerable progress beyond the state-of-the-art. In fact, if the link error detection needs to be reduced under the one-second-barrier as claimed in Table I, Bidirectional

THE

S MART G RID

As shown before, RBridges own many advantages over RSTP and routers, and are potentially faster in recovery time from failures, however they are still unable to fully comply with the requirements stated by some Smart Grid applications such as Active Protection Functions (APF) [1], [6], [15], [16]. Therefore we present a set of alternatives (see Fig. 3) that combine TRILL and PRP to better cope with the aforesaid electrical distribution Smart Grid requirements [15]. The proposed combinations analyzed and evaluated in the context of INTEGRIS are described in what follows:

Fig. 3.

Several alternatives for combining TRILL and PRP

Alternative 1: PRP over TRILL: PRP is based on the duplication of the packets transferred over the LAN infrastructure plus adding software at the end points for duplication and later reordering. Despite it can be directly applied over TRILL, we judge that the cost of creating a second disjoint network is too high and unfeasible, since it is hard to reach every where over an electrical distribution network with two communication networks.

JOURNAL OF LATEX CLASS FILES, VOL. 6, NO. 1, JANUARY 2007

4

Alternative 2: Integrating PRP inside TRILL: This approach aims to (1) enable the frames duplication and forwarding over TRILL’s equal-cost multi-paths, and (2) use reordering software at the receiver similar to the one used in PRP. This approach has the advantage of reducing to zero the recovery from certain failures (those that only affect one path) without having to duplicate the network and with a common management interface. Although this modification of TRILL is quite straightforward, such improvement is not included in the current standard, which prevents it from being deployed on the aforesaid tested real-world scenarios. It is important to bear in mind that TRILL just supports equal-cost load balancing, limitation inherited from IS-IS routing protocol. Alternative 3: Duplicate and reordering outside RBridges: To avoid the problem of modifying the TRILL protocol, Alternative 3 duplicates frames before accessing RBridges (outside TRILL’s domain) and discards duplicated frames at the other end and after the RBridges. In this case, a mechanism to force TRILL to use necessarily different paths is needed. Different outer tags can be used to signal that the two traffic flows have to be transmitted over different paths when available. Note that this approach divides the physical network into several logical networks and it is similar to the Virtual Route and Forwarding done at layer 3 routers. Nevertheless it has a huge tag management cost unaffordable in those situations where the network becomes bigger, which will presumably happen in most of the medium and low voltage networks. Alternative 4: Combine TRILL and PRP: This successful approach uses the protocol that best fits to each situation: use PRP when two IDEVs are connected to the same two link layer technologies, and TRILL on other situations. The feasibility of this idea is assured since (1) RBridges and bridges can coexist as stated before, and (2) each link layer technology knows which IDEVs belong to it as described in previous sections. Hence, IDEV scans and dynamically switch from PRP to RBridge according to every situation. Moreover, this alternative complies with current standards and can be combined with alternative 3, which makes it an appealing solution (see Table II). TABLE II B ENCHMARKING OF T ECHNOLOGIES AND A LTERNATIVES Parameter

RSTP

TRILL

Alt. 1

Alt. 2

Alt. 3

Alt. 4

Response Min./Typ.

1-3s/ 30s

1s/ Tens s.

Bumpless

Bumpless

Bumpless

Standard Multiple path Cost Feasible in practice

Yes No

Yes Yes

Yes Yes

Partial bumpless No Yes

Nearly Yes

Yes Yes

Low Yes

Med. Yes

High No

Med. Yes

Med. Yes

Med. May be

V. C ONCLUSION Overall, creating an ICT infrastructure capable of providing the requirements requested by the distribution networks of Smart Grids is a mandatory challenge. This paper has addressed such demands and discussed a set of alternatives based on the key lessons learnt from INTEGRIS’ field trials.

We have shown that heterogeneous networking at link layer is the right way to address communication in an electrical distribution area. In fact, this strategy allows considering the defined area as a virtual primary substation. Therefore, we have proposed a new distributed ICT architecture for low voltage network management—as a part of the distribution network—using heterogeneous link layer communication systems (e.g., PLC, Wi-Fi, Fiber Optics, and wireless sensor networks). In addition, the presented approach extends the principles of the TRILL [11] protocol to the specific conditions posed by Smart Grids and discusses the usage of SPB [12] as an alternative. Nevertheless, there are still few aggressive situations where this proposal cannot cope with the most stringent requirements defined in [3], [6]. Accordingly, this paper encourages practitioners to supplement it by taking into account some principles of the PRP protocol. ACKNOWLEDGMENT We would like to thank INTEGRIS project from the EU FP7 joint ICT-Energy-2009 call (number 247938). R EFERENCES [1] Yan, Y; Qian, Y; Sharif, H. and Tipper, D. “A survey on Smart Grid communication infrastructures: Motivations, requirements and challenges”. IEEE Com. Surveys & Tutorials, no.99, pp.1-16 (2012). [2] Navarro, J; Zaballos, A; Sancho-Asensio, A; Ravera, G. and ArmendarizInigo, JE. “The information system of INTEGRIS: INTelligent Electrical GRId Sensor Communications”. IEEE Transactions on Industrial Informatics (in press, 2012). [3] INTEGRIS Project (2012). INTelligent Electrical Grid Sensor Communications. ICT-Energy-2009 call (number 247938). [Online]. Available: http://fp7integris.eu. [4] IEEE 1646-2004, “IEEE Standard Communication Delivery Time Performance Requirements for Electric Power Substation Automation”. [5] Vallejo, A; Zaballos, A; Selga, J.M. and Dalmau, J. “Next-generation QoS control architectures for distribution smart grid communication networks”. IEEE Com. Magazine, vol. 50, no. 5, pp. 128-134 (2012). [6] U.S. Department of Energy. “Communication requirements of smart grid technologies” (2010). [7] Della Giustina, D; Andersson, L; Casirati, C. Zanini, S. and Cremaschini, L. “Testing the broadband power line communication for the distribution grid management in a real operational environment”. SPEEDAM 2012, pp:785-789. [8] Sivaneasan, B; So, P.L. and Gunawan, E. “A new routing protocol for PLC-based AMR systems”. IEEE Transactions on Power Delivery, vol. 26, no. 4, pp. 2613-2620 (2011). [9] Eramo, V; Listanti, M. and Cianfrani, A. “Multi-path OSPF performance of a software router in a link failure scenario”. IT-NEWS, pp.197-203 (2008). [10] IEC 62439-3:2012, “Parallel redundancy protocol and high-availability seamless redundancy”. [11] IETF RFC6325, R. Perlman et al. “Transparent Interconnection of Lots of Links (TRILL): Base protocol specification” (2011). [12] IEEE 802.1aq, “Shortest path bridging” (2012). [13] IEC 61850, “Communication networks and systems in substations” (2012). [14] Zaballos, A., Vallejo, A. and Selga, JM. “Heterogeneous communication architecture for the smart grid”. IEEE Network, vol.25, no.5, pp.30-37 (2011). [15] Della Giustina, D. et al. “ICT architecture for an integrated distribution network monitoring”. IEEE Workshop in AMPS, pp.102-106 (2011). [16] Eastlake, D; Manral, V; Ward, D. and Banerjee, A. “TRILL (Transparent Interconnection of Lots of Links): Bidirectional forwarding detection (BFD) support”. TRILL Working Group (2012).