Agent-Based Optimisation Systems for Electrical Load ... - CiteSeerX

Agent-Based Optimisation Systems for Electrical Load Management Rongxin Li, Jiaming Li, Geoff Poulton and Geoff James. ICT Centre, Commonwealth Scientific and Industrial Research Organisation, Australia

{Ron.Li, Jiaming.Li, Geoff.Poulton, Geoff.James}@csiro.au

ABSTRACT

1. INTRODUCTION

Electrical load management can potentially offer significant benefits in reducing both greenhouse gas emission and energy costs. It is a comparatively new application domain for agent-based technologies, with few existing approaches. Three novel optimal management technologies that we have been developing are briefly reported in this paper. Their relative performance merits under various circumstances are then preliminarily examined, both through comparative experiments and theoretical analysis. A first system is a deterministic, hierarchical one. Some of the intelligent agents in this system perform constrained optimisation using complete information, via a novel approach that converts decisionmaking in the temporal domain into path optimisation in the spatial domain. The system offers a firm satisfiability of constraints, and its multi-stage optimisation can adapt to different goals. However, it has a limited scalability. An alternative system, Stigspace, uses a decentralised constraintsatisfying approach based on partial information. It is nondeterministic, and is inspired by emergent intelligence frequently found in nature. Substantially better scalability results from its design of an indirect information exchange mechanism. However, its solutions may be sub-optimal and constraints are not always predictably satisfiable. The energy reduction performances of both systems are quantitatively compared against a theoretical upper bound. In an effort to improve on both systems, a partly decentralised version of the hierarchical system is being developed. It aims to be more scalable than the hierarchical system, but at the same time more optimising and goal-adaptive than Stigspace. Preliminary experiments show that for reaching the full potential for mitigating high electricity prices of a system that comprises small- to medium-scale clusters of devices, the hierarchical system is the best performing and incurs the lowest hardware cost in deployment. The Stigspace approach is both scalable and adaptable to load change. These make Stigspace best suited for extremely large-scale applications in a dynamic environment. The hybrid system, expected to have considerable optimisation power but with decentralised computation and a potential in the future to offer anytime solutions, will be a promising candidate for medium- to large-scale systems.

An electricity distribution system typically consists of loads, generators, transformer stations and distribution networks. Superimposed upon this physical structure is a similarly complex market structure. The electricity price is low most of the time (less than A$20/megawatt-hour in Australia), but can surge to extreme levels (e.g. A$10,000/megawatthour) when supply is strained by demand. Energy demand management is a new technology for coping with the volatility of the energy market. It has started to receive worldwide attention recently. More importantly, decreased demand peaks lead to reduced greenhouse gas (GHG) emission. This is because peak demand dictates the required capacity of electricity generation. In order to adequately respond to peak demand, utilities are forced to build extra power plants to be used as ”spinning reserves”. Most generators are powered by fossil fuels and contribute to GHG emission, even in the stand-by mode. Therefore, lessened demand peaks constitute a basis upon which GHG emission can be reduced. In a modern system, wholesale electricity prices may be available to industrial users and retail suppliers at regular intervals. In Australia, the basic electricity market interval is 5 minutes [7]. This information potentially enables users to regulate their own usage based on market signals. Peak time electrical load management mechanisms include load shifting, load reduction and load shedding. The first involves, where possible, moving consumption to less peaky times. As an example, load shifting in the management of a refrigerator may be realised by postponing the cooling operation. Pre-cooling is another option, where the refrigerator is cooled ahead of the need in anticipation of a demand peak. However, it generally requires control agents to be integrated within a thermostatic appliance to override its thermostatic control. Load reduction involves the temporary modification of the operating characteristics of appliances in order to decrease energy requirement. In the case of a refrigerator this means keeping its dynamic temperature range closer to the ambient temperature. Finally, the term load shedding usually refers to the practise of shutting down appliances. Without detailed knowledge of the appliances and their applications (which may be time varying), however, such a practise incurs unquantifiable risks. Energy load management is a comparatively new application domain for agent-based technologies, with few mature ap-

Stigspace System Decentralised Partial

Hierarchical System Centralised

Constraint Satisfying NonDeterministic

Fully Optimised Deterministic

Hybrid System Partly Decentralised Incomplete

proaches in this area. A notable such approach is the agentbased, market-oriented algorithms [1, 14, 8]. With real or virtual currency, one or more broker agents carry out a negotiation process with each resource agent to fix usage and price. Some significant advantages have been reported [14]. However, this approach’s disadvantages include the following [8]: (1) lack of simple scalability - existing market-based algorithms require hierarchies of brokers to negotiate with very large numbers of resources, leading to potentially fragile structures; (2) adaptation or replication is required to account for relationships between resource controls actions at different times arising from their physical properties; (3) there is no guarantee of an adequate level of service at resource or system level.

Table 1: Comparison of the Stigspace, Hierarchical and Hybrid systems (incomplete).

Since an early effort using genetic algorithms [4], we have been developing novel, agent-based, optimal management technologies, aimed at deployment in the Australian National Electricity Market within the next five years. Our approaches are intervention-based, and as such are fundamentally different to the above.

plicity in test implementations, as they draw approximately constant power during an actively working period and negligible power when idle [13, 9]. This allows a binary state model to be used. However, we must note that the number of states is not a restriction in any of the algorithms.

In this paper we first briefly report our systems, and then examine their relative performance merits under various circumstances. The systems comprise a network of autonomous agents installed on the physical distribution network. First of all, appliances are equipped with simple management agents, called resource agents (RAs). The discussion in this paper will be specific to the scenario where RAs are external to the energy consuming appliances. An external controller is only able to achieve partial control of the device. For example, while it is possible for an external controller to force a refrigerator to turn off, it is often not possible for it to command the compressor inside the refrigerator to turn on. In order to make generic, realistic simulations that are applicable to even the least controllable appliances, in this study we do not assume that agents can turn loads on at will. Essentially, in our systems each RA only controls the power switch to an electrical device. Another group of agents, called virtual load agents (VLAs), perform load aggregation, which incorporates aggregate-wide optimisation and coordination. Load aggregation is the process by which homogeneous and inhomogeneous commercial, residential and industrial loads are combined to form an aggregated load [2], or a virtual load to an observer external to the aggregate. RAs are designed to be simple enough to be embedded in an enhanced power plug, while a VLA representing a household may be incorporated into a smart electricity meter. The final type of agents in our systems have access to market signals. These are called broker agents (BA). Physically, BAs may be implemented in intelligent electronic devices (IED) in a sub-station. Although the algorithms are designed to be generically applicable, our experiments are conducted with the management of thermostatically controlled appliances for cold storage. Such appliances are sufficiently representative for our current study. In OECD countries, they currently account for 13% (290 TWh) of total domestic electricity consumption [13]. One advantage of using such appliances is the sim-

Centralism Information Optimality Determinism

Complete

Partly optimised SemiDeterministic

We report in more detail the three systems in Section 2 and, after examine the different performance aspects of the systems, report our experiments in Section 3.

2. DIFFERENT MULTI-AGENT SYSTEMS 2.1 Multi-Agent Systems for Power Distribution Three different multi-agent systems will be discussed and compared in this paper. The first is a multi-step optimisation system, called the hierarchical system; The second is a constraint-satisfying system, called the Stigspace system. The last one a partly decentralised optimisation system, called the hybrid system. The first two systems are both designed for real-world applications (e.g. Demand Response Programs [10]). The last one is still being researched and is only partially discussed and compared with in this paper. The main distinguishing features of these systems are compared in Table 1. More detailed descriptions are provided in the following sections.

2.2 2.2.1

Hierarchical Agent System Overview

We start with a description of the hierarchical multi-agent system. BAs determine a permitted consumption level for each VLA using Homeotaxis [11] and Reinforcement Learning [5], on the basis of market signals and information received from the VLAs. This information consists of each VLA’s request and constraint for each planning cycle, as described below. A VLA controls the actions of a group of RAs. For generic applicability of our systems, we need to avoid risks associated with load shedding (see Section 1). Therefore, load shifting and load reduction are the only available options. The VLAs calculate the minimal energy necessary, taking into account the potential offered by both these options. A constraint is that the appliances must be allowed to

maintain an operating level that is within a manufactureror user-defined range.

2.2.2 Algorithm Detail Permissibility of Agent Interventions

Through the load aggregation by a VLA, the group of devices externally behave like a single, virtual load. Internally, the VLA fulfils two roles. Firstly, it predicts the minimal and unconstrained energy requirements for each load under its management. The unconstrained requirement is simply the default usage of the device given no intervention from outside. The minimal requirement, on the other hand, is derived in the VLA’s first constrained-optimisation procedure (OPT 1). The minimal load requirements of appliances are constrained by their hardware and operating limits (e.g. maximum temperature). In order to avoid unpredictable and possibly disastrous consequences, the VLA’s actions should never cause these limits to be exceeded. These are therefore used as constraints in OPT 1. The objective of OPT 1 is to minimise the energy usage in the VLA’s group. However, reduction in energy usage is likely to negatively impact on the user’s experience (e.g. raised average temperature for a cooling appliance), the hardware (e.g. shortened shelf life) and the devices’ temporary performance. Therefore, unless an extreme price pressure warrants the imposition of the minimum energy level, a higher level representing a trade-off should be consumed instead. It is therefore necessary to investigate the optimal trade-off. In our system, this optimal trade-off is determined by the VLA by taking into account the requirements of all of the devices that it manages. This level is calculated by in its second constrained-optimisation procedure (OPT 2).

In this paper, we discretize time into planning windows. Suppose that a set of RAs is indexed by an integer set I ∈ Z. Further, let K ∈ Z fully and uniformly parametrise a δt time interval (which is a market dependent parameter) at a predetermined resolution τ . Suppose that ∀j ∈ J sj (i, k) is a discrete 2D binary function defined on the domain I × K that represents the binary state of the appliance indexed by i during the sub-interval indexed by k over the time interval [t0 + jδt , t0 + (j + 1)δt ). For the scope of this paper it is sufficient to discuss only the time interval when j = 0. The RA’s attempt to control the appliance is through simple enabling or disabling actions, which may lead to a state transition of the appliance. However, whether an RA’s intended transition of the appliance’s state is permissible is constrained by the hardware and operating limits of the appliance that it controls, as mentioned above. The constraints in our experiments are that (1) the RA must not disable the operation when the temperature in a thermostatic cooling appliance is expected to reach the upper bound during the next time step and that (2) the appliance may remain in the idle state even when the RA is enabling (as explained above, this is and ensures that our agent simulations are realistic). To facilitate discussions below, let Si0 be the space of all permissible plans defined on [t0 , t0 + δt ) for a given device i. The permissibility constraint is that s0 (i, ·) ∈ Si0 ,

(1)

The objective of OPT 2 is to minimise interference to the devices’ operation, i.e. agent-to-device intervention. The constraint is the energy usage level set by the BA, when the BA imposes supply restriction by requesting that the VLA’s group consume less than in their usual, unconstrained running mode. An energy quota distribution plan corresponding to OPT 2 is devised simultaneously through group-wise coordination.

OPT 1: Minimising Energy Usage Under Appliances’ Operating Constraints

In terms of communication, the VLA’s minimal requirement and default usage are sent to a BA to assist the BA in its decision making. Upon receiving back an energy usage allowance, the VLA sends a coordinated and optimised switching plan to each RA. At initialisation time, each VLA receives the models or the parameters of a predetermined family of models from either the RAs or an independent source. The models may be periodically updated to accommodate any changes. After initialisation and between model updates, an RA’s role is simply to carry out the switching plan.

where di is the power of ith appliance converted to the time resolution in use here by a constant.

As will be expanded into more detail in the following sections, our optimisation algorithms is a novel approach that converts the task of time-series determination into a path optimisation problem in the spatial domain. A tree structure is employed to encode sequences of device states and potential transitions there between, enabling the use of efficient algorithms, such as those for minimal-cost path finding.

The VLA then proceeds to the second optimisation.

Under the permissibility constraint given by Eq.1, the first optimisation is ! X X sˆ0 (i, k) . (2) di s0 = argmin s ˆ0 (i,·)∈Si0

i∈I

k∈K

OPT 2: Minimising Impact on Appliances Suppose that before t = t0 , the VLA receives a total usage limit m0 for the interval [t0 , t0 + δt ). If m0 is below the total of the unconstrained demand of the appliances under the VLA, then an additional constraint is imposed, namely, ! X X s0 (i, k) ≤ m0 . (3) di i∈I

k∈K

Without loss of generality, the optimal plan satisfies the following equation: X s0 = argmin ωi g (ˆ s0 (i, ·)), P s ˆ0 (i,·)∈Si0

di s ˆ0 (i,k)≤m0

∧

i∈I,k∈K

i

(4)

where g (ˆ s0 (i, ·)) is a measure of the agent-to-device intervention that led to sˆ0 (i, ·), ωi is an application-dependent weighting factor assigned to the ith device. In our experiments here, unit weighting has been applied to all devices.

Binary Tree Representing Power Status

g (ˆ s0 (i, ·)) = |{ˆ s0 (i, k) = 0 : sˆ0 (i, k−1) = 1∧(p < Ph )∧(k ∈ K)}|, (5) where Ph is the upper temperature bound of a thermostatically controlled cooling appliance. The condition p < Ph corresponds to an unnatural, forced switch-off.

Branch

g (ˆ s0 (i, ·)) may be, for example, quantified as the total number of disabling operations (forcible switch-offs) performed by the VLA during the planning window. In our tree-based implementation (see below), this is computed as the following cardinality:

Tree-Encoding of Permissibility Patterns A tree (binary in our current experiments) is constructed, which encodes the switching state on its nodes, and the state transitions between them on its edges. In order to reflect the fact the state transitions are synchronised to the sampling/intervention intervals, in Figure 1 the tree is depicted sideways, parallel to the time axis (the horizontal axis), with root node lying at t0 on the time axis, and the treetop at t0 + δt . Each series of the branches from the root to the treetop (left to right ) represents a permissible pattern of state transition. Minimal Cost Paths The permissibility tree enables the use of efficient spatialdomain algorithms for the time series optimisations. In OPT 1, this is achieved by pruning the tree based on the constraint, and subsequently localising the objective function to the nodes of the tree. Thus, OPT 1 is equivalent to finding the minimal cost path (MCP) on the tree. In OPT 2, the objective function (which is the undesirability of each path) is localised to the edges. However, the energy consumption limit set by the BA, being a constraint global over all of the trees in the VLA’s group, cannot be localised to individual trees. Thus, given the inability to prune individual trees, the additional constraint in OPT 2 must be tested globally. Two methods can be used for the computation. The first is a serial one, which computes the paths and their associated costs one after another, starting from the one that has the least total node costs. This method is particularly useful when an anytime solution is required (e.g. when the computation may not complete in time for the next planning window). In finding the MCP, only the node costs are factored into the path cost, while the edge costs are only recorded for later computation. Upon successful completion of an MCP calculation for a tree, the minimal path cost is reported to the BA as the minimum energy required. The last node in the MCP is then removed, and the computation continues to find the next MCP (on the modified tree with a node removed). This process builds up a candidate pool (CP), as required in the next step (described below), and continues until the allocated time is used up. Both Dijkstra’s Algorithm [3] the Fast Marching algorithm [12]) are

0

5

10

15

20

25 Time (min.)

30

35

40

45

50

Figure 1: A Pruned Binary Tree Representing State Transitions and Switching Sequences. The tree is displayed sideways, parallel to the time axis (the horizontal axis), representing all permissible switching-state sequences and transitions of the device by its blue coloured nodes and branches. The red coloured curve corresponds to a sequence finally selected using the optimisation algorithm under a consumption cap.

efficient for calculating an MCP. The second is a parallel method to build up CP. It simultaneously calculates the costs over all possible paths, using either plain tree sweeping or the Viterbi algorithm (which allows a reduction in the number of nodes and branches by merging those that do not need to be on a unique path for the next step computation). This is usually feasible given that the trees are generally extremely sparse, as the majority of would-be nodes correspond to non-permissible states. Optimal Tradeoffs As the additional constraint in OPT 2 is global and cannot be locally implemented, the essential strategy is to iteratively exchange the path-integrated node costs (energy usage) for less accumulative edge costs, to the extent permitted by the global constraint (the group’s energy usage cap). In the first iteration, the subset P ∈ CP comprising all paths resulting from one less agent action than the MCPs is constructed. The members’ respective energy usages are compared. The member (denoted w) that results from the least

increase in energy usage is chosen. w is then removed from P, and replaced with paths on the same tree as w but incur one less intervention than w. If he new total energy usage, incremented by w’s extra consumption, is less than m0 , then w becomes the new plan for the corresponding device. Otherwise the plans are left unchanged. The iteration continues with the above process repeated, until m0 is reached or exceeded.

2.3

Stigspace System

We have developed a biology-inspired system, called Stigspace System [8]. This system is implemented in the GridAgents Framework described in [6] and comprises four components: • A multitude of RAs: Each RA is the decision maker and controller of the resource. It can measure key properties of its resource, detecting changes and adapting its responses as required; can actively control to satisfy local and global goals; is selfish, in that it seeks to satisfy local goals first; cannot communicate directly with other agents, rather, messaging is carried out using a form of stigmergy, via a common Stigspace (bulletin-board or similar); is compliant, in that it will always act to help satisfy broker goals as long as its own local goals are not compromised. • One or more BAs: Each BA makes decisions and exercises control on behalf of the broker. It receives information on predicted market and network usage and prices; interacts with RAs through Stigspace, where it can read and place information. It constructs global goals using market and predicted usage data. An example is a (possibly variable) grid supply ”cap” on total power usage for a certain period of time. • A ”Stigspace” (web-based bulletin board or similar) which is used for indirect (stigmergic) information exchange, and on which all agents may both place and read messages. • A summarising agent. It acts on Stigspace information to produce derived information (for example predicted total RA demand over time). Such derived information is also placed in Stigspace. The RAs have information about their own local constraints and the environment. At convenient intervals the RAs apply these constraints to a physical model of their resource to calculate a plan for electricity demand or supply for a period into the future. These plans are transferred into market cycles, and then sent to Stigspace. In Stigspace summary data is computed from the plans by a summarising agent, the simplest summary data being the total predicted power demand in each interval. This is then made available to any RAs that wish to use it and also to the broker agent. The broker agent has knowledge of predicted electricity market price as well as information about the plans communicated by participating RAs. The broker acts for electricity market participants, such as retailers and network operators, who provide additional information leading to a desired cap on the total demand for power drawn from the grid. This supply cap is placed in the Stigspace and is made available to any RAs that wish to use it. Although the broker

agent has no direct control over resources, RAs agree to help satisfy any global goals as long as local goals remain satisfied. Therefore, the RAs, when they see total demand and supply cap from Stigspace, can revise their plans using our CoordCap algorithm (described below) while continuing to adhere to their local constraints. The agents submit revised plans and the process is iterated in real time until it stabilises. The process is asynchronous: no explicit coordination is needed between plan submission, plan summing, and broker action. When total demand is stable, the broker agent is in a position to buy power for the next time period. The process is repeated for every market cycle. In our system, the constraints for RAs are temperature bounds for a heating/cooling environment, and the plan calculated for electricity demand or supply is for the next 5 minutes to half hour. The CoordCap algorithm is used by each RA to modify its power usage so as to help satisfy global constraints - here the supply cap determined by the BA. If the supply cap is not satisfied for certain time intervals, the agent’s switching strategy in those intervals will be updated as described below. Each agent iterates the process until either (a) local and global goals are satisfied, (b) no further improvement is possible, or (c) a specified time limit is reached. Once this occurs, all agent actions are ”locked in” for the next 5minute interval, the 30-minute planning period is advanced by five minutes and the process begins again. In the CoordCap algorithm the RA modifies its predicted switching sequence to shift power consumption from each cap-violating interval into its left- and right-hand neighbours. The process includes three steps, and is carried out for all offending intervals, as follows: 1) Locate a random point tx in the offending interval. 2) Partly shift power usage in the interval on the left and right of tx into the left- and right-hand neighbouring intervals respectively. 3) Revise the resultant switching strategy to satisfy the RA’s local constraints. Figure 2 illustrates the steps taken by the RA with a single violating interval. If two or more intervals violate the cap, each interval is treated independently.

2.4

Decentralising the Hierarchical System

In order to simultaneously improve performances in the aspects of the adaptiveness of the optimisation objectives, the firmness, and the scalability, a third approach is being developed where RAs bid for energy advertised as being available by another agent. In analysing the scalability of the Stigspace approach, it became apparent that it is to some extent due to the active participation of the RAs in the coordination process. The parallel nature of this processing leads to constant computation time. Examination of the hierarchical system further revealed that part of the optimisation is inherently distributable. That is, both the tree encoding and OPT 1 could be localised to the RAs. In an attempt to gain a similar advantage while maintaining the optimality and firmness advantages of the hierarchical system, RAs are employed in the hybrid system to perform these tasks instead of VLA. These parts then become constant time computation as they are performed in parallel.

been able to proceed with the awarding process, then provided that m0 has not been reached or exceeded, the bidding continues into the next round, where the the RA chosen in the last round submits its next best bid in the next round, while the bids from the other RAs remain unchanged. In order to decentralise OPT 2, an agent-competition framework is being developed which allows optimisation to continue to levels above individual BMAs. For optimisation outside the BMAs (i.e. which BMA gets how much energy quota for its group), the plan is to use a competition network for optimisation, where every BMA sends each of its neighbours an inhibitory signal, the strength of which is proportional to its current δE(tc ). This inhibitory signal is propagated to either the entire network or only to a limited distance. Only a BMA that wins in the global or a regional competition is able to actually proceed to award its chosen RA the requested amount of additional energy. The strength of inhibition from this BMA is then updated according to the next-round best bid δE(tc + τ )

3. PRELIMINARY THEORETICAL ANALYSIS AND EXPERIMENTS In this Section, we first introduce the device model used in our experiments, followed by comparison studies on various performance aspects of the systems.

3.1

Device Model

We needed a set of appliance models for the tests. Two essential characteristics of thermostatic cold-storage and cooling appliances are (e.g. [13, 9]): 1. That the internal temperature oscillates in regular cycles between a high and a low bound;

Figure 2: The steps taken by the RA to deal with a single violating interval. The red blocks show the time when power is being used. Tmax and Tmin are upper and lower temperature constraints of the appliance. The curve is the predicted internal temperature based on the appliance model and currently planned power usage [8]. OPT 2 remains outside of RAs, as it is inherently a global optimisation that cannot be decomposed or distributed as it stands. Presently, this computation is still performed by a dedicated type of agents. We term an agent that performs only OPT 2 as a Bid Manager Agent (BMA), in order to differentiate from the VLAs described in a previous system. In the new, hybrid system, the RAs indicate, in a bid to the BMA in each bidding round (iteration), the minimum energy they would need for a reduction in agent intervention. A BMA then ranks the RA’s bids (energy requests for deceased intervention by the RAs) and moves to a state ready to award an additional energy amount δE(t) to an RA in a similar fashion to a VLA. if a competition network exists, as described below, then the actual action taken by the BMA is subject to signals from its neighbourhood. After it has

2. That temperature changes decelerate on both the rising and lowering parts of the cycle. For example, it rises at a progressively slower rate as it approaches the ambient temperature. Assuming the device can be adequately modelled as a linear system, we use the following model (which can be regarded as a solution to a linear differential equation) that captures the above characteristics. The model is a piecewise exponential function pi (t) in the temporal domain, restricted to a range defined by the pair Pl and Ph , as follows ( t −sd (1 − e− Uc ) + Ph , t mod U ≤ Uo , (6) pi (t) = t−Uo − otherwise su (1 − e Uc ) + Pl , where Uc is a time-normalising constant, su and sd are model constants that regulate the speed of temperature increase l and decrease, Uo = −Uc log(1 − Phs−P ), Uf = −Uc log(1 − d Ph −Pl ) su

and U = Uo + Uf . In our experiments, the timeconstant was randomised, selected in each run from a uniform distribution over the range [5, 15]. Similarly, the initial temperature was chosen from [1, 10]. The switch state was randomly on or off.

3.2 Theoretical Upper Bound of Optimisation

With the model given in Eq. 6, it can be mathematically proven1 that, as τ → 0 and Pl → Ph , the maximum energy usage reduction (for the hierarchical system, this corresponds to OPT 1) is theoretically  Uc (Ph −pi (0)) h +Pl δt > Uc ( sPu −P ) δt(su −(Ph −Pl )) h −pi (0) Rmax = , (7) 1 otherwise; where pi (0) ≥ Pl is the initial state in which pi (t) enters the planning window. Under the random distributions of parameters described above, and further assuming the entry points {pi (0), i ∈ I} are uniformly distributed over [Pl , Ph ], the above evaluates to an expected value of E(Rmax ) = 0.817 if the peak-demand period δt = 30 minutes. E(Rmax ) increases to 0.970 if the demand peak lasts only 5 minutes. In practise, however, the management resolution τ cannot be 0. More importantly, as Pl moves away from Ph , Rmax becomes increasingly greater than what is achievable. Nonetheless, it is useful tool for understanding the performances of algorithms, as presented in the next Section, as the closeness of a practical result to that limit indicates how well an optimisation scheme explores the entire solution space to take advantage of the least curvature part of pi t).

3.3 Minimum Usage during Demand Peaks We respectively study the minimum energy usages achievable by the first two systems as the percentage of the uncontrolled consumption level. The hybrid system is identical to the hierarchical one in this aspect. Energy minimisation is an explicit objective of the hierarchical system (OPT 1). This manifested as an excellent capability in reducing demand for short periods of time using load shifting and reduction. In the experiments simulating the shortest possible price spikes, which lasted for 5 minutes2 , a mean reduction rate of 0.955 with a standard deviation of 0.007 was achieved when managing 100, 1000 and 5000 RAs in 20 independent random trials each (60 in total). This is above a 0.98 level of the theoretical upper bound. However, it was discovered that the Stigspace did not reach its full potential for short periods. To understand the performances in a more likely scenario, the systems were tested on price spikes of 30 minutes. Twenty independent random trials each were attempted for 100, 1000, 5000 and 10000 RAs respectively. Trials with 10000 RAs in a hierarchical system could not be successfully completed, however, due to memory and speed issues. The total completed trials are therefore 80 for Stigspace, but only 60 for the hierarchical system. The results are tabulated in Table 2, where RRR max is the ratio of the achieved performance to the theoretical upper bound, and is an indication of how well the system exploits the curvature of pi (t).

3.4 Minimum Impact on Appliances As discussed earlier, frequent agent intervention is undesirable from the hardware’s or user’s point of view. The hierarchical system has the minimal intervention frequency as another explicit optimisation objective (OPT 2). 1 The proof cannot be presented in this paper due to the page limit. 2 The temporal resolution of the price signal is 5 minutes in Australia, and the market cycle is 30 minutes.

Number of RAs Stigspace

RR

System

RR Rmax

Hierarchical System

RR RR Rmax

100

1000

5000

10000

0.480 ± 0.042 0.59

0.641 ± 0.016 0.78

0.675 ± 0.004 0.83

0.674 ± 0.001 0.83

0.726 ± 0.053 0.89

0.729 ± 0.002 0.89

0.730 ± 0.002 0.89

NC NC

Table 2: Maximum usage reduction rates achievable (RR, as ratios against un-controlled usage) for a 30 minute peak, along with standard deviations estimated from 20 trials. Comparisons against the theoretical upper bound are also displayed as the rain the rows below RR. ”NC” indicates that tios RRR max the full results for a 10000 RA hierarchical system could not be obtained due to computation issues. Plan Num RAs Stigspace HS OPT1 HS OPT2

5 min 100 1000 1.68 0.707 0.430 0.515 0.240 0.133

10 min 100 1000 2.75 1.27 0.740 0.682 0.220 0.015

30 min 100 1000 3.06 1.99 2.41 2.45 0.900 1.04

Table 3: The number of interventions per device for plans of 5 minutes, 10 minutes and 30 minutes. In this set of experiments, the number of interventions per device within a planning window of 5 minutes, 10 minutes and 30 minutes are respectively compared, as tabulated in Table 3. In the table, the hierarchical system’s performance in terms of interventions is evaluated in two aspects for indepth comparison: when the energy usage is minimised at the end of OPT 1 (given in the tables as ”HS OPT1”), and when the intervention is minimised at the end of OPT 2 (the ”HS OPT2” row), under a random total energy usage limit that is uniformly distributed between the minimal demand and the unconstrained, appliance-default level.

3.5 Scalability Performance variation with the scale of the systems is an important indicator of scalability. Table 2 reveals that neither of the first two systems suffers performance deterioration with scale: the hierarchical system’s performance is stable as the size of the cluster increases, while that of the Stigspace actually improves with scale. Possible speed deterioration is another indicator of scalability. While the computation time using the Stigspace is nearly constant, the results for the VLA indicate a linear increase, as shown in Figure 3. On the other hand, in the hybrid system, the BMA’s computation time increases more dramatically, especially when the number of RAs is above a threshold (approximately 2000 for 30 minute plans, 4000 for 5 minute plans). However, the absolute values for a large number of RAs are still very small. It took less than 4 seconds to compute 30 minute plans for 5000 appliances, and less than 0.3 second to com-

pute 5 minute plans, when performed by an Intel Core 2 CPU at 2.33 GHz. Computation-time also increases in the temporal domain, with the size of the plan window. This is another important factor, as it determines the limit of the plan size when the number of RAs is given. Figure 5 shows 2 series with 5 RAs and 5000 RAs, each being the results of 20 independent runs displayed on a logarithmic scale. The error bars on the 5000RAs series are too small to be visible.

Figure 5: Hierarchical System’s Computation time increases with the size of the plan window.The error bars on the 5000-RAs series are too small to be visible.

Figure 3: Computation time increases with the hierarchical approach.

Figure 4: Computation times for the BMA in the hybrid approach.

4.

DISCUSSION AND CONCLUSION

We have reported and preliminarily examined the relative merits of three novel, agent-based, optimal management technologies through theoretical analysis and experiments. The hierarchical system employs an algorithm that converts decision making in the temporal domain into path optimisation in the spatial domain. It is able to determine, in OPT

1, the full capacity of appliances to mitigate high prices. In practical deployment in the future, this will allow a BA to advertise highly ”firm” promises (i.e. predictions with a near-certain confidence). High firmness is a requirement in some potential application areas such as demand response programs [10]. Another advantage is that, as multi-stage optimisation, it is able to incorporate additional concerns to be minimised, such as effects on the appliance and user experience, as shown in OPT 2. A significant drawback of the hierarchical system is the scalability. Section 3.3 demonstrates that computation time at the VLA level grows dramatically with the number of RAs. The Stigspace system, on the other hand, has demonstrated excellent scalability. Furthermore, although none of the systems’ optimisation performance deteriorates with scale, Stigspace’s results actually improve as the system scales up. The scalability mainly stems from its decentralised architecture, coupled with the separation of the coordination mechanism from the indirect information exchange mechanism. Furthermore, it adapts to change in an unsupervised manner, making it intrinsically scalable and robust. In addition, communication cost using Stigspace may be vastly reduced by employing statistical sampling of the RA population to reduce the number of RAs that need to report their energy usage. Its main drawback is that, being a constraint-satisfying approach based on partial information, it is non-deterministic and that the consumption cap needs to be heuristically determined, although an intelligent adaptive mechanism is currently being investigated. Heuristically determined caps suffer the dilemma that a conservative cap may not reach the system’s full capacity for mitigating high prices, and that an aggressive cap may be unachievable so that the system does not converge after a reasonable number of iterative attempts. In addition, costs such as impacts to the hardware and the user may be difficult to incorporate into the Stigspace design. The hybrid system is our attempt to overcome the scalability limitations of the hierarchical system through decentralisation. In the future it may enable energy usage to

be substantially optimised, by agents embedded in appliances, according to the external signals (e.g. price). The network also consists of BMAs that manage small clusters of RAs, ideally on the scale of a household. These BMAs may compete with its peers in the neighbourhood to achieve larger-scale optimum (this is still under development). The RAs bid for energy that the BMAs advertise as being available. The optimisation task of the VLAs in the hierarchical scheme is partly distributed to the RAs, and partly accomplished through competition between peer BMAs. The likely non-determinism that arises from such competition will be the subject of a future study. Most of the computation becomes scalable in this scheme, with the BMA’s assessment of the RAs’ bids remaining the only non-scalable component. On the other hand, although the computation cost remains small in absolute terms for reasonable-sized clusters, our experiments have demonstrated an accelerated increase of the processing time. This approach remains feasible, however, for over 10,000 RAs. Scenarios where the number of RAs grows further have not yet been successfully simulated, due to difficulties arising from limited available memory on a single computer. In practical deployment, however, the memory restriction is not expected to be an issue, assuming that each RA has at least 512KB RAM. Based on extrapolation of the experimental results, it is estimated that a cluster of 2.5 million RAs should take the BMA approximately 30 minutes to optimise plans for each RA for the next 30 minutes. Even though computation times vary substantially depending on a multitude of factors, such an order of magnitude is the upper limit within which the system is able to function. Further measures to improve the scalability that are being considered include multi-scale management. That is, large clusters of loads are decomposed to multiple levels of virtual loads, each being an aggregation manageable by a BMA. As our systems are intended for real-world deployment, cost issues in deployment may also need to be considered. For example, the hybrid and the Stigspace systems both require much more computation-capable RAs than the hierarchical system. In summary, when it is important to reach the full potential of mitigating high electricity prices of a system that comprises small- to medium-scale clusters of devices (e.g. fewer than 5000 each), the hierarchical approach is the best performing and incurs the lowest hardware cost in deployment. The hybrid system, expected to have comparable optimisation power but with decentralised computation, appears to be a promising candidate for medium- to large-scale systems. Finally, the Stigspace approach is both scalable and adaptable to load change. These make Stigspace best suited for extremely large scale applications in a dynamic environment. Finally, it is important to note that Our preliminary study was restricted to the systems’ technical performances. Full-blown studies in the future will need to incorporate domain-feature considerations to demonstrate the viability and benefits of the systems, and should compare with market-oriented approaches.

5.

ACKNOWLEDGEMENTS

The authors thank Dr Oliver Obst and Dr Mikhail Prokopenko for their valuable suggestions.

6. REFERENCES [1] H. Akkermans, J. Schreinemakers, and K. Kok. Microeconomic distributed control: Theory and application of multi-agent electronic markets. In Proc. 4th international joint conference on Autonomous Agents and Multiagent Systems, July 2005. [2] D. Asber, S. Lefebvre, J. Asber, M. Saad, and C. Desbiens. Non-parametric short-term load forecasting. Electrical Power and Energy Systems, 29:630–635, 2007. [3] E. W. Dijkstra. A note on two problems in connexion with graphs. Numerische Mathematik, 1:269–271, 1959. [4] Ying Guo, Jiaming Li, and Geoff James. Evolutionary optimisation of distributed energy resources. In 18th Australian Joint Conference on Artificial Intelligence, pages 1086–1091, 2005. [5] Ying Guo, Astrid Zeman, and Rongxin Li. A reinforcement learning approach to setting multi-objective goals for energy demand management. In Adaptive Learning Agents and Multi-Agent Systems Workshop, 2008. To appear. [6] Geoff James, David Cohen, Robert Dodier, Glenn Platt, and Doug Palmer. A deployed multi-agent framework for distributed energy applications. In Proceedings of AAMAS’06, pages 676–678, 2006. [7] J. Li, G. Poulton, G. James, A. Zeman, P. Wang, M. Chadwick, and M. Piraveenan. Performance of multi-agent coordination of distributed energy resources. WSEAS Transactions on System and Control, 2:52–58, 2007. [8] Jiaming Li, Geoff Poulton, and Geoff James. Agent-based distributed energy management. In Proc. 20th Australian Joint Conference on Artificial Intelligence, pages 569–578. Gold Coast, Australia, 2007. [9] N. Lu, D. P. Chassin, and S. E. Widergren. Modeling uncertainties in aggregated thermostatically controlled loads using a state queueing model. IEEE Trans. Power Systems, 20:725–734, 2005. [10] R. Bharvirkar N. Hopper, C. Goldman and D. Engel. The summer of 2006: a milestone in the ongoing maturation of demand response. The Electricity Journal, 20(5):62–75, June 2007. [11] Mikhail Prokopenko, Astrid Zeman, and Rongxin Li. Homeotaxis: Coordination with persistent time-loops. In Proceedings of the tenth International Conference on the Simulation Of Adaptive Behavior (SAB’08), 2008. To appear. [12] J. A. Sethian and A. Vladimirsky. Ordered upwind methods for static hamilton-jacobi equations. Proc Natl Acad Sci USA, 98(20):11069–11074, 2001. [13] J. A. Short, D. G. Infield, and L. L. Freris. Stabilisation of grid frequency through dynamic demand control. IEEE Trans. Power Systems, 22(3):1284–1293, 2007. [14] F. Ygge. Market-oriented programming and its application to power load management. Ph.D. Thesis, 1998. ISBN 91-628-3055-4, Lund University.