Bilateral Negotiation in a Multi-Agent Energy Market Fernando Lopes1 , A. Q. Novais1 and Helder Coelho2 1
LNEG − National Research Institute, Dept. of Modelling and Simulation, Estrada do Pac¸o do Lumiar 22, 1649-038 Lisbon, Portugal {fernando.lopes,augusto.novais}@ineti.pt 2 University of Lisbon, Dept. of Computer Science, Bloco C6, Piso 3, Campo Grande, 1749-016 Lisbon, Portugal
[email protected]
Abstract. Energy markets are systems for effecting the purchase and sale of electricity using supply and demand to set the price. A typical energy market involves a wholesale market for electricity generation, when competing generators offer their electricity output to retailers, and a retail market for electricity retailing, when end-use customers choose their supplier from competing electricity retailers. This paper addresses the challenges created by competitive energy markets towards ensuring the full benefits of deregulation. It presents a multi-agent energy market composed of multiple autonomous computational agents, each responsible for one or more market functions, and each interacting with other agents in the execution of their responsibilities. Additionally, the paper presents a negotiation model for autonomous agents. The model handles bilateral multi-issue negotiation and formalizes a set of negotiation strategies studied in the social sciences and frequently used by human negotiators.
1
Introduction
Multi-agent systems (MAS) have generated lots of excitement in recent years because of their promise as a new paradigm for conceptualizing, designing, and implementing complex software systems. The major motivations for the increasing interest in MAS research include the ability to solve problems in which data, expertise, or control is distributed, the ability to allow inter-operation of existing legacy systems, and the ability to enhance performance along the dimensions of computational efficiency, reliability, and robustness. Agent technology has been used to solve real-world problems in a range of industrial and commercial applications, including manufacturing, process control, telecommunications, air traffic control, information management, electronic commerce, and business process management (see, e.g., [3, 12, 13]). The electrical power industry provides the production and delivery of electricity to businesses and households through a grid. This industry is commonly split into four processes, namely electricity generation, electric power transmission, electricity distribution, and electricity retailing. Electricity is most often produced at power stations, transmitted at high-voltages to multiple substations near populated areas, and distributed at medium and low-voltages to consumers. Electricity retailing has not changed much over time − customers are normally charged based on the amount of energy consumed.
The electrical power industry was traditionally heavily regulated with extensive public ownership, federalized organisational structures, and a lack of market-price mechanisms. Its deregulation basically separates the contestable functions of electricity generation and retail from the natural monopoly functions of transmission and distribution. This, in turn, leads to the establishment of a wholesale market for electricity generation, when competing generators offer their electricity output to retailers, and a retail market for electricity retailing, when end-use customers choose their supplier from competing electricity retailers. Clearly, opening up electricity production to competition is an important tool to improve the efficiency of the electricity production industry and therefore to benefit all electricity consumers. Competitive forces can drive producers to innovate and operate in more efficient and economic ways. Innovation can lead to lower prices and a better use of energy resources. Energy markets in general and multi-agent energy markets in particular have received some attention lately (see, e.g., [2, 14, 20]). However, despite these and other relevant pieces of work, most challenges created by deregulation are still waiting to be addressed more thoroughly. At present, there is a need to develop computational tools to help manage the complexity of energy markets towards ensuring long-term capacity sustainability. Against this background, the purpose of this paper is twofold: 1. to present a multi-agent energy market − the market is composed of a collection of autonomous computational agents, each responsible for one or more market functions, and each interacting with other agents in the execution of their responsibilities; 2. to present a negotiation model for autonomous agents − the model handles bilateral multi-issue negotiation and formalizes a set of negotiation strategies studied in the social sciences and frequently used by human negotiators. This paper builds on our previous work in the area of automated negotiation [7–9]. In particular, it considers a number of strategies based on rules-of-thumb distilled from behavioral negotiation theory. It also lays the foundation for performing an experiment to investigate the performance of agents operating in the energy market and equipped with the negotiation model. The remainder of the paper is structured as follows. Section 2 describes a multi-agent energy market. Section 3 presents a negotiation model for autonomous agents, focusing on the operational and strategic process of preparing and planning for negotiation (usually referred to as pre-negotiation), and the central process of moving toward agreement (usually referred to as actual negotiation, or simply negotiation). Finally, related work and concluding remarks are presented in sections 4 and 5 respectively.
2
Multi-agent Energy Market
Multi-agent systems are ideally suited to represent problems that have multiple problem solving entities and multiple problem solving methods. Central to the design and effective operation of a multi-agent system are a core set of problems and research questions, notably:
1. the design problem − how to formulate, describe, decompose, and allocate different problems and synthesize results among a group of intelligent agents? 2. the coordination problem − how to ensure that agents act coherently, accommodating the local decisions or non-local effects and avoiding harmful interactions? The design problem is focused on the domain the system is intended to solve in a distributed manner, i.e., a deregulated energy market involving a wholesale market and a retail market. Practically speaking, the role of the wholesale market is to allow trading between generators and retailers both for short-term delivery of electricity and for future delivery periods − competing generators can offer their electricity output to retailers. The role of the retail market is to allow trading between energy consumers and retailers − end-use customers can choose their supplier from competing electricity retailers. Accordingly, we consider the following types of agents: 1. generators or producers, who in aggregate sell to the wholesale market; 2. retailers or suppliers, who in aggregate buy from the wholesale market and sell to the retail market; 3. customers or consumers, who in aggregate buy from the retail market. The agents are computer systems capable of flexible autonomous action in order to meet their design objectives. They can to respond in a timely fashion to changes that occur in the environment, exhibit goal-directed behavior, and interact with other agents in order to reach their design objectives. In particular, competing generators can interact with various retailers to offer their electricity output and, mainly, end-use customers can interact with competing electricity retailers to choose their supplier. The coordination problem is focussed on ensuring that autonomous agents act in a tightly coordinated manner in order to effectively achieve their goals. This problem is addressed, at least in part, by designing agents that are able to coordinate their activities through negotiation. Specifically, for the case of a deregulated market, the agents are charged with executing actions towards the achievement of their private goals and, thus, conflicts inevitably occur among them. Negotiation is the predominant process for resolving conflicts. Accordingly, the agents are equipped with a negotiation model enabling them to: (i) prepare and plan for negotiation, (ii) generate, evaluate and exchange offers, (iii) come to agreements acceptable to all parties, and (iv) implement final agreements, i.e., determine who needs to do what once contracts are signed. Now, in order to move towards the full benefits of deregulation, we put forward the following requirement for market design: the arrangement of customers’ electricity supply should be achieved by having contracts that specify the provision of an amount of energy for a certain period of time (e.g., one hour). At present, the overall energy consumption in several countries seems to be increasing at a faster pace than energy production. The consequences of not being able to support the demand of energy in the near future are seriously worrying: brownouts or blackouts and the subsequent economical loses. Furthermore, peak loads can approach or even go beyond existing supply capabilities. An appealing method to prevent this worst-case scenario consists of reducing the energy demand and, if this is not possible, executing energy intensive processing tasks whenever the demand of energy is lower.
To this end, there have been a number of initiatives to distribute energy demand over time to avoid peak loads. In particular, many companies have already presented a two-rate tariff to smooth the daily demand profile (cheaper night tariff). This dual model can easily be refined if, instead of two rates, companies offer three rates or even an hour-wise tariff. For instance, a company can present a three-rate tariff by considering that from 7 to 9 am and from 7 to 9 pm are peak hours, from 12 am to 3 pm and from 10 pm to 11 pm is a medium-load period and, finally, the rest is low-demand time. Furthermore, we put forward another requirement for market design: suppliers and consumers should be able to enter into contracts to protect themselves from volume and price risks. Volume risk is a term used to denote the phenomenon whereby the market participants have uncertain quantities of consumption or production. Price risk is a term used to denote extremely high price volatilities at times of peak demand and supply shortages.
3
Multi-Agent Negotiation
Negotiation is an important and pervasive form of social interaction − it may involve two parties (bilateral negotiation) or more than two parties (multilateral negotiation), and one issue (single-issue negotiation) or many issues multi-issue negotiation). This section briefly introduces a model for autonomous agents that handles two-party multi-issue negotiation (see [7–9] for an in-depth discussion). 3.1
Pre-Negotiation
Pre-negotiation is the process of preparing and planning for negotiation and involves mainly the creation of a well-laid plan specifying the activities that negotiators should attend to before actually starting to negotiate [6]. Accordingly, we describe below various activities that negotiators make efforts to perform in order to carefully prepare and plan for negotiation. Let Ag = {ag1 , ag2 } be the set of autonomous negotiating agents. Let Agenda = {is1 , . . . , isn } be the negotiating agenda − the set of issues to be deliberated. The issues are quantitative variables, defined over continuous intervals. Effective pre-negotiation requires that negotiators prioritize the issues, define the limits, and specify the targets. Priorities are set by rank-ordering the issues, i.e., by defining the most important, the second most important, and so on. The priority pril of an agent agi ∈ Ag for each issue isl ∈ Agenda is a number that represents its order of preference. The weight wil of isl is a number that represents its relative importance. The limit limil or resistance point is the point where agi decides that it should stop to negotiate, because any settlement beyond this point is not minimally acceptable. The level of aspiration or target point trgil is the point where agi realistically expects to achieve a settlement. Additionally, effective pre-negotiation requires that negotiators agree on an appropriate protocol that defines the rules governing the interaction. The protocol can be simple, allowing agents to exchange only proposals. Alternatively, the protocol can be complex, allowing agents to provide arguments to support their negotiation stance. However, most sophisticated protocols make considerable demands on any
implementation, mainly because they appeal to very rich representations of the agents and their environments (see, e.g., [4, 10]). Therefore, we consider an alternating offers protocol [11]. Two agents or players bargain over the division of the surplus of n ≥ 2 issues (goods or pies) by alternately proposing offers at times in T = {1, 2, . . .}. The negotiation procedure, labelled the “joint-offer procedure”, involves bargaining over the allocation of the entire endowment stream at once. An offer is a vector (x1 , . . . , xn ) specifying a division of the n goods. Once an agreement is reached, the agreed-upon allocations of the goods are implemented. The players’ preferences are modelled by assuming that each player agi discounts future payoffs at some given rate δit , 0 < δit < 1, (δit is referred to as the discount factor). The cost of bargaining derives from the delay in consumption implied by a rejection of an offer. Practically speaking, the justification for this form of preferences takes into account the fact that money today can be used to make money tomorrow. Let Ui be the payoff function of agi . For simplicity and tractability, we assume that Ui is separable in all their arguments and that the per-period delay costs are the same for all issues: (t−1)
Ui (x1 , . . . , xn , t) = δi
Pn
l=1
wil uil (xl )
where wil is the weight of isil and xl denotes the share of agi for isil . The component payoff function uil for isil is a continuous, strictly monotonic, and linear function. The distinguish feature of time preferences with a constant discount rate is the linearity of the function uil [11]. The payoff of disagreement is normalized at 0 for both players. 3.2
Actual Negotiation
Actual negotiation is the process of moving toward agreement (usually by an iterative exchange of offers and counter-offers). The negotiation protocol defines the states (e.g., accepting a proposal), the valid actions of the agents in particular states (e.g., which messages can be sent by whom, to whom, at what stage), and the events that cause states to change (e.g., proposal accepted). It marks branching points at which negotiators have to make decisions according to their strategies. Hence, this section formalizes a set of strategies studied in the social sciences and frequently used by human negotiators. Negotiation strategies can reflect a variety of behaviours and lead to strikingly different outcomes. However, the following two fundamental groups of strategies are commonly discussed in the behavioral negotiation literature [15, 19]: 1. concession making − negotiators who employ strategies in this group reduce their aspirations to accommodate the opponent; 2. problem solving − negotiators maintain their aspirations and try to find ways of reconciling them with the aspirations of the opponent. Two explanatory and cautionary notes are in order here. First, most strategies are implemented through a variety of tactics. The line between strategies and tactics often seems indistinct, but one major difference is that of scope. Tactics are short-term moves designed to enact or pursue broad (high-level) strategies [6]. Second, most strategies are only informally discussed in the behavioral literature − they are not formalized, as typically happens in the game-theoretic literature.
Concession making behaviour aims at partially or totally accommodating the other party. Consider two incompletely informed agents bargaining over n distinct issues {is1 , . . . , isn }. For convenience, each issue isl is modelled as an interval [minl , maxl ]. The opening stance and the pattern of concessions are two central elements of negotiation. Three different opening positions (extreme, reasonable and modest) and three levels of concession magnitude (large, moderate and small) are commonly discussed in the behavioral literature [6]. They can lead to a number of concession strategies, notably: 1. starting high and conceding slowly − negotiators adopt an optimistic opening attitude and make successive small concessions; 2. starting reasonable and conceding moderately − negotiators adopt a realistic opening attitude and make successive moderate concessions; 3. starting low and conceding rapidly − negotiators adopt a pessimistic opening attitude and make successive large concessions. t Let pt−1 j→i be the offer that agj has proposed to agi in period t−1. Likewise, let pi→j be the offer that agi is ready to propose in the next time period t. The formal definition of a generic concession strategy follows.
Definition 1. Let agi ∈ Ag be a negotiating agent. A concession strategy for agi is a function that specifies either the tactic to apply at the beginning of negotiation or the tactic that defines the concessions to be made during the course of negotiation: if agi ’s turn and t = 1 apply tact1i def if agi ’s turn and t > 1 conc = apply tactti t if Ui (pt−1 if agj ’s turn j→i ) ≥ Ui (pi→j ) accept else reject where tact1i is an opening negotiation tactic and tactti is a concession tactic.
”
The two aforementioned concession strategies are defined by considering different tactics. For instance, the “starting reasonable and conceding moderately” strategy is defined by: “tact1i = starting realistic” and “tactti = moderate” (but see below). Problem solving behaviour aims at finding agreements that appeal to all sides, both individually and collectively. This behaviour can take several forms, notably logrolling − negotiators agree to trade-off among the issues under consideration so that each party concedes on issues that are of low priority to itself and high priority to the other party [15, 19]. Effective logrolling requires information about the two parties’ priorities so that concessions can be matched up. This information is not always easy to get. The main reason for this is that negotiators often try to conceal their priorities for fear that they will be forced to concede on issues of lesser importance to themselves without receiving any repayment [15]. Despite this, research evidence indicates that it is often not detrimental for negotiators to disclose information that can reveal their priorities − a simple rank order of the issues does not put negotiators at a strategic disadvantage [19]. Hence, we consider that negotiators willingly disclose information that can help to identify their priorities (e.g., their interests). The formal definition of a generic logrolling strategy follows.
Definition 2. Let agi ∈ Ag be a negotiating agent and agj ∈ Ag be its opponent. Let Agenda denote the negotiating agenda, Agenda⊕ the subset of the agenda containing the issues of high priority for agi (and low priority for agj ), and Agenda the subset of the agenda containing the issues of low priority for agi (and high priority for agj ). A logrolling strategy for agi is a function that specifies either the tactic to apply at the beginning of negotiation or the tactics to make trade-offs during the course of negotiation: 1 if agi ’s turn and t = 1 apply tacti ⊕ def t log = apply tacti and tactti if agi ’s turn and t > 1 t if Ui (pt−1 if agj ’s turn j→i ) ≥ Ui (pi→j ) accept else reject ⊕
where tact1i is an opening negotiation tactic, tactti is a concession tactic (to apply to the issues on Agenda⊕ ), and tactti is another concession tactic (to apply to the issues on Agenda ). ” A number of logrolling strategies can be defined simply by considering different tactics. For instance, a strategy that specifies an optimistic opening attitude followed by null concessions on issues on Agenda⊕ and small concessions on issues on ⊕ Agenda is defined by: “tact1i = starting optimistic”, “tactti = stalemate”, and “tactti = tough”. Similarly, a strategy that specifies a realistic opening attitude followed by null concessions on issues on Agenda⊕ and large concessions on issues ⊕ on Agenda is defined by: “tact1i = starting realistic”, “tactti = stalemate”, and “tactti = sof t” (but see below). At this stage, it is worth making the point that logrolling is a major route − though not the only route − to the development of mutually superior solutions (i.e., solutions that are better for all parties). In fact, the host of existing problem solving strategies includes expanding the “pie”, nonspecific compensation, cost cutting, and bridging. These strategies are implemented by different sets of tactics and require progressively more information about the other parties (see, e.g., [15]). Opening negotiation tactics are functions that specify the initial values for each issue isl at stake. The following three tactics are commonly discussed in the behavioral literature [6]: 1. starting optimistic−specifies a value far from the target point; 2. starting realistic−specifies a value close to the target point; 3. starting pessimistic − specifies a value close to the limit. The definition of the tactic “starting realistic” follows (the definition of the other two tactics is essentially identical, and is omitted). Definition 3. Let agi ∈ Ag be a negotiating agent and isl ∈ Agenda a negotiation issue. Let trgil be the target point of agi for isl . The tactic starting realistic for agi is a function that takes isl and trgil as input and returns the initial value v[isl ]1i of isl : starting realistic(isl , trgil ) = v[isl ]1i where v[isl ]1i ∈ [trgil −, trgil +] and > 0 is small.
”
Concession tactics are functions that compute new values for each issue isl . The following five tactics are commonly discussed in the literature [6]: 1. 2. 3. 4. 5.
stalemate − models a null concession on isl ; tough − models a small concession on isl ; moderate − models a moderate concession on isl ; soft − models a large concession on isl ; accommodate − models a complete concession on isl .
The definition of a generic concession tactic follows (without loss of generality, we consider that agi wants to maximize isl ). Definition 4. Let agi ∈ Ag be a negotiating agent, isl ∈ Agenda a negotiation issue, and limil the limit of isl . Let v[isl ]ti be the value of isl offered by agi at period t. A concession tactic for agi is a function that takes v[isl ]ti , limil and the concession factor Cf ∈ [0, 1] as input and returns the new value v[isl ]t+2 of isl : i concession tactic(v[isl ]ti , limil , Cf ) = v[isl ]t+2 i where v[isl ]t+2 = v[isl ]ti − Cf (v[isl ]ti −limil ). i
”
The five tactics are defined by considering different values for Cf . In particular, the stalemate tactic by Cf = 0, the accommodate tactic by Cf = 1, and the other three tactics by different ranges of values for Cf (e.g., the tough tactic by Cf ∈ ]0.00, 0.05], the moderate tactic by Cf ∈ ]0.05, 0.10], and the soft tactic by Cf ∈ ]0.10, 0.15]).
4
Related Work
Artificial intelligence (AI) researchers have investigated the design of agents with negotiation competence from two main perspectives: a theoretical or formal mathematical perspective and a practical or system-building perspective. Researchers following the theoretical perspective have drawn heavily from game-theoretic and economic methods (see, e.g., [1, 5, 17]). Most researchers have primarily focused on formal bargaining, auctions, market-oriented programming, contracting, and coalition formation. On the other hand, researchers following the practical perspective have drawn heavily on social sciences techniques for understanding interaction and negotiation (see, e.g., [4, 10, 16]. Most researchers have primarily focused on the central process of moving toward agreement, notably the design and evaluation of negotiation protocols and negotiation strategies. The theoretical models have some highly desirable properties such as Pareto efficiency, stability, and the ability to guarantee convergence. However, most models work with abstract problems and often fail to capture the richness of detail that would be necessary to successfully apply them in realistic domains. Furthermore, most models make the following restrictive assumptions: (i) the agents are rational, (ii) the set of candidate solutions is fixed and known by all the agents, (iii) each agent knows either the other agents’ potential payoffs for all candidate solutions or the other agents’ potential attitudes toward risk and expected-utility calculations.
Most computational models are being used successfully in a wide variety of real-world domains. These models exhibit the following desirable features: (i) they are based on realistic assumptions, and (iii) they make use of moderate computational resources to find acceptable solutions (according to the principles of bounded rationality [18]). However, most models lack a rigorous theoretical underpinning − they are essentially ad hoc in nature. Also, they often lead to outcomes that are sub-optimal. Finally, there is often no precise understanding of how and why they behave the way they do. Consequently, they need extensive evaluation. Nevertheless, the class of models referred to as computational models is gaining increasing popularity within the mainstream AI community and therefore has received our attention in this paper. Furthermore, despite the aforementioned pieces of work and other relevant models, we are aware of no similar efforts to define strategies as functions that specify the tactics to be used at every period of negotiation. Tactics, in turn, are defined as functions that specify the short-term moves to be made throughout negotiation. Also, our interest lies in formalizing important strategies studied in the social sciences and frequently used by human negotiators, and in evaluating the effectiveness of these strategies in different situations.
5
Conclusion
This article has presented a simplified multi-agent energy market composed of a collection of autonomous computational agents, each responsible for one or more market functions, and each interacting with other agents in the execution of their responsibilities. Additionally, the article has presented a negotiation model for autonomous agents that handles bilateral multi-issue negotiation and formalizes a set of negotiation strategies studied in the social sciences and frequently used by human negotiators. The strategies are based on rules-of-thumb distilled from behavioral negotiation theory. Autonomous agents equipped with the negotiation model are currently being developed using the Jade framework and the Java programming language. Our aim for the future is to perform a set of inter-related experiments to empirically evaluate the key components of the agents operating in the energy market. Each experiment will lay the foundation for subsequent experimental work. Also, the current work forms a basis for the development of more sophisticated agents that are able to negotiate under both complete and incomplete information. In particular, we intend to study the bargaining game of alternating offers in order to define equilibrium strategies for two incompletely informed players.
References 1. Fatima S., Wooldridge M., Jennings N.: A Comparative Study of Game Theoretic and Evolutionary Models of Bargaining for Software Agents. Artificial Intelligence Review 23:185–203 (2005) 2. Gustavsson R.: Agents with Power. Communications of the ACM 42:41–47 (1999)
3. Jennings N., Sycara K., Wooldridge M.: A Roadmap of Agent Research and Development. Autonomous Agents and Multi-Agent Systems 1:7–38 (1998) 4. Jennings N., Faratin P., Lomuscio A., Parsons S., Wooldridge M., Sierra C.: Automated Negotiation: Prospects, Methods and Challenges. Group Decision and Negotiation 10:199– 215 (2001) 5. Kraus S.: Strategic Negotiation in Multi-Agent Environments. MIT Press, Cambridge (2001) 6. Lewicki R., Barry B., Saunders D., Minton J.: Negotiation. McGraw Hill, New York (2003) 7. Lopes F., Mamede N., Novais A. Q., Coelho H.: A Negotiation Model for Autonomous Computational Agents: Formal Description and Empirical Evaluation. Journal of Intelligent&Fuzzy Systems 12:195–212 (2002) 8. Lopes F., Mamede N., Novais A. Q., Coelho H.: Negotiation Strategies for Autonomous Computational Agents. In: ECAI 2004, pp 38–42. IOS Press (2004) 9. Lopes F., Novais A. Q., Coelho H.: Towards an Interdisciplinary Framework for Automated Negotiation. In: E-Commerce and Web Technologies (LNCS 5138), pp. 81–91. Springer, Heidelberg (2008) 10. Lopes F., Wooldridge M., Novais A. Q.: Negotiation Among Autonomous Computational Agents: Principles, Analysis and Challenges. Artificial Intelligence Review (in press, 2009) 11. Osborne M., A. Rubinstein A.: Bargaining and Markets. Academic Press, London (1990) 12. Parunak H.: A Practitioners Review of Industrial Agent Applications. Autonomous Agents and Multi-Agent Systems 3:389–407 (2000) 13. Pechoucek M., Vladimir Marik V.: Industrial Deployment of Multi-agent Technologies: Review and Selected Case Studies. Autonomous Agents and Multi-Agent Systems 17:397– 431 (2008) 14. Penya Y.: Optimal allocation and Scheduling of Demand in Deregulated Energy Markets. Dissertation, Vienna University of Technology, Austria (2006) 15. Pruitt D., Kim S.: Social Conflict: Escalation, Stalemate, and Settlement. McGraw Hill, New York (2004) 16. Rahwan I., Ramchurn S., Jennings N., McBurney P., Parsons S., Sonenberg L.: Argumentation-based Negotiation. The Knowledge Engineering Review, 18:343–375 (2004) 17. Sandholm T.: Distributed Rational Decision Making. In: MultiAgent Systems – A Modern Approach to Distributed Artificial Intelligence, pp 201–259. MIT Press, Cambridge (1999) 18. Simon H.: The Sciences of the Artificial. MIT Press, Cambridge (1981) 19. Thompson L.: The Mind and Heart of the Negotiator. Prentice-Hall, Englewood Cliffs (2005) 20. Ygge F.: Market-Oriented Programming and its Application to Power Load management. Dissertation, Lund University, Sweden (1998)
