Investigating Trust between Users and Agents in A Multi ... - CiteSeerX

Investigating Trust between Users and Agents in A Multi Agent Portfolio Management System: a Preliminary Report Tiffany Y. Tang, Pinata Winoto and XiaoLin Niu Department of Computer Science University of Saskatchewan 57 Campus Drive, Saskatoon Saskatchewan, S7N 5A9 Canada {yat751, piw410, xin978}@mail.usask.ca

Abstract We have witnessed considerable research investigating trust between agents in multi-agent systems. However, the issue of trust between agents and users has rarely been reported in the literature. In this paper, we describe our experiences with ITRUST, a multi-agent artificial market system whose software broker agent can learn to build a relatively long-term trust relationship with their clients. The goals of these broker agents are not only to maximize the total revenue subject to their clients’ risk preference as most other agents do in [10, 14, 17], but also to maximize the trust they receive from their clients. Trust is introduced into I-TRUST as a relationship between clients and their software broker agents in terms of the amount of money they are willing to give to these agents to invest on their behalf. To achieve this, broker agents first elicit user models explicitly through questionnaires and implicitly through three games. Then based on the initial user models, they will learn to invest and later update the models when necessary. Keywords: multi-agent systems, reinforcement learning, trust, user modeling, portfolio management

1.

Introduction

Trust is multi-dimension which concerns many different attributes such as reliability, dependability, security, honesty, competence, etc, which may have to be addressed based upon the environment where it is specified [7]. It has been studied extensively in Multi-agent Systems [5, 8, 11, 12, 13, 16], where it is the “attitude an agent has with respect to the dependability/capabilities of some other agent (maybe itself)” [8]. However, dealing with trust between humans and agents in multi-agent e-commerce system is also important because in order to attract more clients to take advantage of the services they provide, wining the trust of their clients becomes central. As Kini and Choobinch put it that trust in a system involves “a belief that is influenced by the individual’s opinion about certain critical system features” [9]. This definition highlights human trust towards agents in electronic commerce, which motivates our study in this paper. In particular, we address the issue of the degree of trust a client has towards her/his broker agent to invest on her/his behalf: how can a client be sure that the broker agent will make a sound judgment based upon the risk-return preference of her/him. In the context of I-TRUST, trust relationships are different from those derived from the human-computer intersectional design perspective where trust is communicative in MoTEC [4]. To describe our approach, let us first look at two examples. An example Suppose Bank A offers customers a 24-hour on-line portfolio management service (investment in stocks, bonds, mutual funds, etc.) where one software agent will represent one customer to invest/manage her/his portfolio1. For Bank A, the overall goal is to attract as many customers as possible along with their optimal investment, and through that, receive higher revenues. Therefore, all the broker agents belonging to the bank are expected to cooperate to convince their respective clients that they are trustworthy2. As a result, customers will be willing to rely on them and invest more money into the market. From the company’s perspective, the cooperation between broker agents is the expected collective behaviors; whereas, for 1

A client can be an individual investor or a business investor representing a company. Of course, for each broker agent, she/he might have his own self-interested goal to maximize his own revenue by building strong trust between him and his client(s), and through that, attracting more customers. But in the context of this paper, we only study the cooperation among agents to achieve a common goal. 2

each individual broker agent, she/he must also do her/his own jobs to build trust of her/his client. ! Another example For busy customers, they cannot afford to monitor stock market and respond it themselves. Therefore, there is a growing trend for these customers to delegate (human) agents to manage their portfolio on their behalves. For retired customers, on the other hand, they are willing to spend time monitoring stock market and invest themselves. However, they are relatively more cautious. Therefore it will be in their interest if they can consult their plans with their agents more frequently anytime through Internet. In both cases, online software agents can be very useful if their customers can trust them. ! These two examples highlight an important issue facing agent-based e-commerce systems: win the trust of their clients and thus attract more customers to take advantage of the services they provide. It is the theme of our study in this paper: investigating trust between software broker agents and their clients in the context of an artificial stock market. Specifically, our study is based on a multi-agent portfolio management system called I-TRUST where each broker agent represents a client to invest on his/her behalf in the market based on the client’s characteristics (esp. risk-return preference). After each investment period, a broker agent will give an investment report to his/her client, and the client will evaluate and rate it. A broker agent needs not only to maximize the total revenue subject to her/his client’s characteristics, but also has to learn the best portfolio selection strategy so as to attract her/his client to follow her/his expertise, e.g., invest more money in the next investment period. The higher amount of money a client is willing to put into the market, a higher degree of trust she/he has towards her/his broker agent. Therefore, the degree of trust between a client and her/his broker agent is measured in terms of the amount of money a broker agent invest on her/his client’s behalf. This trust can also be regarded as the degree of the client’s reliability on the capabilities/competence of a broker agent and the service the agent controls or provides. It can be seen that this trust relationship is difficult to create; yet very easy to lose, therefore a broker agent has to learn to elicit her/his client risk-return preference model so as to make a best investment strategy. From the perspective of the client, she/he implements a partial trust towards her/his client, which is different from most of current studies of this kind [3, 10, 14, 17]. In [3, 10, 14], agents are fully delegated to make decisions in the artificial stock market; while in [17], users have no trust at all towards their agents: a broker agent acts only as an information agent to collect relevant information, and the ‘intelligent client’ will make final decisions on their own. Therefore, the client implements no delegation at all to her/his broker agents. However, we argue that it is reasonable to add another layer of delegation (partial trust) into the trust relationships because it can generalize trust into a wider and general spectrum by introducing trust between human and agents. The understanding of this kind of relationship can make an e-commerce system more successful, profitable, since the underlying aim of electronic commerce is to attract more users to take advantage of the trusted high-quality services the system can provide. In [6], the authors analyze trust, autonomy and delegation in multi-agent system and a framework is given for the theory of adjustable social autonomy in complex scenario. Our reported work in this paper can be regarded as investigating trust, autonomy and delegation in a specific domain and expanding it when necessary. Several assumptions are made for this study: • Broker agents are benevolent (i.e. they will not cheat their clients!) • Each client can only have one broker agent at one time The organization of this paper is as follows: in the next section, an overview of the system architecture is given. We will then describe how to elicit initial user models through one questionnaire and three-game-playing. And a brief discussion of trust building strategy broker agents adopt is given. We will then focus on the experiments conducted so far followed by some discussions. We conclude this paper by pointing out our current as well as future research. 1.2 System Architecture Trust building for I-TRUST begins with an initial-user-mode-eliciting process, during which users fill out one questionnaire and then play three games. The result of these activities is constructed as an initial user model. This process is consistent with real-life scenarios, when a client requests to open a trading account, he/she will be required to fill in questionnaires for the bank. After that, broker agents must adjust their portfolio-managing strategy based upon the feedback (including money delegated to the agent and instructions, if any) her/his client provides after each pre-defined investment period. Figure 1 illustrates the structure of I-TRUST. We assume that our artificial stock market only consists of few stocks and does not allow short selling or options.

Broker agent KB

User models

Money & Instruction Stock info.

Figure 1. System architecture for I-Trust

2

Eliciting User Models: Combining Questionnaires and Games

Building a client’s model is crucial to the decision making of a broker agent to act optimally; a model of a client’s riskreturn preferences is required. User models, in our domain, may seek to identify user’s behavioral patterns and risk-return preferences. Generally it is assumed that user models are built manually, or at least the complete attributes necessary to build the user model is given as input. Thus, what the system needs to do is to calculate it based on some pre-defined predictive techniques. In other words, the expected utility of a client is a priori known (e.g., in terms of mean-variance) and this utility function is regarded as the user model based upon an optimal portfolio selection is made. However, the elicitation of user models needs a lot of information from the user. An alternative proposed is to generate initial user models both explicitly through questionnaires and implicitly through game playing scenarios (currently we use three games only). Then, a more detailed user model will be inferred automatically over multiple interactions between broker agent and a client. Generally, users are required to fill in some forms giving out their preferences explicitly. However, this is not always accurate given user’s misperceptions of their own behavior and the ambiguities inherent in any kind of qualitative elicitation of knowledge. Thus, in I-TRUST, in additions to the questionnaires there are three game-playing scenarios related to our problem domain (i.e. stock portfolio management) for the users to take part in. The game-playing scenario (with salient rewards) for eliciting users’ preferences has been adopted extensively in the experimental economics community (cf. [1, 2]), and has proven to be a more accurate approach to infer users’ preferences compared to a pure questionnaire [2, 15]. In fact, the introduction of these game-playing scenarios to form an initial user model is a special feature of I-TRUST. Figure 2 shows one of the game-playing scenarios-game C. It is designed to investigate a client’s tolerance towards loss. In particular, in time t, a client is asked to observe one stock’s wave pattern (information part). she/he must then decide whether she/he intends to sell her/his stock given this wave pattern. Six decisions will be made in this game. At one time, the stock might plunge, but it is unclear whether it is the lowest point or not. Thus, the user must decide herself/himself that whether she/he wants to sell at this time or not. (In a real-life scenario, a client is explicitly asked in questionnaires what is the maximum amount of money (in percentage) she/he will tolerate to lose). Figure 3 is the screenshot for Game B in I-TRUST. In this game, a client is requested to choose one of the five options listed in the left hand side of the lower screen. If she/he wins, she/he will get $500, but if she/he loses, she/he will get nothing. In order to increase the chances of winning, she/he might choose to buy information listed in the right hand side of the lower screen. But when each time she/he buys one piece of information, $100 is deducted from her/his prizes. The game implicitly infers clients’ trust towards hints/inside information she/he can collect (e.g. released by her/his own brokers, or other people). For some people, they might disregard the ‘inside information’ and just gambles.

Instructions to the user

Information part

Figure 2. A snapshot of Game C for I-TRUST

Figure 3. A snapshot of Game B for I-TRUST Formally, user model U j=={RW j, EU j }, where j represents the j-th client, combines the expected utility of this client (i.e. risk-return preferences) and the reward RWj she/he gives out. Therefore, from the agent’s perspective, in addition to maximize the total revenue and minimize risks for her/his client, a broker agent should also maximize the reward collected, which in turn determines the portfolio selections and thus influences her/his client’s expected utility EUj. In our user model, the client’s expected utility does not necessarily mean the vonNeumann-Morgenstern expected utility, instead it captures a broader notion of the mapping from her/his risk-return preference to portfolio preference. Agent elicits her/his client’s riskreturn preference from the initial questionnaire and games and then uses it to select portfolio. Ideally, we could categorize every user based upon his/her attributes, such as risk-return preferences, patience toward future income (inter-temporal substitution), tolerance toward loss, age group, income group, etc. In our current work, we combine all information into a one-dimensional metric: user_type. The formula is as follow, user_type = w’• x

where w is the weight vector satisfying Σiw = 1 and x is the result vector, and user_type∈[0,1].

3. Winning the trust of client: single agent reinforcement learning In this section we will briefly describe how agents adopt reinforcement learning technique to learn to win the trust of her/his client. Specifically, how a broker agent analyzes the feedback from her/his client, and adjusts her/his portfolio selection strategy to collect higher rewards (trust) from the client. In this environment, the broker agent will learn what to do, i.e. how to map situations to actions, so as to maximize a reward from the environment (i.e. ratings from her/his client in terms of the amount of money the client is willing to invest for the next investment period) [19]3. Figure 4 illustrates the single-agentreinforcement learning process. Note here that the dashed arrows show agents’ interactions with I-TRUST. Although a client might give inconsistent rewards to her/his agent over some investment periods, which might indicate that the client’s risk attitudes might have changed. However, we will not consider it till it exceeds some threshold. It is consistent with real-life experiences when you (risk averse) find yourself angry when you suffer a loss because you think that your agent is too cautious, which does not necessarily mean that you are becoming risky. In I-TRUST, only after inconsistent rewards are significantly observed will the agent consider that her/his client has become more risky and update the user model. User modeling

Portfolio selections Stock info. User evaluations

reinforcement learning by broker agents

Figure 4. A single-agent-reinforcement learning process A formal model of the single-agent reinforcement learning mechanism can be found in [21].

4. Experimentations and Proof of Concept In order to verify our model, we start with three experiments for the purpose of ‘proof of concepts’ (trust between humans and agents). Specifically, we shall attempt to investigate whether trust is build upon based on any of the following features or both: • Client’s trust in agent’s expertise • Whether the agent’s portfolio selection strategy Pf should be more customer oriented (i.e. based more on Pu ) In order to investigate it, we design three experiments. In this paper, we will only describe experiment 3 in some details. A more detailed analysis and evaluations for I-TRUST can be found in [21]. Experiment 1 (trust without delegation) In the first experiment, the final portfolio is selected according to the client, although her/his broker agent will also propose her/his own, which, of course, could be adopted by the client. After each round, a brief report is given out to the client listing the final profit made 3

π tf , and the profit made by following agent’s advice π ta . Clients are expected to compare π tf and

In MAS, reinforcement learning techniques have been studied extensively (e.g. [18, 20]).

π ta . Here, the measure of trust is based on the deviation of client’s decision towards agent’s advice. For instance, if agent’s opinion is to buy $5000 worth of stocks ‘A’ and her/his client only agrees to buy $2000, then the trust level towards the agent is set to 0.7 (complete trust = 1, distrust = 0, it is determined subjectively). However, if client chooses to sell all her/his stocks ‘A’, then the trust level towards her/his agent is set to 0. Experiment 2 (trust with full autonomy) In the second experiment, an agent selects the final portfolio and a client cannot intervene with agent’s decision. Agent’s decision is merely to find optimal portfolio investment based on client’s risk-return preference (user model) and market situation (environment). After each round, a brief report is given out to the client listing the final profit made (feedback). Then, the client decides whether to add/withdraw money to/from her/his agent or takes no actions. This is the only control implemented by a client. The measure of trust is based on how much money the client is willing to give to her/his agent in run-time adjustment. It is known that clients can punish their agents by reducing the money they are willing to invest for the next investment period. Experiment 3 (delegation with mixed control) In the third experiment, a client delegates her/his agent a certain amount of money and then control her/his agent’s behavior by adjusting the amount of money delegated and suggesting her/his preferred portfolio. However, an agent may not follow the client’s suggestion on portfolio selection but must follow her/his instruction on amount of money for the next investment period (add/withdraw money). Both of these would be interpreted by agent as the trust given by the client. Thus, agent will use the rewards to adjust her/his investment policy (in terms of α in convex combination of Pa and Pu) by means of reinforcement learning technique described in previous section. Figure 5 shows one of the adjustment processes of α over time. The value of α is directly influenced by client’s rewards (feedback), therefore can be used to measure the trust. α=0 is reached iff the client always agrees with her/his agent’s opinion and ‘dumps’ all her/his money to agent (trust = 1). And α=1 iff the client does not agree with her/his agent’s opinion at all or withdraws all her/his money (trust = 0). From the third experiment the average value of α ≈ 0.2.

0.4 0.35 0.3

Alpha

0.25 0.2 0.15 0.1 0.05 0 1

2

3

4

5

6

7

8

9

10

11 round

Figure 5. The learning curve of agent in the third experiment

profit (normalized)

1

profit (agent) Final profit

profit (c lient)

0.5 0 -0.5

1

2

3

4

5

6

7

8

9

10

-1 -1.5

Figure 6. Risk sharing from the third experiment

Unfortunately, most of the experiments only run for 10 until 20 periods, which is too short to be contributed to the justification of the stability of learning parameters (long term relation.). However, we are convinced that this setting is better than settings in experiment 1 and 2 for two reasons: 1. Agent can learn client’s behavior (from client’s feedback), thus improve the flexibility of the system to prevent the breakdown of the client-agent relationship. From our preliminary experiment, none of the users withdraw all her/his money as what happen in experiment 2. In practice, users in this experiment have more control compared to those in experiment 2, which in turn reduce their reluctance to use the system. 2. The autonomy of agent in decision-making could help clients who are busy (unwilling to intervene agent frequently) management their portfolios. And the system could reduce the risk of investment by considering both agent and client’s opinions (using convex combination method), which in turn could reduce the error which might made by any party (see figure 6). It should be mentioned here that so far, our results should not be interpreted as a statistically significance, but only as a ‘proof of concept’.

5. Discussions • Trust models in I-TRUST and MoTEC I-TRUST is different from MoTEC trust model [4] in that the various human factors affecting trust in MoTEC contribute to a relatively short-term trust relationship. According to [4], the MoTEC trust components include pre-purchase knowledge (vendor’s reputation), interface properties (familiarity, usability of the interface) and information contents (for example, “transparency of the vendor’s openness with respect to business policies”). It is obvious that this kind of trust-facilitating user interface might affect clients’ trust towards the services a vendor provides in the short-run. However, in the long run, the most enduring trust relationships come from the quality of the services (for example, in our domain, how reliable and profitable broker agents’ portfolio management strategies are, not how the interface would affect clients). Thus, what we concern is the quality of the services e-commerce vendors can provide to their clients rather than the merely selling of the interface in order gain a relatively long-term trust building between clients and vendors. Although trust-facilitating user interface might affect clients’ trust. Thus, in their view, for B2C e-commerce, trust is not communicated as claimed in [4], but evolved. • Trust and punishment One important issue associated with trust is punishment. In I-Trust, clients can punish their broker agents by reducing the amount of money delegated to agents to invest in the next investment period. From agents’ perspective, they will try to avoid this by not only investing to the best of their expertise, but also considering their clients’ risk preferences. For example, in one specific investment period, agents might consider buying $5000-worth of stock A after a careful estimation of market information, however, his/her client might disagree and suggests buying only $2000. In this case, broker agents must sacrifice her/his expertise and follow, to some degree, her/his client’s advice, and thus, invest, say, $2500-worth of stock A. Therefore, broker agents might try to balance their investing expertise and feedback from her/his client. Results of the third experiments show that broker agents struggles

6. Concluding remarks and future works As noticed from the real-life example, currently we only considered single agent’s learning. We are studying the cooperation between broker agents to win the trust of their client. Besides, all subjects voluntarily took part in the experiments, thus a bias is expected to be observed in the results of all experiments (without salient rewards). Therefore, a large-scale experiment with salient rewards is expected to be conducted as our future study. It is our hope that our work here will benefit both academic and e-commerce practitioners alike.

Acknowledgements We are indebted to our supervisor Gordon McCalla for his many insightful comments and suggestions; Pinata and Xiaolin’s supervisor Julita Vassileva for her support, and valuable inputs. And we appreciate all the volunteers for taking their time to take part in our experiments. We would like to thank the Canadian Natural Sciences and Engineering Research Council for funding Tiffany Y. Tang and Pinata Winoto through research assistantships and the University of Saskatchewan for funding Xiaolin Niu through a graduate teaching fellowship.

Appendix

A snapshot for market information (Left) and the portfolio report (right) in Experiment 2

A snapshot for market information (Left) and the portfolio report (right) in Experiment 3

References [1] C. Camerer, Individual decision making, The Handbook of Experimental Economics, J.H. Kagel and A.E. Roth (eds.), (Princeton University Press, 1995), 587-703. [2] D. D.Davis and C.A. Holt, Experimental Economics. (Princeton University Press, 1993).

[3] K. Decker, K. Sycara, A. Pannu and M. Williamson, Designing behaviors for information agents, in: Proc. AGENTS’97, (ACM Press, 1997). [4] F.N. Egger, "Trust Me, I'm an Online Vendor": Towards a Model of Trust for E-Commerce System Design. In: G. Szwillus and T. Turner (Eds.), CHI2000 Extended Abstracts: Conference on Human Factors in Computing Systems, The Hague (The Netherlands), April 1-6, 2000: 101-102. [5] G. Elfoson, Developing trust with intelligent agents: an exploratory study, in: Proceedings of the 1st Int’l Workshops on Trust, 125-138, 1998. [6] R. Falcone, and C. Castelfranchi, The human in the loop of a delegated agent: the theory of adjustable social autonomy, IEEE Trans. on Systems, Man and Cybernetics, Part A: Systems and Humans, 31 (5), 2001. [7] T. Grandison and M.Sloman, A survey of trust in Internet application, IEEE Communications Surveys, Fourth Quarter, 2000. [8] C.M. Jonker and J. Treur, Formal analysis of models for the dynamics of trust based on experiences, in: Proceedings of MAAMAW’99, LNAI 1647, (Springer-Verlag, 1999). [9] A. Kini, and J. Choobinch, Trust in electronic commerce: definitions and theoretical considerations, in: Proceedings of the 31st Hawaii Int’l Conf. System Sciences, Hawaii, 1998. [10] B. LeBaron, W.B. Arthur, J.H. Holland, R. Palmer and P. Tayler, Asset pricing under endogenous expectations in an artificial stock market”, The Economy as an Evolving Complex System II, W.B. Arthur et al (eds.), (Addison-Wesley, Reading 1997). [11] S. Marsh, Trust and reliance in multi-agent systems: a preliminary report, in: Proceedings of the 4th European Workshop on Modeling Autonomous Agents in a Multi-Agent World (MAAMAW’92), Rome. [12] S. Marsh, Formalizing trust as a computational concept, PhD Thesis, Department of Mathematics and Computer Science, University of Stirling, UK, 1994a. [13] S. Marsh, Trust in distributed artificial intelligence, Artificial Social Systems, C. Castelfranchi, and E. Werner (eds), LNAI 830, (Springer-Verlag, Berlin, 1994b), 94-112. [14] D.C. Parkes and B.A. Huberman, Multi-agent cooperative search for portfolio selection, Games & Economic Behavior, Special issue on artificial intelligence and economics, 35 (2001), 124-165. [15] A.E. Roth, Introduction to experimental economics, in The Handbook of Experimental Economics, J.H. Kagel and A.E. Roth (eds.), 3-109, (Princeton University Press,1995). [16] M. Schillo, P. Funk, and M. Rovatsos, Using trust for detecting deceitful agents artificial socities, in Applied Artificial Intelligence Journal, Special Issue edited by C. Castelfranci, Y.Tan, R. Fancone, and B. Firozabadi, on Deception, Fraud and Trust in Agent Societies, 2000. [17] M. Schroeder, J. McCann, and D. Haynes, Rough traders and intelligent clients, in: Proceedings of the International Conferences on Autonomous Agents (Agents’2000), Barcelona, Spain, (ACM Press, 2000), 259-260. [18] P. Stone and M. Veloso, Multiagent systems: a survey from a machine learning perspective, Autonomous Robots, 8 (3), 2000. [19] R.S. Sutton and A.G. Barto, Reinforcement learning: an introduction. (MIT Press, 1998). [20] M. Tan, Multi-agent reinforcement learning: independent vs. cooperative agents, in: Proceedings of the 10th International Conference on Machine Learning (ICML’ 93), 330-337. [21] T.Y. Tang, P. Winoto and X.L. Niu, Who can I trust? Investigating trust between users and agents in a multi-agent portfolio management system. To appear, in the AAAI-02 Workshop on Autonomy, Delegation, and Control: From Interagent to Groups, (AAAI, 2002).