AN AGENT-BASED FUZZY RECOMMENDATION ... - Semantic Scholar

International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems Vol. 18, No. 4 (2010) 471−492 © World Scientific Publishing Company DOI: 10.1142/S0218488510006659

AN AGENT-BASED FUZZY RECOMMENDATION SYSTEM USING SHOPPERS’ PREFERENCES FOR E-COMMERCE APPLICATIONS ZHENGPING WU∗ and HAO WU† Department of Computer Science and Engineering, University of Bridgeport, 221 University Avenue, Bridgeport, CT 06604, USA ∗ [email protected] † [email protected] Received 30 January 2010 Revised 15 May 2010 Accepted 5 June 2010 In e-commerce applications, the magnitude of products and the diversity of venders cause confusion and difficulty for the common consumer to choose the right product from a trustworthy vender. Although people have recognized the importance of feedback and reputation for the trustworthiness of individual venders and products, they still have difficulty when they have to make a shopping decision from a massive number of options. This paper introduces fuzzy logic into rule definition for users’ preferences and designs a novel agent-based decision system using fuzzy rules. This system can help users to find the right product recommendation from a trustworthy vender following users’ own preferences. Keywords: Decision-making; fuzzy logic; agent; recommendation; e-commerce.

1. Introduction Trust and trustworthiness have been discussed in the e-commerce environment for many years.1 The importance of trust management in the e-commerce environment is obvious. From end users’ points of view, the trust or trustworthiness toward a vender includes many factors.2 Due to the diversity of these factors, management of trust in the e-commerce environment is very difficult. The research of trust management in related areas such as studies in the context of access control,3,23 public key architecture,4,24 and reputation systems for peer-to-peer networks,5,6 can help us understand key points in trust management for e-commerce. Thus, in the e-commerce environment, similar criteria can be borrowed to increase end users’ confidence in online shopping decisions, such that “an e-commerce site should provide secure transactions for online payment,” “an ecommerce site’s reputation should be monitored by an authority,” “the reputation of a vender in an e-commerce site should be directly related to the number of its customers’ feedbacks” and, “the reputation of a product in an e-commerce site should be directly related to its customers’ feedback scores.” Management of various trust factors enables us to increase our confidence in online shopping activities and prevent ambiguity in shopping decisions. From the end user’s viewpoint, an online shopping decision is very difficult to make due to these various trust factors plus various price factors. An end user may choose a 471

472 Z.-P. Wu & H. Wu

favorite brand first. The brand may include various types of products. It is difficult to decide the brand’s overall trustworthiness using an aggregated reputation from its various types of products. Even if a brand is selected, choosing a product is also difficult for an end user because the trustworthiness of that specific product cannot be easily determined from available information. An end user may employ the volume of feedback or the scores from previous consumers to determine this trustworthiness, but this mixed feedback is difficult to qualify and quantify, not to mention aggregate for a final decision. In terms of the price factor, an end user may have various price choices from different vendors. Finally, this user also needs to evaluate the trustworthiness of different venders to ensure that his or her purchase is at a low price from a trusted vender with decent services such as shipping and handling. So the overall decision for an online purchase will become totally uncontrollable. The major reasons for this difficulty are that there are so many uncertain factors in this decision process and there are so many options of different venders and opinions from previous shoppers that influence this decision process. It is a time-consuming and an almost impossible task for the common shopper to review and evaluate all these factors and make a decision. In order to manage a collection of uncertain factors in the decision-making process for online shopping, we need to provide a flexible platform that would include various uncertain factors and manage these factors, following end users’ preferences. For example, there are many comparison shopping sites, such as dealtimes.com and pricegrabber.com, which provide a list of venders selling the product searched for by a user through a user query. Although these websites list prices and venders’ reputations, the final result lists are simply ordered by price or by following these websites’ own ranking mechanisms. Even if an end user prefers to choose a product from a trustable vender without considering the price factor, or prefers to buy from a trustable vender with a reasonable price, he or she may not get a list that reflects his or her preferences. So the novel, agent-based system proposed and implemented for shopping decision-making discussed in this paper is designed to help end users make shopping decisions, according to their own shopping preferences. Since fuzzy logic has the capability to quantify imprecise data or uncertainty, and it can handle multiple factors. The introduction of fuzzy logic into this system will help users manage various uncertain factors in their online shopping activities. Furthermore, agents can search and compare a user’s favorite product among different e-commerce sites and provide the most attractive offers to that end user, which is impossible to do manually. 2. Related Work 2.1. Trust models Trust is a complex subject related to belief in honesty, trustfulness, competence, and reliability of an entity. In McKnight et al.’s work,8 the most tangible aspects of trust are trust behavior and trust intention. In the context of e-commerce, trust is usually specified in terms of a temporary relationship between a consumer and a vender or product. In this relationship, trust intention formed from a number of fuzzy factors such as an acceptable price range and a reasonable vender’s reputation in the decision process leads to the actual trust behavior (purchases). In today’s e-commerce environment, the management of trust needs to deal with many factors over multiple websites or domains. Beth et al.9 categorizes inter-domain trust relationships into two classes: direct trust and recommended trust. A trust relationship formed from direct experience or negotiations

An Agent-Based Fuzzy Recommendation System 473

can be characterized as direct trust; a trust relationship or a potential trust relationship built from recommendations by a trusted third party or a chain of trusted parties, which create a trust path, is called recommended trust. Based on the expectation of an entity being able to finish a task, the system can calculate the probability of whether the entity will complete the task based upon positive and negative experiences, measure the trustworthiness using this probability, and create a formula for calculating a number value for trustworthiness with a set of derivation and integration rules. But this mechanism simplifies real life by modeling trustworthiness based only on probability, and equates the subjectivity and uncertainty to randomness. At the same time, it uses the mean value of multiple sources of trustworthiness as the indicator of aggregate trust and the final trust value number, which omits possible weights on each trust source. In Ref. 10, Josang proposes a trust model based on subjective logic, which introduces the concepts of evidence space and opinion space to describe and measure trustworthiness. Based upon the Beta distribution function that describes the posteriori probability for binary events, the author calculates the trustworthiness for every possible event from every entity. Meanwhile, Josang defines a set of operators for the calculation of trustworthiness. Josang’s model also literally equates the subjectivity and uncertainty to randomness. The subjectivity and uncertainty of trustworthiness as a cognitive activity is mainly expressed in its fuzziness. The challenge is how to model this fuzziness and apply this model to the management of multiple factors in e-commerce activities. 2.2. E-commerce applications There are three major e-commerce categories: business to business (B2B), business to consumer (B2C), and consumer to consumer (C2C). Since this paper focuses on personal online shopping activities, B2B e-commerce applications will not be discussed. B2C is the earliest e-commerce category with slow growth,11 B2C e-commerce has been widely adopted by consumers since most of the B2C websites, such as buy.com, bestbuy.com and walmart.com, not only have a long and steady business history but also have brickand-mortar stores. The existence of actual stores can increase a shopper’s confidence to shop at the store’s B2C website. On the other hand, C2C websites, such as eBay.com25 and Amazon marketplace,26 have become the most popular type of e-commerce in recent years due to convenience and flexibility.12 Everyone is able to participate as a buyer and/or a seller on these websites. However, C2C platforms are not able to provide the sense of a sufficiently secure and confident shopping environment for all online shoppers. Consumers can barely distinguish between untrustworthy and reliable venders. Although C2C websites have begun providing feedback systems to evaluate vender reputation, it is still difficult for consumers to make decisions based on different opinions and uncertainty. Nowadays, comparison shopping websites (illustrated in Fig. 1) are able to list a number of venders that they have assembled from the Internet for consumers to compare shopping options. When users search for a product in these websites, the websites will return a list of venders that sell that product and the selling prices of the product. But few of them can compare C2C venders with B2C shopping sites, which typically are equally viewed by shoppers. Additionally, there is no reputation calculation standard for reputation evaluation on different types of websites. Each comparison shopping website provides its own reputation evaluation mechanism, which is not clear to consumers. In recent years, some new researchers have discussed decision making and user preferences in e-commerce environments,18,19 but similar to comparison shopping sites, they still

474 Z.-P. Wu & H. Wu

cannot help common users choose suitable products, based on their own preferences, from both B2C and C2C websites. So we propose an agent-based, personalized recommendation system to help consumers choose the right products from both B2C and C2C websites.

Fig. 1. Comparison Shopping Websites (Yahoo! Shopping.26 Dealtime,27 Pricagrabber28)

Fig. 1. Comparison Shopping Websites (Yahoo! Shopping,26 Dealtime27 and Pricagrabber.28

2.3. Online reputation The online reputation system, also known as the online feedback-based rating system, is a system through which individual shoppers can share opinions and experiences through rating products, venders, websites, or other individuals.13 In e-commerce environments, consumers can rate products by quality, performance and product price; consumers can also rate venders so that other potential consumers are informed of vender credibility and service quality; and venders can also evaluate consumers to rate consumer reliability in transactions. As the largest C2C website, eBay provides a complete rating system for consumers and venders to evaluate products and each other. A potential consumer can manually evaluate a vender’s reputation according to positive feedback, feedback scores, and detailed feedback. However, some eBay users intentionally manipulate their reputations by buying and selling feedback.14 These reputations and trustworthiness of


venders become weakly reliable. For B2C venders, third-party organizations, such as Better Business Bureau30 and epinions,31 document their reputations. For C2C websites, we will use the model proposed below to evaluate vender reputation. 3. Model of Uncertainty and Trustworthiness 3.1. Categorization of uncertainty In order to manage a collection of trustworthy-based shopping activities over multiple ecommerce websites, we need to understand trustworthiness itself. From different points of view, trust can be categorized into different classes. Following the categorization criteria described by Beth et al.,9 we categorize trust into two classes − direct trust and indirect trust. A trust relationship formed from direct experience or negotiations can be characterized as direct trust; a trust relationship or a potential trust relationship built from recommendations by a trusted third party or a chain of trusted parties, which creates a trust path, is called indirect trust. Indirect trust is derived from direct trust. Indirect trust is actually a function of direct trust(s). It may add new values to direct or indirect trust from a trusted party. The added values are uncertain at some level and tend to be fuzzy. From another point of view, trust is a concept everyone understands at some personal level, but most people have difficulty providing a specific definition of this concept. Some people have objective measures they use to evaluate their level of trust in a person or company, while others rely on a more subjective feeling to determine whether or not to trust someone. So trust can be either derived from one’s belief or feeling, or can be based on an evaluation founded on specific measurements. An entity’s (brand’s or vender’s) trustworthiness is associated with the quality of services or products it provides to others. The quality of a service or product can be either objectively or subjectively measured. Intuitively, if the quality of a service or product can be objectively measured, then the trustworthiness toward that service or product reflects some intrinsic property, which should be independent of the source of the trust evaluation. However, subjective trust may vary greatly when different sources of trust evaluation are considered. Due to this variation, subjective trust is uncertain at some level, and therefore needs special representation and enforcement processes to handle it. 3.2. Fuzzy model of uncertainty The trustworthiness in cross-site e-commerce applications is hard to assess due to involved uncertainty. If trustworthiness relies upon a subjective judgment based on indirect information, it will be very uncertain and any activities based on that trustworthiness may cause unexpected results such as bad service or bad products. Fuzzy logic is a suitable way to represent uncertainties, especially when they need to be handled quantitatively. Two advantages of using fuzzy logic to quantify uncertainty in trustworthiness management are: (1) fuzzy inference is capable of quantifying imprecise data or uncertainty in measuring different levels of trustworthiness; and (2) different membership functions and inference rules can be developed for different types of trustworthiness without changing the fuzzy inference engines and enforcement mechanisms. L. Zadeh first introduced fuzzy logic in the development of the theory of fuzzy sets.15 The theory of fuzzy logic extends the capability of mathematical research to a composite that leverages quality and quantity and contains certain fuzziness. In recent

476 Z.-P. Wu & H. Wu

years, L. Zadeh proposed an extended perception based fuzzy logic,16−19 which is aim to lay the groundwork for a radical enlargement of the role of natural languages in probability theory and its applications, especially in the realm of decision analysis. We will attempt to solve the issues associated with uncertainty in trustworthiness management for e-commerce applications using fuzzy logic. First, we need to identify the subjects of those issues. These subjects are either the sources of trust-related information needed in e-commerce activities or the entities with which trustworthiness is built. This subject set can be defined as follows. Definition 3.1. Set of subjects in trustworthiness management. The set of subjects in trustworthiness management is all the subjects that are either the sources of trust-related information or are the entities with which trustworthiness is built. This set is represented as X in this paper. Then, we need to define a general fuzzy set in trustworthiness management. Definition 3.2. Fuzzy set for trustworthiness management. For every element x in the set of subjects X, there is a mapping x ֏ δ ( x ) , in which δ(x)∈[0,1]. The set ∆ = {(x,δ(x))} for ∀ x ∈ X is defined as a fuzzy set for trustworthiness management. δ(x) and is defined as the membership function for every X in ∆. All fuzzy sets on X are represented as Z(X). Then we can use a group of fuzzy sets from Z(X) to group all the elements of X into several sets with different levels of uncertainty. For example, we can use a group of three sets Zi ∈ Z(x) to represent different levels of uncertainty of recommendations in trust management. Z1 represents not recommended; Z2 represents normally recommended; Z3 represents highly recommended. In real life, the level of uncertainty cannot be limited to only one set, and the belongingness degrees of these sets are not simply “total” or “none.” Additionally, it is sometimes difficult to determine which set or sets should be used for certain kinds of uncertainty. In other words, these sets are not exclusive of each other. So, when we deal with certain kinds of uncertainty, a vector consisting of the degrees of belongingness to each set D = {d1,d2,d3} is more appropriate for describing the actual trustworthinessbased judgment from daily life, in which di(i = 1,2,3) is the degree of belongingness to set Zi(i = 1,2,3). Meanwhile, there are several ways to determine or calculate the degrees di. One way is direct judgment that determines the degree from direct experience or evaluation. Another one is indirect inference that determines the degree via an analysis of an indirect source. For example, reputation is relatively subjective while the evaluation method may be very objective, and recommendation is relatively objective while the source of information may be subjective. 4. Fuzzy Representation of Uncertainty In order to reason among the degrees of uncertainty in trustworthiness management for further inference or decision-making, we need to represent uncertainty formally. Direct D trust is formally described as a →b[Z ], which means entity a is willing to rely upon entity b to degree D for the categorized uncertainty Z. D is a vector with corresponding degrees


of belongingness for each set in categorization Z. Direct trust is from direct experience of the trustworthiness of the other entity or from a judgment with subjective or objective D evaluation. Indirect trust is described as a → b [Z ] , which means entity a is willing to P rely upon b to degree D following P’s recommendation for the categorized uncertainty Z. P is one or more entities constructing a path that gives a recommendation to entity a for entity b. D is a vector with corresponding degrees of belongingness for each set in categorization Z. Indirect trust is derived from the recommendation passed through one or more intermediate entities. There are also two types of recommendations. One type is that the recommender had direct experience with the recommended entity so that the P has only one entity; the other is that the final recommender formed the recommendation from further recommendations of other recommenders so that the P has more than one entity constructing a chained recommending path or a compound recommending graph. However, from the recommendee’s (entity b’s) point of view, there is no big significance related to the number of entities forming the recommending path; the recommendee (entity b) only cares about the final recommender’s capability to make an accurate recommendation based on its own experience and trustworthiness. The usage of fuzzy rules to describe uncertainty in trustworthiness management can involve rules in which we have antecedent terms of the form, such as: “If the probability of (some event) is high, then a certain action is performed.” Here, we will incorporate granular probability distribution for evaluating the uncertainty level of this type of antecedent by comparing alternative events for this case. In order to formalize the antecedent in the example rule above, we use Y to represent a fuzzy subset of Z(X) in the domain of X. This corresponds to a general fuzzy event. We also use a fuzzy probability H corresponding to the description of “high” in the previous example. Then the rule becomes: “If the probability that (X is Y) is H, then a certain action is performed.” If we use W to indicate the variable corresponding to the “probability of the event is,” the rule can be represented as: “If W is H, then a certain action is performed.” We will apply this general form to describe fuzzy rules in a trustworthiness-based decision system for shopping activities in e-commerce applications to express real life uncertainty in trustworthiness management and decision-making with human linguistics. Here, different formats of the probability function W introduce different types of rules. If W is a threshold function, the rule becomes a binary decision rule; if W has a fuzzy definition, the rule is a fuzzy rule; if W uses a granular probability distribution, the rule becomes most suitable for uncertainty description in human linguistics. A detailed comparison of these three probability functions can be found in Ref. 16. 5. Enforcement of Fuzzy Policy for Uncertainty Management Currently, most people use Zadeh’s operators ∧ and ∨ to perform fuzzy calculation and fuzzy logic analysis with fuzzy rules. However, these operators are too imprecise in that too much information will be lost if these are the only operators used. Thus, several general class fuzzy operators are proposed.16 To adapt to different sources of uncertainties in trustworthiness management, a parameterized general intersection operator and union operator are needed. They are also called T-norm and S-norm. With different values of the parameters, these operators can maximize the expressiveness and flexibility of the system to capture people’s intentions toward these uncertainties. Here

478 Z.-P. Wu & H. Wu

we choose a general class of parameterized fuzzy operators, proposed by Dubois and Prade,7 to perform further calculation and analysis. Because these operators are suitable for fuzzy rule set analysis and have clear semantic meanings, the intention embedded in fuzzy sets can be easily enforced. So, we define T-norm and S-norm as follows. Definition 5.1. T-norm For fuzzy set A, B ∈ Z(X) and α ∈ [0,1],

(A ∩ B )(x ) = T (A (x ), B (x ), α ) =

A (x )B (x ) max {A (x ), B (x ), α } ,

in which A(x) and B(x) represent x’s degrees of member function to fuzzy sets A and B. Definition 5.2. S-norm For fuzzy set A, B ∈ Z(X) and α ∈ [0,1],

(A ∪ B)(x) = S(A(x), B(x),α ) = A(x) + B(x) − A(x)B(x) − min{A(x), B(x), (1 − α )} max{1 − A(x ),1 − B(x ), α } in which A(x) and B(x) represent x’s degrees of member function to fuzzy sets A and B. Then we define two operators on vectors of fuzzy values. Suppose we have two fuzzy value vectors D 1 = {d 11 , d 12 ,..., d 1 P } and D 2 = {d 21 , d 22 ,..., d 2 P }. Definition 5.3. Connection operator

D1 ⊗ D2 = {T (d11 , d 21 ,α ), T (d12 , d 22 ,α ),...,T (d1P , d 2 P ,α )} Definition 5.4. Union operator

D1 ⊕ D2 = {S (d11 , d 21 ,α ), S (d12 , d 22 ,α ),..., S (d1P , d 2 P ,α )} After we define the above operators, we can perform formal analysis on fuzzy sets and fuzzy rules used for uncertainty expressions. Here, we define two sets of derivation rules (deduction rules and consensus rules) to handle different types of uncertainty. Below are the formal descriptions of deduction rules. Definition 5.5. Deduction rules D

D′

D′′

D

D′

P′′ D′′

a →b[Z ] ∧ b →c[Z ] ⇒ a →c[Z ] ∧ (P′′ = {b}) ∧ (D′′ = D ⊗ D′)

a →b[Z] ∧ b →c[Z] ⇒ a →c[Z] ∧ (P′′ = {b, P′}) ∧ (D′′ = D ⊗ D′) P′

P′

D

D′

D′′

P

P′

P′

a →b[Z] ∧ b →c[Z] ⇒ a →c[Z] ∧ (P′′ = {P, P′}) ∧ (D′ = D ⊗ D′) Deduction rules are used for a recommendation’s connection to construct a whole recommendation chain that allows the trustworthiness to be transferred from one end to the other. For two types of trustworthiness from the same categorization, deduction rules can form a new connection using the trust relationship between the recommender and the


recommendee and embed the content of that recommendation into the new connection. Below are the formal descriptions of consensus rules. Definition 5.6. Consensus rules D1

D2

Dn

D1

D2

Dn

D ′′

P1

P2

Pn

P′′

D′′

a →b[Z ] ∧ a →b[Z ] ∧ ... ∧ a →b[Z ] ⇒ a →b[Z ] ∧ (D′′ = D1 ⊕ D2 ⊕ ... ⊕ Dn ) a → b[Z ]∧ a → b[Z ] ∧ ... ∧ a → b[Z ] ⇒ a → b[Z ] ∧

(P′′ = {P

m

| Pm = min{Pi (i = 1...n)}})

∧ (D′′ = D1 ⊕ D2 ⊕ ...⊕ Dn ) Consensus rules are used to combine multiple recommendations for the same kind of categorization. When two or more recommendation paths appear simultaneously, consensus rules can synthesize the opinions to form a comprehensive recommendation. The shortest recommending path is the easiest path through which to verify that indirect information, even if the value of the trust degree vector is not as high as others. We use this path as the recommending path for verification of that recommendation. However, it is more likely that we will only use the unified trust degree vector alone after composition. With the help of the fuzzy operations and rules defined above, we can form a formal decision-making process to handle uncertainty in personalized recommendations involving the management of trustworthiness. Users need to define the categorization of uncertainty. Then, the decision-making process uses fuzzy operations to combine uncertain information from different sources. After defuzzification of trustworthiness degrees, users need to judge whether the final degree is consistent with their own preferences. If not, the parameters of the fuzzy operations need to be adjusted. 6. Agent-Based Recommendation System 6.1. System architecture Figure 2 illustrates the system architecture of our prototype recommendation system. Users are allowed to define their own preferences in the format of policy rules for subjective judgment and indirect information they will handle. For instance, the level of reputation is determined by indirect information and the range of price is decided by a subjective judgment. After users define their fuzzy rules in a policy set, the system stores these policies in a database. If users are not totally satisfied with the policies they have defined, they can delete them or modify them later by changing the structure of these policy rules or the membership functions for fuzzy terms in these rules. Then, when users need to use the system for selecting products, our system will extract product information and venders’ reputations through e-commerce sites using a number of agents in the system. Each agent corresponds to one e-commerce site. Every agent retrieves policies from the policy database and applies these policies onto product information and vender’s reputation from each e-commerce web site. Most venders’ reputations are retrieved from consumer-reporting websites such as BBB.org or epinion.com. Finally a list of recommendations is returned to users after individual venders have been ranked and multiple websites’ information has been aggregated.

480 Z.-P. Wu & H. Wu

Fig. 2. System architecture.

6.2. On-demand agents As illustrated in Fig. 3, there is an agent pool in the system that contains all active agents deployed together with the system. The system provides a list of default agents corresponding to a number of most popular e-commerce websites that will be searched for a particular product. If a user finds that one of his or her favorite e-commerce websites not in the list, or he or she wants to try a new specialized website, a new agent will be added to take care of information from that new e-commerce website. For example, Bob wants to buy a laptop, and he finds that the default websites provided by the system do not include his favorite electronics sellers − bestbuy.com and newegg.com. He can add these two websites into the search list. Then two new agents are created to retrieve product and vender information from bestbuy.com and newegg.com, respectively. Afterwards Bob wants to buy a pair of shoes online, and he discovers that electronics ecommerce websites such as bestbuy.com and newegg.com do not sell shoes. He can


remove these websites from the search list. The agents for these websites will be returned to the agent pool and Bob can add new websites to reuse these agents. If an agent has not been used for some time, it will be removed from the agent pool automatically to improve system performance. Furthermore, users may have priorities on different ecommerce websites. For example, Bob has made a lot of purchases on Amazon.com and has built up trust for this website, so he prefers Amazon.com to any other e-commerce websites for his next shopping. For example, Bob gets the recommendation result for his next shopping target − a laptop. Even if the system says the laptop from buy.com is highly recommended and the same model from Amazon.com is only normally recommended, he may still choose the one from Amazon.com. So, in order to meet users’ preferences on different priorities for different websites, the system also takes the priority information into consideration when making final recommendations.

Fig. 3. Details of agent deployment and on-demand execution.

6.3. Details for personalized decision-making and recommendation In the prototype system, each agent takes care of one website, from which it retrieves product information and venders’ reputations. Then it retrieves policies defined by users from the policy database and applies them to the information retrieved from that website. Then, the agent will calculate a recommendation score for each vender. However, it is difficult to find a standard reputation for different venders from different websites. Two usage types for agents are defined, taking into consideration the existence of two different e-commerce website types, which are B2C and C2C websites. Most B2C websites, such as Amazon.com and buy.com, only provide users a simple feedback system for products rather than the venders who sell the products. So users can hardly tell the reputation or trustworthiness of those venders or websites. Some users may search for reputation information for different venders from comparison shopping websites, such as dealtime.com, pricegrabber.com and yahoo shopping. However, each of

482 Z.-P. Wu & H. Wu

these comparison shopping websites has its own method to evaluate venders’ reputations. So, the same vender may have different reputations on different comparison shopping websites, and the difference can be significant. Thus, consumers may get confused when they are looking for a specific vender’s reputation. Fortunately, there is a non-profit authoritative organization called the Better Business Bureau (BBB.org) that evaluates the reputation of almost all legitimate e-commerce websites. Our system is able to use the BBB.org as a source for vender reputation. So, each agent for a B2C website can grab the vender’s reputation from BBB.org directly. Then the agent will calculate a trust score for each vender following the algorithms described in Table 1. Table 1. Pseudo-code of decision-making and recommendation algorithms.

1

2

3

FOREACH prvi IN PRV FOREACH fti IN prvi IF fti is number of feedbacks AND degree is dnfi THEN Mnf is M dnfi ELSE IF fti is percentage of positive feedbacks AND degree is dpfi THEN Mpf is M dpfi ELSE IF fti is activity index AND degree is dai THEN Ma is M dai RV=Def( frv (Mnf (number of feedbacks), Mpf (PF), Ma(AI)) FOREACH pi IN P FOREACH fti IN pi IF fti is price AND degree is dpi THEN Mpr is M dpi ELSE IF fti is reputation of vender AND degree is drvi THEN Mrv is M drvi ELSE IF fti is reputation of product AND degree is drpi THEN Mrp is M drpi rsi = frs ( Mp(price), Mrv(RV), Mrp(RP)) Recommendation Score = Def(rs) FOREACH vender Final recommendation degree = frd(Recommendation Score, priority index) Sort(vender)

When considering C2C websites, reputation information is even more unattainable than for B2C websites. There is no comparison shopping website or organization that provides reputation information for venders from C2C websites directly. The only way to evaluate the trustworthiness of a vender is to review its feedback in that C2C website. As one of the largest C2C websites, eBay provides us with a complete feedback system that allows users to evaluate venders for different features and performance. In our system, we use various aspects provided by the eBay feedback system to evaluate C2C venders’ reputations. We use the number of feedbacks, positive feedback, detailed feedback and activity index to evaluate the reputation of a vender. The detailed calculation is illustrated in Table 1. The activity index is a parameter that represents the activity level of a vender


over the past twelve months compared with its historical transaction number. After the reputation of a vender is calculated, policies will be retrieved from the policy database and then applied to product information and vender reputation to calculate a trust score. Since thousands of venders in C2C websites sell similar products, the agent will sort these venders according to their trust scores and select the top five venders for the final decision-making process, as illustrated in Table 1. Table 2. Term definitions for decision-making and recommendation algorithms.

PRV, prvi P, pi fti Mnf M dnfi RV RP PF AI Def( ) frv ( ) frs ( ) frd( )

PRV is the sub policy set of reputation of venders which is defined by end user for C2C websites. prvi is one policy in the PRV policy set P is the general policy set and pi is one policy in the policy set Means one fuzzy term in one policy sentence Membership function of the corresponding fuzzy term Membership function of the corresponding fuzzy term at its degree Reputation of vender Reputation of product Percentage of positive feedbacks Activity Index Defuzzification function Fuzzy function to calculate the value of reputation of vender Fuzzy function to calculate the value of recommendation score Function to calculate the final recommendation degree of each vender

6.4. Result aggregation After trust scores have been generated for the venders, all the venders’ details including reputation, feedback details and trust scores, will be sent for a final decision-making process. According to the trust scores and website preferences defined by users, the prototype system will calculate a final recommendation score for each vender and sort venders by their recommendation scores, from high to low. Here, a recommendation score is not one number but rather, is comprised of a recommendation term (as defined in Sec. 3.2) and its confidence. Since we have two types of agents (B2C and C2C) and five venders chosen from each C2C website, there might be some venders who sell the same products over multiple C2C websites and might have different reputations due to different reputation systems and different prices over these websites. So, we cannot simply put the results from multiple agents into one list. As described before, we use the BBB.org as our source for vender reputation if the vender is an entire B2C website. Different C2C websites that provide platforms for various e-commerce venders also receive reputations by the BBB.org for their entire websites. So, we include both C2C websites’ overall reputations and individual venders’ reputations, calculated from feedback systems of multiple C2C websites, into the aggregation process to evaluate individual venders’ reputations, which are defined below.

484 Z.-P. Wu & H. Wu

Definition 6.1. Aggregated reputation for an individual vender

∑ Aggregated reputation =

n i =0

BRpi × VenRpi n

, in which BRpi represents the reputation

from BBB.org for the ith C2C website; VenRpi represents each vender’s reputation on that specific ith C2C website. The reputations from the BBB.org are mapped to a range between 0 and 1 ([0,1]), which can be considered as a weight parameter of each C2C website. We choose the lowest price for venders that sell the same product with different prices on different C2C websites. If the prices are the same, we select the most reputable C2C website for consumers. The complete algorithm of aggregation is illustrated in Table 3.

Table 3. Pseudo-code of aggregation algorithm.

FOREACH vender IN VenderList k0, Rep0 FOR i FROM 0 TO VenderList.Number IF( VenderList[i] == vender) Rep = Rep + VenderList[i].BRep * VenderList[i].VenRep k++ IF(VenderList[i].price < vender.price) vender.price = VenderList[i].price vender.platform = VenderList[i].platform ELSE IF (VenderList[i].price == vender.price) IF (VenderList[i].BRep > vender.BRep) vender.BRep = VenderList[i].BRep vender.platform = VenderList[i].platform END IF END IF DELETE VenderList[i] END IF vender.reputation = Rep / k Following the aggregation process, the system will sort the vender list by recommendation score, which is calculated from the trust score of each vender and the preferences over each vender. The algorithm itself is illustrated in the third part of Table 1. Then a final recommendation list will be shown to users. Users can also view the details of each vender in this list, which include the price of the product, the reputation of the vender, the reputation of the product, and the recommendation score.


7. Implementation Following the decision-making process and the workflow of the entire system architecture, more implementation details such as some practical fuzzy policies, the user interface to input fuzzy policies, and the enforcement mechanism to apply these policies to personalized recommendation for e-commerce applications, are described here. Since the shopping decision of a product involves both indirect information and subjective judgment, we generate our policies using both indirect information from reputations and subjective judgment for price preference of the product. All policies follow the general rule (policy) format discussed in Sec. 4. One example of fuzzy policy is illustrated below. − If the reputation of the vendor is excellent/average/below average and(or) the reputation of the product is good/average/poor and(or) the price of the product is expensive/average/cheap then the product is highly /normally/not recommended. The system also provides a user interface to assist users in inputting these policies, to be consistent with accurate rules or their intentions. It allows users to change the flexible parts in fuzzy policies according to their own shopping preferences. Furthermore, users can modify membership functions rather than use default ones, and can combine membership functions using “and” or “or” operators between any two of the factors. The system also allows users to add or remove e-commerce sites into the search list, from which agents select product information and a vender’s reputation. Default membership functions follow general shopping practices. However, the system still allows users (consumers) to modify all membership functions by changing sampling points, which are used to sketch the shape for fuzzy membership functions, if default membership functions do not accurately reflect the consumer’s own rules or intentions. Once the definitions of fuzzy polices are finally determined, the system uses a policy generator to translate the fuzzy policies into a formal format, and stores them into a policy database. Then, consumers can input product information, such as the product name or model, to get a personalized recommendation list. Agents will select price information from e-commerce sites and reputation information from the BBB.org, epinion, or other consumer reporting agencies. Once the agents get all needed information, the system will generate a recommendation list using the decision-making and aggregation processes. Users can also refine the system’s accuracy by adjusting a parameter in this decision process, following the feedback loop illustrated in Fig. 2. Figure 4 illustrates the interfaces for policy definition, membership function definition, and recommendation list. Since people always have their own standards of reputations and price ranges, and they do not change their minds very frequently, once policy sets are defined by users, they can be used for almost all the products in their follow-up shopping activities. The only thing users may change every time they use the system is the membership function of price, because the price may vary as the product changes.

486 Z.-P. Wu & H. Wu

Fig. 4. User interfaces.

8. Experiments and Discussion To examine the performance and adaptability of the system, we selected a Nokia N97 cell phone as the target product for shopping. Then we ran the system on two different sets of policies, which reflect two different shopping preferences, to compare the final recommendation lists and compare the recommendation list with the user’s own decision. Furthermore, we searched for the same product on several popular comparison shopping websites that provide product comparisons and compared our recommendations to their results. 8.1. Experiments on different policies We define two different policy sets that represent two different shopping preferences. The first policy set follows common sense, which prefers the balance of reputation and price. The second policy set prefers high price (expensive always equals good). The first policy set is defined below.

− IF [the price of the product is CHEAP AND the reputation of the product is POSITIVE] AND [the reputation of the vender is HIGH] THEN HIGHLY recommended.

− IF [the price of the product is AVERAGE AND the reputation of the product is AVERAGE] AND the reputation of the vender is AVERAGE THEN NORMALLY recommended.

− IF [the price of the product is EXPENSIVE AND the reputation of the product is NEGATIVE] OR [the reputation of the vender is LOW] THEN NOT recommended.


The rules of policy set 2 are defined below.

− IF the price of the product is CHEAP AND the reputation of the vender is HIGH THEN NOT recommended. − IF the price of the product is AVERAGE AND the reputation of the vender is HIGH THEN NORMALLY recommended. − IF the price of the product is EXPENSIVE AND the reputation of the vender is HIGH THEN HIGHLY recommended. Table 4. Recommendations from Policy Set 1 (R.V. represents “reputation of vender”). E-commerce site

Price

R.V.

Recommendation

1

eBay (vender 1)

449.95

High (99.6%,2616)

Highly (100%)

2

eBay (vender2)

380

Normal (100%,135)

Highly (92%)

3

Buy.com

479

B+ (BBB.ORG)

Amazon marketplace (Buy.com)

488.99

High (95%,31083)

4

Newegg.com

499.99

A+ (BBB.ORG)

5

Dell

499.99

B (BBB.ORG)

Highly (72%)

6

eBay (vender3)

470

Normal (100%,65)

Highly (67%)

7

Amazon

498.75

B- (BBB.ORG)

Highly (63%)

Highly (87%) Highly (75%)

After we define these two policy sets, we run our system under each policy set separately. Part of these recommendation lists are illustrated in Tables 4 and 5. As shown in the R.V. column, there are two different ways to represent vender’ reputation using B2C and C2C agents. For the venders from C2C websites, the reputations are described by different levels, followed by positive feedback rates and feedback scores. For B2C venders, agents capture reputation information directly from BBB.org. The final recommendation scores are represented by different recommendation levels, followed by a percentage to indicate the confidence for that recommendation level. The first eight venders in the final recommendation list are included in Table 4. They are all highly recommended by the system, but with different confidence levels. EBay vender 1 has a 100 percent confidence for its “highly recommended” recommendation score since it perfectly fits this user’s preference with a cheap price and an excellent reputation. Although eBay vender 2 sells the cell phone for a cheaper price than eBay vender 1, it is still ranked second due to its poor reputation. The third place is taken by buy.com. Although an agent is dedicated to buy.com because it is a B2C website, buy.com is also vender listed in Amazon marketplace. So, two reputations exist for these two “buy.com”s. And buy.com lists different product prices on different websites. The system chooses the lower price as the product price and aggregate reputations from two types of websites, following the algorithm discussed in Sec. 6.4, to calculate the final recommendation level. Among the top five venders, two venders (newegg.com and dell.com) offer the same sales price but with different reputations. It is this difference in reputation that distinguishes them in the final recommendation list.

488 Z.-P. Wu & H. Wu Table 5. Recommendations from Policy Set 2. E-commerce site

Price

R.V.

Recommendation

1

Amazon Marketplace (DataVision Computer Video)

637.50

High (94%,13003)

Highly (100%)

2

Amazon Marketplace (ApexSuppliers)

613.58

High (96%,5221)

Highly (98%)

3

Amazon Marketplace (eXpansys)

614.99

High (93%, 670)

Highly (95%)

4

eBay (vender4)

589.99

High (98.1%,743)

Highly (92%)

5

Amazon Marketplace (The Price Pros)

571.29

High (94%, 13827)

Highly (84%)

6

Amazon Marketplace (TigerDirect)

559.99

High (93%, 37483)

Highly (76%)

7

eBay (vender5)

649.66

Average (100%,57)

Highly (75%)

8

eBay (vender6)

540

High (98.4%, 3974)

Normally (75%)

Table 5 illustrates the top eight venders in the recommendation list from policy set 2, which prefers buying an expensive cell phone from a reputable vender. This final recommendation list is quite different from the one in Table 4, which reflects a very different user preference. For the same target product, the sales prices from all top eight recommended venders in Table 5 are higher than any of venders in Table 4, and all the recommended venders are either from Amazon marketplace or eBay, because there are many more venders doing business in C2C websites than B2C venders. Different supply chains and different sales strategies create a larger price range for venders in C2C websites than B2C venders. As illustrated in the Table 5, all the venders from Amazon marketplace have very good reputations, so they occupy the top slots in Table 5. They are basically ranked only by their sales prices. There is one exception, which is between ApexSuppliers and eXpansys from Amazon marketplace. ApexSupplier sells this Nokia N97 cell phone for one dollar less than eXpansys. But ApexSupplier’s higher reputation results in a three percent higher recommendation confidence, which perfectly reflects that both price and reputation will determine the final recommendation in policy set 2. EBay vender 5 has the highest price, but its poor reputation makes its final recommendation only 75 percent of the highly recommended level, which is seventh in the final list. So, we can see these two policy sets reflect two different shopping preferences from users. Policy set 2 will recommend reputable venders with a high price, and the recommendations based on policy set 1 will recommend a vender with a balance between the price factor and reputation information. 8.2. Adjustment of favorite websites Sometimes users have their own preferences for different venders or for certain favorite e-commerce sites. For instance, Alice used to shop frequently on Amazon.com, and she prefers to buy products from Amazon.com even if other venders offer lower prices and/or have a better reputation. So we allow users to set their preferences for e-commerce websites as well. Table 6 illustrates the recommendations for policy set 1 when the user sets Amazon.com as his or her favorite site.

An Agent-Based Fuzzy Recommendation System 489 Table 6. Recommendations from Policy Set 1 including website preferences. E-commerce site

Price

R.V.

Recommendation Highly (100%)

1

eBay (vender 1)

449.95

High (99.6%,2616)

2

Buy.com

479

B+ (BBB.ORG)

Amazon marketplace (Buy.com)

488.99

High(95%,31083)

Highly (87%) 3

eBay (vender2)

380

Normal (100%,135)

Highly (92%)

4

Newegg.com

499.99

A+ (BBB.ORG)

Highly (75%)

5

Dell

499.99

B (BBB.ORG)

Highly (72%)

6

Amazon

498.75

B- (BBB.ORG)

Highly (63%)

7

eBay (vender3)

470

Normal (100%,65)

Highly (67%)

Compared with Table 4, Amazon.com clearly ranks higher in Table 6 than in Table 4’s list. Because buy.com is also a vender in Amazon marketplace, it benefits from the website preference setting and ranks second in Table 6 (third in Table 4) and Amazon ranks sixth in Table 6 (seventh in Table 4). However, they are still not ranked at top of the recommendation list since our system not only ranks venders by preferences but also considers other factors following user-defined policies. This result is consistent with the first policy set, and this user prefers low-priced offers from his or her favorite sites. So our system can clearly capture the preferences of a user through defined policies, and can recommend the appropriate vender or e-commerce site following the user’s own preferences. 8.3. Comparison with other comparison shopping websites Table.7. Compare with other comparison shopping websites.

Dealtime

PriceGrabber R. Sites Price V.

NexTag R. Price V.

Our System R. Pric Sites V. e

Site s

R.V.

Price

1

newe gg

4.5

499.99

Dell

3

499.99

Dell

3

499.99

eBay(ve nder1)

2

dell

3.5

499.99

Tiger Direct

4.5

559.99

PCConn ection

5

551

eBay(ve nder2)

3.5

498.75

4.5

551.63

mwave

5

529.99

Buy.co m

Hi gh

479

3.5

479.00

4.5

559.99

Amazon

3

488.99

Newegg

A+

499.9 9

4.5

519.00

4

520.29

antonlin e

5

548.99

Dell

B

499.9 9

3 4 5

Ama zon.c om Buy. com J&R

PCCo nnecti on Comp USA Electr onics PLU A

Sites

Hi gh No rm al

449.9 5 380

When consumers search for products from the Internet, there are several comparison shopping websites that can provide product specifications and a list of venders, prices and some feedback on the venders. We searched for our target product Nokia N97 cell phone from three popular comparison shopping websites: dealtime.com,

490 Z.-P. Wu & H. Wu

pricegrabber.com and nextag.com. Table 6 illustrates the top five recommended e-commerce websites from each of these recommendation websites. From this table, we can clearly see that each of these comparison shopping websites has different recommendations. However, users can hardly tell which vender is the most recommended or cannot event find any particular ranking criteria in the recommendation lists. They either rank venders simply by prices or reputation. In addition, venders may have different reputations on different shopping websites. For example, dell.com gets 3.5 points out of 5 for its vender’s reputation on dealtime.com but gets just 3 points on both pricegrabber.com and nextag.com. Thus, users must first look into the reputation systems used by these comparison shopping websites. Plus, most of these comparison shopping websites can only retrieve information from B2C venders, and they rarely have information from C2C websites. Thus, many great shopping opportunities are missed. Compared to these comparison shopping websites, our system always recommends the shopping websites that can match users’ preferences most, and rank them according to the recommendation score calculated by the policies defined by the users. Users can clearly identify which shopping sites are most recommended by the system. All the results described in this section indicate that our agent-based, fuzzy recommendation system is not only able to handle indirect information and subjective judgment, but is also able to reflect more users’ preference, and offer consumers straightforward shopping recommendations. 9. Comparison with Other Systems An intelligent fuzzy recommendation system was proposed in Ref. 20, which can help consumers automatically select products in e-Commerce environments using fuzzy logic and data mining techniques. In their system, consumers define their preferences first; then fuzzy rules between customer needs and product features are set up; after that the system will evaluate the requirement of each product feature according to consumers’ needs and alternatives using fuzzy geometrical distance and fuzzy synthetic distance to search for the optimal combination; at last, the system recommend the optimal choice to consumers. In Ref. 21, authors presented a decision-making toolset for human reasoning about eCommerce transactions. This fuzzy expert system they proposed in the article aims to support decision making process when consumers do purchasing on B2C sites. When using the system, a crisp set of numbers, which represent factors for the B2C transactions (security, familiarity, design, trust and competitiveness), are input into the system; then these numbers are converted to linguistic terms; after that fuzzy policies are applied through a fuzzy engine and a defuzzification process; a numeric value for the likelihood of a B2C transaction will be provided to users. Another recommendation system was proposed in Ref. 22. The authors presented an associative classification-based recommendation system for personalization in B2C ecommerce applications. The system adopted knowledge discovery techniques to support personalization according to consumers’ requirements. The three- tier web-based architecture adopted by the system provides a convenient environment to consumers.


The first and third systems introduced above are good for choosing products following users’ preferences. However, they cannot provide enough purchase advice such as buy the chosen products and which vendor provides the lowest selling prices. Although the second system gives recommendations on the likelihood of a transaction, it lacks the flexibility to accommodate users preferences and íts limitation to B2C environment decreases its usability. On the other hand, our system can not only provide comparisons of different vendors and prices in both B2C and C2C environments but also recommend consumers the most suitable product with an ideal price within the userdefined price range. 10. Conclusions This paper proposed a model of uncertainty based on fuzzy logic to handle uncertainty and fuzziness in the decision-making process for online shopping activities, based on trustworthiness. Compared with the trust management model proposed by Josang,10 this paper identified different sources of uncertainty in trustworthiness for e-commerce applications, and found that this uncertainty could not be simply treated as a probability and thus could not be described by a simple probability model. This paper introduced a general categorization to describe various types of trustworthiness in practical ecommerce applications. In addition, the derivation rules proposed in this paper incorporated a parameter to allow users to adjust the membership function through a feedback mechanism in order to adapt the decision-making process to users’ changing intentions and preferences. The model proposed in this paper can be used in evaluation, analysis, and derivation of policies in management of trustworthiness directly. As illustrated in Sec. 6, application of this model in an agent-based, e-commerce recommendation application can help consumers make accurate online shopping decisions that apply their own preferences using indirect information and their subjective price judgments. The experiments described in Sec. 8 confirmed the accuracy, flexibility and adaptability of the model, and the usability of the system. References 1. D. Gefen, Reflections on the dimensions of trust and trustworthiness among online consumers, ACM SIGMIS Database 33(3) (2002) 38−53. 2. S. C. Chen and G. S. Dhillon, Interpreting dimensions of consumer trust in E-commerce, J. Information Technology and Management 4 (2−3) (2004) 303−318. 3. N. Li, J. C. Mitchell and W. H. Winsborough, Beyond Proof-of-compliance: Security Analysis in Trust Management, J. ACM 52 (2005) 474−514. 4. J.-W, Huang and D. Nicol, A calculus of trust and its application to PKI and identity management, in Proc. 8th Symp. on Identity and Trust on the Internet, 2009, pp. 23−37. 5. Y.-Z. Ren, M.-C. Li, C. Guo, Y.-R. Cui and K. Sakurai, Trust similarity measure in trust model for P2P networks, IEEE/IFIP Int. Conf. on Embedded and Ubiquitous Computing, 2008, pp. 107−112. 6. G. Ruffo and R. Schifanella, A peer-to-peer recommender system based on spontaneous affinities, ACM Trans. Internet Technology 9(1) (2009) 1−34.

492 Z.-P. Wu & H. Wu

7. D. Dubois and H. Prade, New results about properties and semantics of fuzzy set theoretic operators, Fuzzy Sets: Theory and Applications to Policy Analysis and Information Systems (Plenum, New York, 1980), pp. 59−75. 8. D. H. McKnight and N. L. Chervany, The Meanings of Trust, MISRC Working Papers Series, 2000. 9. T. Beth, M. Borcherding and B. Klein, Valuation of trust in open networks, Proc. 1994 European Symposium on Research in Security, 1994, pp. 3–18. 10. A. Jøsang, R. Ismail and C. Boyd, A survey of trust and reputation systems for online service provision, Decision Support Systems 43(2) (2007) 618−644. 11. R. Clarke, A major impediment to B2C success is...the concept ‘B2C’, in Proc. 8th Int. Conf. on Electronic Commerce, 2006, pp. 441−450. 12. X.-Y. Wu and A. Bolivar, Predicting the conversion probability for items on C2C ecommerce sites, in Proc. 18th ACM Conf. on Information and Knowledge Management, 2009, pp. 1377−1386. 13. Y.-F. Yang, Y. L. Sun, S. Kay and Q. Yang, Defending online reputation systems against collaborative unfair raters through signal modeling and trust, in Proc. 2009 ACM Symp. on Applied Computing, 2009, pp. 1308−1315. 14. J. Brown and J. Morgan, Reputation in online auctions: The market for trust, California Management Review 49(1) (2006) 61–81. 15. L. A. Zadeh, Fuzzy logic and approximate reasoning, Synthes 30 (1975) 407–428. 16. L. A. Zadeh, Toward a perception-based theory of probabilistic reasoning with imprecise probabilities, J. Statistical Planning and Inference 105 (2002) 233–264. 17. L. A. Zadeh, Toward a theory of fuzzy information granulation and its centrality in human reasoning and fuzzy logic, Fuzzy Sets and Systems 90(2) (1997) 111−127. 18. L. A. Zadeh, Toward extended fuzzy logic-A first step, Fuzzy Sets and Systems 160(21) (2009) 3175−3181. 19. L. A. Zadeh, Toward a perception-based theory of probabilistic reasoning with imprecise probabilities, Intelligent Systems for Information Processing, 2003, pp. 3−34. 20. Y.-K. Cao and Y.-F. Li, An intelligent fuzzy-based recommendation system for consumer electronic products, Expert Systems with Applications 33(1) (2007) 230−240. 21. F. Akhter, Z. Maamar and D. Hobbs, A fuzzy logic-based approach for supporting decisionmaking process in B2C electronic commerce transaction, Int. J. E-Business Research 2(2) (2006) 54−67. 22. Y.-Y. Zhang and J.-X. Jiao, An associative classification-based recommendation system for personalization in B2C e-commerce applications, Expert Systems with Applications 33 (2007) 357−367. 23. D.-Q. He and J. Yang, Authorization control in collaborative healthcare systems, J. Theoretical and Applied Electronic Commerce Research 4(2) (2009) 88−109. 24. G. L. Millán, M. G. Pérez, G. M. Pérez and A. F. G. Skarmeta, PKI-based trust management in inter-domain scenarios, Computers & Security 29(2) (2010) 278−290. 25. eBay, http://www.ebay.com 26. Amazon, http://www.amazon.com 27. Yahoo shopping, http://shopping.yahoo.com 28. Dealtime, http://www.dealtime.com 29. Price Grabber, http://www.pricegrabber.com 30. Better Business Bureau, http://www.bbb.org 31. Epinions, http://www.epinions.org