test scores. Khalid M. Dubas. Associate Professor of Marketing, School of Business and Economics,. Fayetteville State University, Fayetteville, North Carolina, ...
An executive summary for managers and executive readers can be found at the end of this article
Some difficulties in predicting new product trial using concept test scores Khalid M. Dubas
Associate Professor of Marketing, School of Business and Economics, Fayetteville State University, Fayetteville, North Carolina, USA
Saeed M. Dubas
Assistant Professor of Mathematics, University of Pittsburgh at Titusville, Titusville, Pennsylvania, USA
Catherine Atwong
Associate Professor of Marketing, College of Business Administration, California State University at Fullerton, Fullerton, California, USA Keywords Consumer behaviour, Marketing research, Marketing strategy, New Product Development, Product management Abstract At an early stage in the new product development process, marketers often evaluate several concept statements in terms of customer preferences to choose the best concept for further development. Purchase intention scale is often used to measure consumer preferences at this stage when the product is still a concept statement or a mathematical position on a perceptual map. This paper discusses the limitations of two methods of aggregating individual preferences, namely plurality (first-choice) and the Condorcet (pair-wise majority) methods. The plurality method is subject to the top-box paradox while the Condorcet method suffers from the paradox of voting. The Copeland method is presented as an alternative to the Condorcet method when the latter fails to identify the majority's choice. Some limitations of predicting product trial are also presented.
Introduction Often marketers are faced with situations that involve selecting one or more items from several alternatives like product concepts, package designs, vendors or ads. For this purpose, responses may be solicited from several individuals and combined using an aggregation rule. The following discussion of aggregation rules is applied to the new product development (NPD) process. Evaluates three rules
Early in the NPD process, concept statements are formulated to represent various potential alternative products. Customers are asked about their intent to purchase or their probability of purchase regarding product concepts presented to them. Customer responses are then combined using an aggregation rule and translated into rough estimates of sales potential. Various rules for aggregating individual preferences, however, may lead to different guidelines for product development. This article evaluates three rules for aggregating individual preferences (plurality, the Condorcet, and the Copeland methods) and discusses some paradoxes of aggregation rules. Later in the NPD process, actual purchase is measured through some form of new product testing. At this stage, logit model is utilized to translate preference into purchase predictions (Urban and Hauser, 1993). Predicting new product trial for a choice set Consumer packaged goods marketers utilize mail interviews or mall/in-store intercept to measure respondents' intention to try the product. The most
50
JOURNAL OF PRODUCT & BRAND MANAGEMENT, VOL. 8 NO. 1, 1999, pp. 50-60 # MCB UNIVERSITY PRESS, 1061-0421
common form of the purchase intention question is a five-point scale (Crawford, 1994; Urban and Hauser, 1993) where: (1) definitely would buy; (2) probably would buy; (3) might or might not buy; (5) definitely would not buy. The number of respondents in the first category represents the top box. The plurality method orders alternative concepts based on the top-box score. Some marketers, however, utilize the combined scores of the top two boxes (categories) instead of the top box for predicting product trial. The prediction system
The simplest prediction system for new product trial would be an estimate of the number of people who would try the product if they were aware of the new product and could find it easily. This prediction system is expressed by equation (1) that can be utilized to predict product trial at the end of the first year (Moore and Pessemier, 1993): CTR1 = TBS 6 A1 6 D1
(1)
Where CTR1, the cumulative trial rate at time 1, is the proportion of people who will try the product by the end of the first year, TBS is the top-box score (percentage), A1 is the proportion of people expected to be made aware of the product during the first year, and D1 is the fraction of distribution coverage that is expected by the end of the first year. The number of trial purchasers can be estimated by multiplying CTR1 by the number of product class buyers, #PCB. For example, if TBS = 0.50, A1 = 0.20, D1 = 0.10 then CTR1 = 0.01 or 1 percent. If #PCB = 20 million, then 200,000 people (0.01 6 20 million) would have been trial purchasers at the end of the first year. Moore and Pessemier suggest that the estimates of A1 and D1 should come from expected expenditures and experience with past new product introductions. TBS
Several variations of this method exist. Various research suppliers offer proprietary techniques to determine trial and repeat buying intentions from concept statements. For example, Burke Marketing Research, Inc. offers BASES, a pretest market volume estimation technique that utilizes in-store intercept sampling, with measure on a five-point intent-to-buy scale (Gruenwald, 1992). The NPD research offers ESP (estimated sales potential) that is designed to project trial, repeat, and volume purchases of new products (Gruenwald, 1992). The NPD research uses the top two boxes in a five-point scale (instead of the top box) in their ESP system (Moore and Pessemier, 1993). Others (for example, Urban and Hauser, 1993) have used different measures of TBS, such as the percentage who marked the top box plus half of the percentage who marked the second box. Still others use some weighted combination of all five items. Two rules for aggregating individual scores Gillett (1990) noted that marketers often utilize the plurality method based on the top box of the purchase intention scale to calculate concept test scores. The respondents are asked to rank alternative concept statements in terms of most preferred to least preferred. The respondent preferences are aggregated by counting the number of first-ranks for each concept and the concept with most first-ranks is chosen for further development. Gillett utilized the
JOURNAL OF PRODUCT & BRAND MANAGEMENT, VOL. 8 NO. 1, 1999
51
Condorcet method to show that an alternative that wins a plurality (most first-choice) of votes may not be the choice of a majority of respondents. Unlike the plurality method, the Condorcet method considers respondent preference among all concepts in terms of pair-wise comparisons. The plurality method orders alternatives in terms of the group's first-rank and may not indicate the majority's choice. However, the Condorcet method orders items based on pair-wise majority. Condorcet method
Gillett did not, however, mention that for some datasets the Condorcet method may fail to provide a well-behaved (transitive) ordering among the items. That is, the group's choice may be sensitive to the order in which the pairs of items are considered. This manuscript shows that the Condorcet method can fail to provide a well-behaved ordering for some datasets and therefore majority's choice based on the Condorcet method (the Condorcet candidate) may not exist. When the Condorcet candidate does not exist then the Condorcet method is not appropriate for choosing among the various alternatives. The Copeland method can be utilized for ordering items when the Condorcet candidate (winner) does not exist. In addition, there are several other methods that can be utilized to aggregate individual preferences (Dubas and Strong, 1993; Gupta and Kohli, 1990). This article extends Gillett's work in four ways: (1) by recognizing the need to include leading competing brands in the choice set; (2) by discussing a limitation of the Condorcet method called the paradox of voting; (3) by discussing the Copeland method as a substitute for the Condorcet method; and (4) by linking concept scores based on plurality and the Condorcet method with awareness and availability to predict new product trial.
Concept tests
Ordering of alternatives Market researchers who perform concept tests often assume that individuals can order concept statements based on their preference by making transitive (or consistent) comparisons among concept statements. Transitivity A well-behaved preference function is transitive, i.e. it is consistent among the alternatives being compared. Consider three alternatives A, B and C. Let A > B state that the respondent prefers A over B and A = B state that the respondent is indifferent between A and B. Preference is transitive if A > B, and B > C then A > C. This preference function can be written as A > B > C. Also, if A = B, and B = C then A = C. This preference function can be written as A = B = C. Intransitivity If an individual ranks three alternatives as A > B, B > C, and C > A then this individual's preference is not transitive. In this case, the choice of the most preferred alternative depends on the order in which alternatives are evaluated. For three alternatives, A, B and C, there are three pairs A B, A C, and B C. Consider the following three different ways of comparing these pairs. First, start the pair-wise comparison by comparing A and B. A will be chosen over B. Comparing A with C favors C. Therefore,
52
JOURNAL OF PRODUCT & BRAND MANAGEMENT, VOL. 8 NO. 1, 1999
the choice will be C. Second, start pair-wise comparison by comparing A and C. C will be chosen over A. Comparing C and B will favor B. Therefore, B will be chosen. Finally, start pair-wise comparison by comparing B and C. B will be chosen over C. Comparing B with A will favor A. Therefore, A will be chosen. This shows that the choice of the most preferred alternative depends upon the order in which alternatives are evaluated. The preference function for this individual is C > A > B > C. The aggregate preference may be intransitive Many researchers assume transitivity (or consistency) of individual preferences as a minimum condition of rational behavior. However, transitivity of aggregate preference function can not be assumed since it depends on the aggregation method. Some aggregation methods lead to intransitive aggregate preference functions for some datasets. For example, the aggregate preference function based on the Condorcet method is intransitive for some datasets (Dubas and Strong, 1993).
Preferences of 100 respondents
When the Condorcet candidate exists (Gillett's data) The top-box paradox Gillett (1990) utilized a dataset consisting of preferences of 100 respondents for four alternative concepts. These preferences are represented by six preference orderings of four alternatives A, B, C, and D. An ordering is transitive in terms of preference or indifference among alternatives. Gillett's dataset can be made more realistic by assuming that alternatives C and D are leading competing brands in the marketplace while the company in question wants to compare its product alternatives A and B against each other and against the leading rivals based on customer preferences. The choice set should include a company's existing brands and leading competing brands so a company could avoid product cannibalization and get a more realistic picture of customer preferences since ultimately a new product will have to compete against those alternatives already available to the customer. Plurality method. The plurality method is based on the first-choice (top-box) data only. For Gillett's (1990) data in Table I, 40 respondents choose A as their first choice while only 20 people choose B, C, or D as their first choice. Therefore, based on the plurality method, concept A will be selected. The aggregate preference function for this method is A > B = C = D. The Condorcet method. To determine the aggregate preference function based on the Condorcet method, one needs to compare all pairs of alternatives. The formula for determining the number of pairs to compare is n(n ± 1)/2, where n is the number of alternatives. In this case, there are four alternatives that give rise to six pairs. Gillett noted that using the Condorcet method based on pair-wise comparisons, it can be shown that A is the choice 1
2
3
4
5
6
± A B C D ± 40
± B C D A ± 20
± C B D A ± 10
± C D B A ± 10
± D B C A ± 10
± D C B A ± 10
Table I. Gillett's dataset on six preference orderings of 100 individuals JOURNAL OF PRODUCT & BRAND MANAGEMENT, VOL. 8 NO. 1, 1999
53
of a minority of respondents. Comparing A and B shows that 40 respondents prefer A over B while 60 respondents prefer B over A. Comparing B and C shows that 70 people prefer B over C while only 30 people prefer C over B. Comparing B and D shows that 70 people prefer B over D while only 30 people choose D over B. Therefore, in pair-wise comparisons, B is the choice of a majority of people. The three remaining pairs A C, A D, and C D can also be compared to determine the aggregate preference function for the four alternatives. The dataset in Table I provides a transitive aggregate preference function B > C > D > A which is based on information from all pair-wise comparisons as shown in Table II. New product trial
Condorcet candidate
An illustration of predicting new product trial Equation (1) can be utilized to predict new product trial for Gillett's dataset. First, using the top-box ordering of alternatives A > B = C = D, item A will be chosen. Item A is chosen by 40 percent of the respondents as their first choice over all other items. Therefore, TBS = 0.40 will be entered in equation (1) to predict trial of item A and further development will proceed along the line of concept A. However, the Condorcet method orders items as B > C > D > A based on pair-wise majority. This demonstrates that item A is not the Condorcet winner. In fact, A is the minority's choice. Therefore, item B is the Condorcet candidate and further product development should proceed along the line of item B. What score should be entered in equation (1) in place of TBS to predict trial of item B? It can be seen from Table I that 60 percent of the respondents order B > C > D. Therefore, 0.60 should be entered for TBS in equation (1). Notice that 60 percent of respondents choose B over both of the leading competing brands C and D, while only 40 percent of the respondents choose A over C and D. Similar ideas are expressed by Tauber (1981). He refers to the intention scale and points out that the second, third and fourth preferences could be very sizable and should not be ignored. The Condorcet method, by performing all pair-wise comparisons, orders alternatives in terms of the majority's choice and not the first preference only. When the Condorcet candidate does not exist (modified data) The paradox of voting (majority cycles) A slight modification of Gillett's dataset shows that the Condorcet candidate (majority's choice) may not exist. The preference orderings are not changed for the new dataset. Only the distribution of respondents is changed for the six preference orderings as shown in Table III. This modified dataset shows the major limitation of the Condorcet method. To see this limitation, perform all pair-wise comparisons among alternatives and determine the aggregate preference function. The group's preference among B, C and D is as follows: B > C, C > D, and D > B. This preference is intransitive. The aggregate Aggregate preference function 1 2 3
Pair AB AC AD
Rank B>A C>A D>A
Numbers 60 to 40 60 to 40 60 to 40
4 5 6
BC BD CD
B>C B>D C>D
70 to 30 70 to 30 80 to 20
B>C>D>A
Table II. The Condorcet method using Gillett's dataset 54
JOURNAL OF PRODUCT & BRAND MANAGEMENT, VOL. 8 NO. 1, 1999
1
2
3
4
5
6
± A B C D ± 10
± B C D A ± 20
± C B D A ± 10
± C D B A ± 20
± D B C A ± 30
± D C B A ± 10
Table III. Modified dataset on six preference orderings of 100 individuals
preference function for B, C and D is B > C > D > B. This is called a majority cycle or the paradox of voting. Aggregate preference function
Table IV shows all pair-wise comparisons and the aggregate preference function. The aggregate preference function among the four alternatives is B > C > D > B > A. Clearly this function cannot be used to choose among B, C, and D since the Condorcet candidate does not exist. However, it is clear that A is the least preferred alternative. While a rational individual is assumed to have a transitive preference function, the aggregate preference function may not be transitive since it may be sensitive to the particular aggregation method being utilized. When the aggregate preference function is intransitive for a particular dataset, the researcher may perform pair-wise comparisons (to determine the majority's choice) and reach different conclusions depending upon the order in which pairs are evaluated. Compare these three alternative scenarios. First, start pair-wise comparisons by comparing A and B. B will be chosen over A. Comparing B with C, B will be favored. Comparing B with D, D will be chosen. So the choice of the majority is D. Second, start pair-wise comparisons by comparing A and C. C will be chosen over A. Comparing C with D will favor C. Comparing C with B will favor B. This leads to B as the majority's choice. Finally, start pair-wise comparisons by comparing A and D. D will be chosen over A. Comparing D with B will favor D. Comparing D with C will indicate that the group prefers C over D. This leads to C as the majority's choice. Therefore, the Condorcet method is indecisive when a majority cycle exists. The Copeland method. When the Condorcet winner does not exist then one way to obtain a transitive group preference function is by utilizing the Copeland method. This method suggests that the alternatives be ordered according to the number of items over which they are preferred by a majority of individuals. That is order alternatives according to the number of pairwise comparisons that each alternative wins. Applying the Copeland method Aggregate preference function 1 2 3
Pair AB AC AD
Rank B>A C>A D>A
Numbers 90 to 10 90 to 10 90 to 10
4 5 6
BC BD CD
B>C D>B C>D
60 to 40 60 to 40 60 to 40
B>C>D>B>A
Table IV. The Condorcet method using modified dataset JOURNAL OF PRODUCT & BRAND MANAGEMENT, VOL. 8 NO. 1, 1999
55
to Gillett's data in Table I, we note that the necessary pair-wise comparisons among alternatives have already been performed in Table II. Therefore, one can simply count the number of pair-wise comparisons that each alternative wins as follows: A (zero), B (three), C (two), and D (one). Therefore, these alternatives will be ordered as B > C > D > A which is the same as the aggregate preference function for the Condorcet method. Note that when a Condorcet winner exists it will also be the Copeland winner. Majority relation is intransitive
However, when the majority relation is intransitive then there will be a majority cycle as demonstrated by applying the Condorcet method to the modified Gillett's data. The Copeland method, however, always provides a transitive aggregate preference function even when the Condorcet method suffers from cyclical majorities. Consider the modified Gillett's dataset in Table III. The necessary pair-wise comparisons are provided in Table IV. The alternatives can be arranged in terms of the number of pairs that each item wins as follows: A (zero), B (two), C (two), and D (two). Therefore, the Copeland ordering is B = C = D > A. This shows that the Copeland method orders alternatives like the Condorcet method when the Condorcet winner exists. However, when the Condorcet winner does not exist then the Copeland method can provide a transitive ordering among alternatives. It should be noted that even though the Copeland ordering is transitive, it is indecisive among B, C, and D since the group is indifferent among these three items while it clearly prefers each of these items over A. Therefore, choice among B, C, and D should be based on some other criteria. However, as far as A and B are concerned, the group clearly favors B over A. The aggregate preference functions for plurality, the Condorcet, and the Copeland methods for two different datasets are presented in Table V.
Purchase intention scale
Limitations of purchase intention scale The purchase intention scale provides a direct estimate of the customer's belief about choosing the new product. While not exact, the scale provides a good estimate of purchase behavior. The disadvantage is that a direct scale like the intention scale is limited to testing specific concepts or relatively few changes to those concepts. In addition to the intention scale, therefore, multiattribute preference models like conjoint analysis can be utilized to indirectly evaluate customer preferences for a large number of product concepts. Further, marketers utilize the logit model at a later stage in new product development process to refine demand estimates obtained at an earlier stage using intention scale or multi-attribute methods. The logit model is more realistic since it incorporates measurement error and tries to explain as much customer behavior as feasible (Urban and Hauser, 1993). Limitations of preference aggregation methods All three aggregation methods discussed here, namely, plurality, the Condorcet, and the Copeland methods, have their limitations. These methods do not measure the intensity of preferences in terms of comparing whether item A is strongly preferred over B or barely preferred over B. Market Method
Gillett's data
Modified data
Plurality Condorcet Copeland
A>B=C=D B>C>D>A B>C>D>A
D>C>B>A B>C>D>B>A B=C=D>A
Table V. Aggregate preference functions 56
JOURNAL OF PRODUCT & BRAND MANAGEMENT, VOL. 8 NO. 1, 1999
researchers should explore other methods that measure preference intensity (Dubas and Strong, 1993; Gupta and Kohli, 1990). Our discussion of a wellbehaved aggregate preference function has utilized the concept of transitivity. Other desirable characteristics of preference functions should also be explored (Dubas and Strong, 1993). Influencing factors
Limitations of predicting product trial Our discussion has focussed on ordering of items based on the preferences of a group of respondents. It is assumed here that the participants truthfully express their preferences about the items under study. The findings of the sample of respondents can be extended to a larger population if the sample is representative of the population. There are several other factors that influence the predictive quality of concept tests. First, the concepts may change from original concepts at testing stage to the time when new products are introduced to the market. Second, the company may change its marketing plan and as a result it may change the amount it spends to promote and distribute its products to prospective customers. Finally, the timing of introduction of new products is critical. The product may be late to the market if competitors have already introduced new products. In short, the responses of a group of participants in a study do not once and for all establish the preference ordering among items evaluated. People change their preferences over time and the competitors may improve their products or marketing plans. Therefore, a company should carefully utilize consumer research throughout the product development stage and also after a new product is introduced in order to manage the product life cycle. Consumer research has its limitations and some of these are discussed by Dubas et al. (1998).
NPD process
Condorcet method is useful
Conclusion At an early stage in NPD process, marketers evaluate several product concepts to guide further product development. For this purpose, individual preferences are collected and aggregated. This study evaluated plurality and the Condorcet methods of aggregating individual preferences to choose the most preferred alternative based on customer preferences. In the plurality method, each individual has only one vote, which is cast for the individual's first choice. The plurality method generates transitive aggregate preference function. The plurality method considers only the first-choice data and it may suffer from the top-box paradox, that is, it can choose an alternative that is favored by a minority of voters over an alternative that may have the approval of a larger number of respondents. The Condorcet method is useful in identifying the majority's choice if one exists. This method is difficult to apply if a large number of alternatives are to be compared since it requires a respondent to rank all alternatives so that pair-wise comparisons can be made. A Condorcet candidate (the majority's choice) does not exist if the aggregate preference function is intransitive. The Copeland method provides a transitive preference function when the Condorcet method suffers from the paradox of voting. However, when the Condorcet winner does not exist, the Copeland method, while providing a transitive aggregate preference function, may be unable to provide a clear winner among the alternatives due to indifference relations among items. The choice of appropriate method depends on the question being asked of the data. If the aim is to determine the concept statement that is ranked first most
JOURNAL OF PRODUCT & BRAND MANAGEMENT, VOL. 8 NO. 1, 1999
57
often then the plurality method is appropriate. If the purpose of research is to choose the concept statement preferred by a majority of respondents then the Condorcet method is appropriate (Gillett, 1990). However, the Condorcet method can report cyclical majorities for some datasets by providing an intransitive aggregate preference function. In such a situation, it cannot be utilized to identify the majority's choice. Marketers should search for other methods to identify the majority's choice when the Condorcet method fails. The Copeland method was presented as a point of departure in this search. Primary criterion, diagnostic and classification
In concept tests, marketers typically collect extensive information that may be classified into three categories: primary criterion, diagnostic, and classification (Moore and Pessemier, 1993). While direct measurement of preference using intention scale may provide a good indication of purchase behavior, it is limited to a few specific concepts. Indirect measurement of preference using multi-attributed preference models (like preference regression or conjoint analysis) measures customer preference for a large number of concept statements in full-profile models using orthogonal arrays. Concept tests should utilize multiple methods of data collection and aggregation. While intention scale and multi-attribute models are appropriate at early stage, of product development, logit model should be utilized at later stages of the development process to transform preference into purchase probability (Urban and Hauser, 1993). Finally, market researchers should also consider conducting tests with brand names identified. This should help in avoiding the problem that Coca-Cola Corporation faced when people overwhelmingly chose the New Coke over Old Coke and Pepsi in blind-taste test comparisons but when brand names were added in the real world, they wanted the Old Coke over the New Coke. References Crawford, C.M. (1994), New Product Development, Richard D. Irwin, Burr Ridge, IL. Dubas, K.M. and Strong, J.T. (1993), ``Arrow's general impossibility theorem and five collective choice rules: Pareto, Condorcet, Plurality, Approval Voting, and Borda'', Proceedings of the Annual Conference of the Academy of Marketing Science, Vol. 6, Miami Beach, FL, pp. 334-8. Dubas, K.M., Atwong, C. and Mehta, R. (1998), ``Limitations of consumer research for new product development'', Review of Business, forthcoming. Gillett, R. (1990), ``The top-box paradox'', Marketing Research: A Magazine of Management and Applications, pp. 37-9. Gruenwald, G. (1993), New Product Development, NTC Business Books, Lincolnwood, IL. Gupta, S. and Kohli, R. (1990), ``Designing products and services for consumer welfare: theoretical and empirical issues'', Marketing Science, Vol. 9 No 3, pp. 230-46. Moore, W.L. and Pessemier, E.A. (1993), Product Planning and Management: Designing and Delivering Value, McGraw-Hill, New York, NY. Tauber, E.M. (1981), ``Utilization of concept testing for new-product forecasting: traditional versus multi-attribute approaches'', in Wind, Y., Mahajan, V. and Cardozo, R.N. (Eds), New Product Forecasting: Models and Applications, D.C. Heath and Company, Lexington, MA, pp. 169-78. Urban, G.L. and Hauser, J.R. (1993), Design and Marketing of New Products, Prentice-Hall, Englewood Cliffs, NJ.
&
58
JOURNAL OF PRODUCT & BRAND MANAGEMENT, VOL. 8 NO. 1, 1999
