Measurement Methods for Product Evaluation - CiteSeerX

Measurement Methods for Product Evaluation Kevin N. Ottoy Department of Mechanical Engineering Massachusetts Institute of Technology Cambridge, Massachusetts October 24, 1994

Abstract

Among the many tasks designers must perform, evaluation of product options based on performance criteria is fundamental. Yet I have found that the methods commonly used remain controversial and uncertain among those who apply them. In this paper, I apply mathematical measurement theory to analyze and clarify common design methods. The methods can be analyzed to determine the level of information required and the quality of the answer provided. Most simple, a method using an ordinal scale only arranges options based on a performance objective. More complex, an interval scale also indicates the dierence in performance provided. To construct an interval scale, the designer must provide two basic a-priori items of information. First, a base-point design is required from which the remaining designs are relatively measured. Second, the deviation of each remaining design is compared from the base point design using a metric datum design. Given these two datums, any other design can be evaluated numerically. I show that concept selection charts operate with interval scales. After an interval scale, the next more complex scale is a ratio scale, where the objective has a well de ned zero value. I show that QFD methods operate with ratio scales. Of all measurement scales, the most complex are extensively measurable scales. Extensively measurable scales have a well de ned base value, metric value and a concatenation operation for adding values. I show that standard optimization methods operate with extensively measurable scales. Finally, it is also possible to make evaluations with non-numeric scales. These maybe more convenient, but are no more general.

1 Introduction A mechanical design engineer has many dierent methods available to help design a product. Concept selection charts [2, 22, 31] help to choose among designs. QFD [1, 11] helps consider the ability to satisfy customer requirements. Optimization models [21] help negotiate among dierent design concepts. These methods are designed to help make a determination, yet personal experience has shown that many designers, students and instructors do not believe in the mechanics of the methods. When using a Pugh selection chart, people often feel uncomfortable using the highest scoring result. They often ask what the dierences in  y

Accepted for publication in Research in Engineering Design . 3-449B MIT, 77 Massachusetts Avenue, Cambridge, MA 02139 USA, [email protected].

1

numbers really mean. The methods are, however, usually eective; they can quickly form group consensus on major issues. They help in overcoming social problems in the design process. Nonetheless most persons involved do not put faith in the mechanics of what they are doing. This is especially true with concept evaluation. There remains much confusion and misleading arguments over the needs and bene ts of the dierent methods. This paper will provide an analysis so that a designer can determine the suitability of any method for a design task. Every design method can be analyzed for the a-priori information required and the meaning of the solution produced. I have found product evaluation methods pose the most serious contention among design methods. Therefore this paper will focus particularly on product evaluation, though diversions to other design methods (e.g., QFD) will be discussed as well. A product evaluation forms a rating to help determine which option to pursue. A rating of a product concept is a measurement : a ranking of the product among others on some scale. Therefore evaluation can be viewed mathematically as a problem of measurement. Measurement theory [14] offers a complete, axiomatic theory that addresses the measurement of any quantity, including design concepts, on a single scale. This paper will present basic principles of measurement theory and then analyze common design methods, with emphasis on evaluation methods. The quality of the information required and the quality of the evaluation result produced will be the criteria to understand how the methods dier. Best practices for each method can then also be de ned. Thus I will not develop any new theory on how to make a design decision; rather, I will analyze existing methods, clarify the steps involved and oer recommendations that a designer should observe. This paper is organized in terms of the complexity of a method. First, the basic problem structure common to all evaluation will be detailed. Then in Section 2, the rst level of evaluation complexity is discussed, an ordinal scale common to preliminary concept evaluation. In Section 3, the next level of complexity is discussed, an interval scale common to numerical concept evaluation. In Section 4 the next higher level of complexity is discussed, a ratio scale also used in numerical evaluation. In Section 5, the most complex evaluation scale is discussed, an extensively measurable scale that comes complete with a \unit." Finally, Section 6 discusses evaluation with non-numerical scales.

1.1 Problem Structure

Evaluation methods in engineering design are utilized to decide among alternatives that a designer has identi ed: the set of alternatives to be considered. Though evaluation may help spark creative thoughts in a designer, evaluation methods in and of themselves do not fundamentally generate new options. This obvious statement bounds the study of this paper, and indeed all evaluation. The most general object considered by an evaluation method is a set . If a designer needs to evaluate product con gurations to make a selection, these con gurations must rst be identi ed. This identi cation can be as simple as a label. Nonetheless, this requirement is fundamental: only well identi ed objects can be evaluated and discussed. Set structure is needed in the information of a design problem before any evaluation can be made. For example, matrix evaluation methods [1, 2, 22, 31] represent a set of designs with a nite set of labels, each label being listed along the columns of the matrix. Catalog lookup systems [34] represents a set of product con gurations as a set of catalogued components. Expert system methods [17] may represent the set of options as all possible combinations of the attributes characterizing the product design. Similarly, any computational method 2

must operate with a well identi ed set of objects [6], by virtue of the fact that the method operates with a representational model. Discussion about \everything in the universe of possibilities," or \anything that will do my task" on the other hand, is not suitable for evaluation. Only those objects actually described collectively as a set can be evaluated. This formalization requirement is the rst axiom of any evaluation method. Qualitatively, set structure of a collection is identi ed by two fundamental characteristics. First, given any arbitrary object, one can determine whether or not the object is in the set. Given a new product option, a designer must be able to determine whether or not the new product option is already in the collection of evaluation options. The second characteristic of set structure is that any two given elements in the collection can be dierentiated. A designer must be able to distinguish the options being evaluated. More complete de nitions of sets can be given [4], but this qualitative de nition suces for this paper. After identifying the concepts options, then a designer can proceed to evaluate them. The next section discusses the most structurally primitive method for constructing a justi able evaluation.

2 Ordinal Scales Having a set of product options is not sucient to make a decision. A \best" option must be identi ed: an evaluation structure must be constructed. The simplest structure is to assign each option a label of \equal or better" or \equal or worse." Such assignments form an ordinal scale over the set: each option is deemed better or worse than its counterparts, and placed in order. No numbers are used at all. To construct this ordering, the designer must be able to judge the options. Out of the set X of all product design options and in terms of the objective under consideration, a designer must determine if a given option is better or worse than the others. If the designer can do this, the con guration options can be formed into a weak order across the set X of product design options, x1  x2  : : :  xn: (1) This ordering is weak since equivalence can hold. Mathematically, the relation is re exive, transitive, and comparable. Elements with equivalent amounts (to within the resolving power of the designer) are allowed. The ordering provided by (1) is also called an ordinal ranking: the elements are only ordered. An ordinal statement of \x1  x2  x3 " says nothing about how far apart x1 is from x2 relative to the separation of x2 from x3 , it says only that x1 is lower than x2 is lower than x3 . A designer must have a reason for making such statements to form a weak ordering. There must be reasons for the judgement in each evaluation. These reasons might be customer requirements such as cost, appearance considerations or desired colors. The reasons could well be engineering criteria such as stress levels. Also availability concerns such as lead times or available sizes can be used. Any reasoning upon which a product design can be judged forms an objective for rating designs. The existence of at least one such reason is the second axiomatic assumption required to form an evaluation. Denote an objective by . This is not to signify that an objective is a numerical function, rather  is merely a label. An evaluation procedure commonly used but, as far as I can uncover, never precisely identi ed or stated until now is the better/worse method . First select an arbitrary option from X to form the beginning of a list fx0 g indexed by j . This 3

Spring Actuation Switch

Mass

Backstop

Direction of Frame Acceleration

Figure 1: Accelerometer switch. concept will be called a base point or \datum" concept for the evaluation. For every other product design ask: \Is design xi as good as or better than design xj in ?

(Q1)

This line of questioning interactively produces a bubble sort of the options. The best option is at the head of the list, and the evaluation among the options is complete. Given a single performance objective and a set of product concepts, this procedure allows a designer to construct a complete evaluation of the alternatives and thereby make a selection. The concept at the head of the list is better than the rest in the objective  and is therefore recommended. Given this development, methods commonly used in product development which form ordinal scales can be analyzed, such as Pro/Con charts and the Pugh concept selection process.

2.1 Example: Pro/Con Charts

As an example of using an ordinal measurement scale for product evaluation, consider the design of an accelerometer switch. The switch uses a mass that pulls against a mechanical spring. With sucient acceleration, the spring stretches and allows the mass to actuate a switch. See Figure 1. The performance objective of the switch designers is to ensure a precise actuation time under a speci ed acceleration. This design has a manufacturing problem. It proves dicult to ensure a precise actuation time due to assembly errors. In particular, it proves dicult to make the spring constant precise, due to the positioning and pre-loading operations. Suppose the designer has determined possible actions to remedy the situation on the factory oor. These possible actions may include: 1. Improve the assembly operation so that a precise spring constant is achieved on every accelerometer 4

2. Adjust the mass of each accelerometer so that the spring errors are accounted for 3. Adjust the backstop position of each accelerometer so that the spring errors are accounted for 4. Do nothing and inspect out the bad parts. This can be condensed into a set of labels representing each of the concepts as envisioned by the designer: X = fNothing; Mass; Spring; Lengthg: The objectives that the designer could use to make an evaluation over these options may be: 1. The ability of the operation to ensure the timing performance of the product 2. The cleanliness of the operation, as contamination can aect the product 3. The anticipated product quality 4. The projected cost of the operation 5. The simplicity of the operation. There are therefore multiple objectives in this evaluation:  = fTiming; Cleanliness; Cost; Simplicity; Qualityg: Suppose the designer must evaluate these alternatives and make a decision. One method is to list the Pros and Cons of each option. Typically such a Pro/Con is used to help identify one's thoughts succinctly or to agree as a group on a concept. For example, many times a Pro/Con is used to formulate the set of objectives  that a group feels is relevant. If a Pro/Con is used as a basis of evaluation, however, it should be understood what underlies the mechanics of the process. The information required and the elemental steps that must be applied should be analyzed. With a Pro/Con, the option set is ranked into the categories \Pro", \Con" or \neither" for each of the dierent performance objectives. This can be identi ed into a simple set of rank values S = fPro; Con; neitherg: The ordinal question asked of each design xi on each objective j 2  is: \Does design xi get a \Pro" or a \Con" in j ?"

(Q2)

Generally speaking, if j is better, then xi gets a \Pro" rating; if worse, then xi gets a \Con" rating. If it is neither better nor worse, then the objective j remains blank on xi . Suppose the Pro/Con evaluations are determined as shown in Table 1. Notice that a diculty is that there is no base point for the ranking. For each objective, the Pro/Con method attempts to rank each con guration in a general reference frame that does not indicate what either \Pro" or \Con" means. \Pro", \Con" or \neither" as compared to what? As others agree [22], evaluation without a base point concept can prove dicult. 5

Table 1: Pro/Con Evaluation of Product Options. Improve Adjust Adjust Do Spring Mass Length Nothing Pros Great Timing Good Timing Good Timing Inexpensive High quality High quality High Quality Simple Clean Inexpensive Inexpensive Clean Simple Simple Clean Cons Expensive Dirty None Poor Timing Complex Poor Quality After the ranking is complete for each objective, the options must still be ordered. With a Pro/Con chart, all the objectives are considered simultaneously when placing each option xi in order. Therefore the original ordinal-rank-construction question (Q1) is asked not on just one objective , but on all of the objectives ~ at once. All objectives must be considered simultaneously with the aid of the Pro/Con ranks in a table. Upon reviewing Table 1, the designer may decide that adjusting Nothing  Spring  Mass  Length: No numerical calculations are used with a Pro/Con evaluation, rather a designer must mentally make a non-sophisticated (though probably very dicult) determination on which option is \on the whole" better than the others. We will return to this accelerometer example throughout the paper. Notice that to perform the Pro/Con evaluation, a designer must simultaneously consider all of the objectives when comparing one option with another. This involves simultaneously comparing qualitative amounts of incommensurate objectives. The Pro/Con does help, in that it provides a 2-level ordinal ranking of \Pro" and \Con." The remainder of the evaluation is informal. This limits the number of objectives that can be considered with a Pro/Con evaluation, since most humans can only process around 7 or 8 chunks of information [28]. Given this limit, a Pro/Con chart is limited to around 3 or 4 objectives that must be traded-o. This can be veri ed by trying to compare the \Improve Spring" with the \Adjust Mass" concepts on all 5 criteria shown in Table 1. On the other hand, very little structure is required to construct the evaluation. The only structure assumed was that the options were well identi ed, and that a reason existed for making the evaluation. For design evaluations with more than 3 or 4 objectives, some more formal ranking procedure becomes necessary. Pugh's method is one such method that will now be addressed before moving on beyond ordinal scales.

2.2 Example: Pugh Concept Selection Charts

Pugh's method of concept selection [22, 31] is a popular means to conduct a preliminary concept selection. In this method, the alternatives are listed along the columns of a matrix. The objectives that form the basis of the selection are listed along the rows. Entries in the matrix are the ratings of each alternative for each objective, and are assigned ranks from 6

Table 2: Pugh Concept Evaluation of Product Options. Timing Quality Production Cost Complexity Cleanliness (+) (-) Sum

Do Adjust Improve Adjust Nothing Length Spring Mass p + + + p + + + p

p p

? ? p

? ? p

0 p0

+2 ?p2

?p2

Base Point

Metric

the set

+2

? ? ?

+2

?3 ?

S = (?; p; +); signifying \better, equal, worse" respectively. As with a Pro/Con, Pugh's method is commonly used to help organize a thought process and form consensus among a design team. Addressing how a designer interprets and eectively uses a Pugh chart can become mired in personal sentiments quickly. Questions over the mechanics of the Pugh evaluation process, however, can be rigorously addressed. We can clarify the need to accomplish three separate tasks when constructing a Pugh evaluation: 1. Identify a base point con guration 2. Rank every other concept as better or worse for each objective j 3. Sum the ranks for an overall evaluation of each con guration. Consider the rst task. When constructing the chart, ordinal measurement (Q1) demands that a designer rst select a base point design concept. This concurs with common views on best design practice [22], which demands a datum concept. In re-design applications, the base point p is usually the existing product, or perhaps a competitor's product. This case is assigned a rank in every objective. Clearly identifying such a base point for preliminary evaluation anchors the evaluation, and is a best practice with ordinal scales. As an example, reconsider the accelerometer introduced earlier. Suppose the p base point was chosen as the existing condition of the do-nothing option. It is ranked with a in every objective. Every other option must be rated on each objective relative to this reference, choosing ranks from the rank set S . The results are shown in Table 2. For example, on the cleanliness objective, the mass-adjustment option could allow more contaminants to enter the product than the do-nothing option, and so is assigned a (?). We now must combine these measured results, Step 3. The procedure is to \add" the ranks as if they were numbers. This implies that a numerical order has been assigned to the rank set ?1 ' worse 0 ' same (2) +1 ' better: 7

This mapping (2) must be examined to determine it's meaning. Every objective is assigned an impact of 1. This must mean that selection based upon the nal additive sum is in fact based on the net number of objectives that a concept oers improvement. If a nal result has a concept ranked at +3, this mean that the concept is better than the base point by p a net 3 objectives. This is the only interpretation of the nal result whereby the (?; ; +) ranks can be evaluated ordinally using the equivalence of (2). The nal summed numbers cannot be interpreted as overall numerical gures-of-merit. This point will be re-examined in Section 3.2.2 after introducing the next most complex evaluation, interval scales.

3 Interval Scales Suppose a designer wants information re ecting not just whether one option is better or worse than another, but how much better or worse. To provide this information, more than an ordinal scale is needed. The ordering provided by an ordinal scale only states that option xi is better or equal to option xj , (xi  xj ), in the objective. The ordering does not provide information as to how much better. To represent this information, more than order structure must be constructed over the set. An interval scale must be constructed [14]. Interval scales are constructed for objectives that have no identi able units of measure, but a quantitative scale is desired. When rating the accelerometer concepts based on simplicity, no quanti able measuring system is evident. Here, the set X in question (the product options) is only identi ed as a set and no additional structure is provided by \simplicity," since \simplicity" has no units. What is zero \simplicity?" What is a unit of \simplicity?" Interval scales are used to answer these questions. To state how much better one option is relative to others, additional structure is constructed by rst identifying an arbitrary reference option, as was done with ordinal scales. This option will again be called the base point option, from which all others are referenced relatively. For example, when measuring the accelerometer based upon timing deviation, a base point design which provides no deviation (zero deviation) can serve as a base point of any measurement of error. As another example, when rating product options on simplicity, the most complex option (\least simple") can form an acceptable base point. A base point, however, is not the only structural requirement needed. A metric option must additionally be identi ed, to compare the deviation of the objective from the base point. A metric option is an element in the set X of product options against which all other options relative to the base point will be compared. For example, with the error objective, an option which exhibits the next smallest amount of error from the no-error option might be used to compare the remaining options in error from the zero error option. For instance, we can evaluate another option that has 4 times the error of the metric option as -4. Likewise when measuring product options based on simplicity, the most simple option can form an acceptable metric option and assigned a value of 1. The remaining options can then be rated between this simplest metric option and the most complex base point option. A design that is midway between the most simple design and the most complex design can be rated 0.5 in \simplicity." The need for identifying both a base point and a metric product option is axiomatic to forming interval scales. Further, interval scales are required to quantify how well dierent options perform. Therefore without two datum product options, quantitative ratings of alternatives are not possible. This is a fundamental measurement fact that must be incorporated into quantitative product evaluation. 8

3.1 Constructing Measurement Scales

There are many methods available for constructing interval scales over arbitrary sets. Two methods stand out as important, the basic form of the established lottery method [14], and a numerical form of the better/worse method made explicit in this paper.

3.1.1 The Lottery Method

To apply the basic lottery method1, a set of product options X is required, and an objective  [10, 13, 14]. This method will allow a designer to construct a real valued scale f that re ects the informal judgement . The designer must rst identify which options in X have the least and most amount of the objective , denoted x0 (the base point option) and x1 (the metric option) respectively. These are designated with amounts zero and one on the real valued scale being constructed. Then for each other element xi 2 X , the designer must answer the lottery question: \On a scale of zero to one, what is your belief f that you are indierent between: 1) receiving the objective performance provided by xi , or (Q3) 2) receiving the objective performance provided by x1 with certainty f and receiving the objective performance provided by x0 with certainty (1 ? f )?" This constructs a real valued (measurable) scale f : X ! [0; 1] that directly preserves the weak ordering of the elements of X , and also the relative separation. A dierence of 0:5 compared to 0:3 in f means a larger dierence in the informal objective . This basic lottery method also obviously assumes nite sets. On uncountable sets, further continuity assumptions are required for the procedure to be practical. The designer must provide lottery answers on a nite subset that can then be interpolated if an additional piece-wise continuity axiom is assumed. Examples of using the lottery method in design are given by Thurston [30] and Raia and Keeney [24]. The lottery method is one way of measuring the dierences among objects in an arbitrary set. The method starts with a worst case object and best case object in the set: the worst case object is used as the base point, and the best case object is used as the metric option. All other options are compared to the metric option relative to the base point. The measurement scale constructed is always positive, it has an image subset in R of [0; 1], f : X ! [0; 1]. Notice that an interval scale in and of itself does not speak to comparing dierent objectives in amounts. It rigorously constructs a scale for a single objective.

3.1.2 The Better/Worse Method

Rather than use the best and worst case product options as in the lottery method, it is also possible to use dierent, mathematically arbitrary options to construct the measurement. An arbitrary option, for example, might be the current product line. Doing so, however, requires the designer to measure in both positive and negative directions from the arbitrary base point using the arbitrary metric option: some product options may be worse than the base point. Therefore, if the designer selects arbitrary product options for measurement, questions of \as poor as the metric option was good" must be asked in addition to \as good as the

1 Usually the term lottery is reserved for when the options have probabilities associated with them. The method given is the reduced case of assured results.

9

metric option." A method similar to the lottery method (though again never precisely stated before this paper and so will now be called the better/worse method ) might be: given a base point option b and a metric option m (that has a dierent level of performance than b), for all x 2 X : \Using a real number f , how many times the metric option m do you believe option x is better/worse than the base point b in the objective ?"

(Q4)

Worse ratings get assigned negative numbers. This construction results in a scale whose image subset of R will have both negative and positive values: f : A ! R. A designer must be able to rate some options as worse than the base point option that was arbitrary and not necessarily the worst case.

3.2 Concept Selection Charts

A numerical concept selection chart [22, 31] is a simple device used by many designers to conduct a preliminary concept selection. With a numerical concept selection (sometimes called \Concept Scoring" [32]), the alternatives are listed along the columns of a matrix. The objectives that form the basis of the selection are listed along the rows. Entries in the matrix are the ratings of each alternative on each objective, and are assigned ranks from the set S = (??; ?; p; +; ++); or perhaps with additional or fewer multiple (+)'s and (?)'s. The set S has an understood scale placed over it: p (??)  (?)   (+)  (++): (3) In the evaluation process, the ranks are summed across the dierent objectives for each design concept, and the design with the highest sum is nominally determined to be the most promising candidate. When analyzing purely the mechanics of the method, we can clarify the need for four separate tasks when constructing the ranks in the chart: 1. Identify a base point and metric con guration 2. Construct a rating across the con gurations for each of the objectives 3. Normalize the dierent objective's measuring scales 4. Sum the normalized ratings for an overall evaluation of each con guration.

3.2.1 Example: Accelerometer

We can apply the mechanics of numerical concept selection method to the accelerometer example introduced earlier. Consider the rst task. When constructing the chart, again best design practice calls for a designer to rst select a reference design concept [22]. In re-design applications, this reference casepis usually the existing product or perhaps a competitor's product. This case is assigned ranks in every objective. Inpthis example, the base point can again be chosen as the \do nothing" option. It gets a for every objective. Every other option must be rated on each objective relative to this reference, choosing ranks from the rank set S . To do this rating, a metric option must be chosen that will be assigned a 10

(+) if it is better than the base point, or a (?) if it is worse than the base point. Thus, using numerical concept selection implies the better/worse method for constructing ratings. In this example, the metric case was chosen to be the \adjust backstop length" option and rated as a (+). We can now proceed to the second step. For each objective, the remaining options are now ranked, scaling o the base point in proportion to the deviation of the metric. The results are shown in Table 3. For example, on the cleanliness objective, the backstop length adjustment option allows for more contaminants to enter the product, and so is assigned a (?). Adjusting the spring also involves mechanical adjustment, and so is assigned the same rank. Adjusting the mass, however, involves material removal through grinding, and so is deemed to have twice the ability to contaminate the product, and is assigned a (??) rank. Notice that this implies an open-ended scale. If one concept is in nitely worse, it would have an uncountable number of (?)'s and ought to be dropped. If one concept has an uncountable number of (+)'s, it is in nitely better than the others and is the winner. If there are multiple designs with this feature, then the problem reduces to only considering these in nitely better designs. We must now combine these measurements. We cannot simply add them, however, since the objectives have dierent scales. We must \normalize" them. This is a concept that interval scales, in and of themselves, are not capable of handling. The interval scale provided a rigorous numerical construction that can then be quantitatively manipulated. The normalization process remains one steeped in academic argument, and oering a recommendation on best practice is beyond the scope of this paper. It is my position, however, that for any method presented to solve the problem, precise steps must be stated and axiomatically justi ed. This will allow a comparison of the axioms and thereby the restrictions that a designer must assume to use the method. An approach I have often used eectively in industrial practice is to limit each objective to a maximum impact in the evaluation, now called the maximum impact normalization. This is shown in Table 4 for the accelerometer example. These values from S are also used to modulate the relative values determined from the better/worse construction. The better/worse method constructed a relative scale among the concept options. To combine the ranks into an overall gure-of-merit, the scale ranges must be modulated so they are \appropriate." To do this when using the maximum impact normalization, multiply each value by the maximum allowed eect, and divide by the dierence between the highest and lowest score. For example, the third row in Table 3 is multiplied by 0 ?2?4 to give the values in the third row of Table 5. The complete modulated results for the entire evaluation are shown in Table 5. Notice this approach is therefore mathematically independent of the base-point and metric options used. Practically, of course, this is not so due to \round-o" errors. Is p (+1=2) a or a (+)? Clearly, designer judgement must enter the mechanics of the process. Completing the normalization task allows the nal step 4 to nish the concept evaluation. The modulated values are summed. The evaluation process provides a recommendation for the design concept with the highest sum.

3.2.2 Analysis of Concept Selection Mathematics

Analyzing this rigorous form of numerical concept selection, several features are clear. First, the concept selection method has pa clear base point for each objective, namely the reference case that is assigned a rank of ( ). Second, any design assigned a rating of (+) can be 11

Table 3: Interval Measurement Chart of Product Options. Timing Quality Production Cost Complexity Cleanliness

Do Adjust Improve Adjust Nothing Length Spring Mass p + ++ + p + + + p

? ? ?

p p

Base Point

Metric

? ? ?? ??? ?

?? ? ??

Table 4: Objective Maximum Allowed Eect. Maximum Eect Objective Timing ++ ++ Quality ++ Production Cost Complexity + + Cleanliness

Table 5: Interval Measurement Evaluation of Product Options. Timing Quality Production Cost Complexity Cleanliness Sum (+) Sum (?) Sum

Do Adjust Improve Adjust Nothing Length Spring Mass p + ++ + p ++ ++ ++ p p

p p

p ?

0 0p

3 1 ++

12

?? ? ? 4 4p

? ? ?

3 3p

used as a metric option. That is, for numerical evaluation two concepts are needed to rank the remaining concepts. Merely identifying a datum is insucient. That two concepts are needed to construct numerical concept selection ranks (and not just one datum concept) is a measurement fact not commonly recognized. After identifying the base point option and a design with a (+) rank, each subsequent design can be rated as, for example, \twice as good as the metric option from the base point option" and assigned a (++), or \as bad as the metric option was good from the base point option" and assigned a (?), using the better/worse method. Note that this is the interpretation that must be associated to these ranks: that is what the ranking level mathematically contributes. When restricted to the rank set S = f+; p; ?g, a question arises over whether the evaluation mechanics are ordinal or numerical. Such is the case with Pugh's method. Do p the ranks f?; ; +g only order the designs, or do they provide information re ecting how much better or worse the dierent designs are? The answer depends on the interpretation of the nal overall gure-of-merit. If the resultant overall gure-of-merit is interpreted as the number of objectives that a design is better/worse at, then the individual ranks are ordinal. If on the other hand the nal gure-of-merit is interpreted as a numerical value of \overall goodness," then the individual ranks are being manipulated as numerical values. A (+) is better than a p to the same degree that a (?) is worse than a p. There is no getting around this fact if the values are summed: a (?) cancels a (+). If the scale were only ordinal, it would not be known how much of the (+) a (?) could cancel. Thus, the assumption when using concept selection charts with only a rank set of f+; p; ?g is that all the objectives have equal importance, or that the nal gure-of-merit represents the number of objectives. This is a subtle but signi cant fact and a cause of much confusion. An issue, of course, is the quality of information available. At the preliminary design phase, often all that can be speci ed is an ordinal rank. The point here is to clarify how the methods work to determine an evaluation. If at the preliminary design stage the information available only permits the assignment of an ordinal rank, then the overall gure-of-merit must be \net number of objectives." If better information is available, then interval scales can be constructed. Understanding this and using the appropriate scale and overall gureof-merit interpretation for the quality of information available de nes a best practice. Quality of information is an inherent judgement which must be made to determine which scale should be used. The next section will introduce and discuss the next most complex scale and compare the quality of information available and the quality of the provided result.

4 Ratio Scales Interval scales construct a quantitative numerical evaluation over an arbitrary set with arbitrary objectives. The next level of structure that can be considered is an objective that inherently has some reason for anchoring the numerical scale. An interval scale does not de ne a \zero" value for the objective. Instead, an arbitrary value (usually zero) is assigned to the base point option, with the understanding that any arbitrary constant could be \added" to this value, and the results of the evaluation are invariant. The evaluation result is dependent on the numerical dierences, not the absolute numerical values. Rather than assign the base-point option a value of zero, perhaps there is a speci c value of the objective that is understood as \zero." Consider evaluation p based upon the contamination objective. No contamination can be assigned the value , not an arbitrary 13

base point con guration that may cause some contamination. Such a \zero" is a base value of an objective, rather than the value of the objective assigned to a base point con guration. If there is a naturally de ned base value for the objective, there is more structure inherent to the problem. The base value can be used in the measurements. There is still no unit of measure associated to the scale, since there is not sucient structure to naturally de ne a metric value. What is a unit of contamination? It is not easily de ned. There is, however, more structure to operate with than a simple interval construction. \No contamination" is easily understood.

4.1 Constructing Ratio Scales

The procedure for constructing a ratio scale is very similar to constructing an interval scale. The only dierence is that the objective value returned by the base point con guration is replaced by the base value of the objective. Given a con guration m 2 X chosen as a metric con guration and a base value 0f of the objective , the better/worse method can be used to construct a rating over all other con gurations x 2 X : \Using a real number f , how many times the metric design m do you believe option x is better/worse in the objective  than the base value 0f ?"

(Q5)

Worse ratings get assigned negative numbers. This construction results in a scale whose image subset of R will have both negative and positive values: f : X ! R. this scale, no single con guration x 2 X may end up getting assigned a 0 value (a p). With Since it is known what 0 means on this scale for , it may be dierent from the values of the considered con gurations. Now three examples of ratio scales commonly found in product development will be examined: QFD, concept selection charts and methods to determine importance.

4.2 Example: Quality Function Deployment

Quality function deployment (QFD) [1, 11] is a matrix method used in product development practice to ensure customer satisfaction through proper problem formulation and allocation of resources. It is commonly used by product development teams to form consensus on the functional requirements that a product must satisfy and their priorities. Again it is useful to clarify what underlies the mechanics of the method. Consider the basic House of Quality matrix. The requirements desired by a customer are determined: they are identi ed as a set. These customer requirements are placed along the rows of the House of Quality matrix. The relative importance of each requirement is also determined, and entered in a column. Then the performance objectives that are to be included in a design are also identi ed (thus forming a set), and placed along the columns of the House of Quality matrix. Identifying a complete set of objectives is actually one of the purposes in constructing the QFD matrix. Given this structure, the degree to which each performance objective satis es each customer requirement must be identi ed. This is done by ranking each performance objective's ability to satisfy each customer requirement, using a non-numerical rank from the rank set

S = (0 0 ; 4; ; ): 14

(4)

Ranks from this set indicate the ability of a performance objective to satisfy the customer requirement. A blank indicates the objective does nothing, and a indicates that objective fully satis es the customer requirement. These symbols obviously provide an understood ordinal rank, with 0 0  4   : (5) Given the completed House of Quality chart speci cation, the ( ; 4; ; ) symbols are assigned numerical equivalents. That is, a map t is used to convert the symbols to real numbers, t : S ! R. Typically, t( ) = 9 t( ) = 5 (6) t(4) = 3 t(0 0 ) = 0: Since 0 < 3 < 5 < 9; (7) t preserves the ordering of (5). Given this, it is clear that QFD involves at least an interval scale: the relative separation among the values is known from t. Thus QFD involves at least interval measurement. The question then becomes the nature of the assignment of a blank value. It is commonly understood that a blank is assigned if a target speci cation does nothing toward satisfying the particular customer requirement in question. Thus, if zero amount of customer satisfaction is provided by this target speci cation then leave the matrix entry blank. Clearly the blank assignment represents \zero" value in providing customer satisfaction. This means that QFD involves more than an interval scale, it uses a full ratio scale: a zero value is used to anchor the scale. Thus, to complete a House of Quality chart, designers must consider each customer requirement one at a time. They must identify what adding \zero customer satisfaction" means for the customer requirement. Then for the target speci cations that provide more than zero satisfaction, a maximum must be assigned a to provide a metric, and nally the remaining in-between target speci cations that provide non-trivial customer satisfaction can be assigned values with the lottery method (Q3), rounded o to two values f ; 4g. The two in-between symbols used in QFD are to be subjectively associated to performance targets that provide t(4)=t( ) ' 13 and t( )=t( ) ' 59 percent satisfaction of the customer requirement, as compared to ones that provide customer satisfaction. Thus in this approach, QFD assigns value using the lottery method (Q3). When using a House of Quality matrix, the real values given from t are scaled by the importance weightings. Clearly if dierent values are used for t, then a team will determine dierent results. It is therefore important to choose the weights and map t properly. Reasonable means will be discussed in Section 4.4. A potential problem with this QFD ranking can be identi ed with the assignment of the base value as \no customer satisfaction." Using this, there are two possible interpretations of what a symbol in the matrix means, and indeed they are both found in the literature. The rst interpretation is that a symbol re ects absolutely how much a target speci cation aects the customer requirement. The second interpretation is that a symbol re ects directly how much a target speci cation aects the customer requirement. The disagreement is over whether the sign of the eect should be included or not. If a target speci cation causes a strong negative eect on a customer requirement, is it assigned a ? Or should it be 15

Table 6: Ratio Measurement Evaluation of Product Options. Timing Quality Production Cost Complexity Cleanliness Sum (+) Sum (?) Sum

Do Adjust Adjust Adjust Nothing Length Spring Mass p

?? ?? p p p

? p p p ?

0 4 ?4

0 2 ?2

p ?? ? ? 0 4 ?4

? p ? ? ?

0 4 ?4

assigned a ? ? In Hauser and Clausing's article on QFD [11], rather than the rank set S from (4) they suggest that sign should be included and use a rank set S = f; ;0 0 ; p; ppg where each is given an interpretation of pp ' strong positive p ' medium positive 0 0 ' no eect  ' medium negative  ' strong negative: Thus Hauser and Clausing noted the need to measure positive and negative eects. Others currently do not [1, 32]. If sign is included, the QFD assignments can be made using a better/worse method (Q5). If sign is not important, then the QFD assignments can be made using a lottery method (Q3). Generally it is better to not lose information, such as whether an engineering characteristic positively or negatively aects an customer requirement. Ratio scales not only appear in QFD to determine product objectives, but are sometimes used in concept evaluation. We can analyze the validity in doing so.

4.3 Example: Concept Selection Charts

Reconsider the accelerometer example introduced earlier. Suppose each of the objectives has a naturally de ned zero: timing performance is best when every resulting product actuates precisely on target with zero deviation. Quality has a zero of no added defectiveness. Product cost has a zero of no added expense. Complexity has a zero of no added complexity beyond existing facilities. Cleanliness has a zero of no contamination. These objectives, on the other hand, may not necessarily have a metric value. Using this additional structure of the problem, the concept evaluation can use the ratio scale better/worse method (Q5). The resulting numbers are then dierent as shown in Table 6. The results on this absolute scale more clearly demonstrate the dierence between using ratio scales or interval scales when evaluating concepts. Compare the resulting sums of Table 5 with Table 6. The relative spacing of the options and the nal determination remain the same. However, the resulting gure-of-merit with the ratio scales are all negative. A 16

designer can well ask if any of the concepts should be pursued, or whether more concepts need to be generated. The ratio scale results of Table 6pare all negative because the scale on every objective had a base point of perfection. The on each scale was the perfect result. Thus, the nal p sum gure-of-merit had a of perfection. Of course all physically possible options are worse than the implied imaginary base point con guration that can deliver perfect results. In the example, such a utopian option would not require any modi cations of the existing product production methods and yet magically have it solve the quality problem at no cost. Thus the prime dierence when using a ratio scale for concept selection: a datum base point con guration is not required, since a zero for each scale is already known. The nal summed number then represents an evaluation on the option relative to an imaginary utopian base point con guration, rather than a evaluation compared to an actual base point con guration. Thus, unlike with an interval scale that uses an actual base point, the fact that there may be no (+)'s (that the nal overall summations result in all negative numbers) is not an indication of poorly performing results. Pugh also argues that one should always use a base point con guration when using concept selection charts [22]. This can now be interpreted as an argument against using ratio scales for concept evaluation and instead always using interval scales. A base point con guration, rather than any available zero values of the objectives, is deemed more useful for anchoring an evaluation. This line of reasoning is further clari ed by the ratio scales. The nal summations are dicult to interpret, as shown in Table 6. They are compared to an imaginary utopian base point that is usually dicult to consider. Thus, a seemingly paradoxical conclusion can be drawn: even in situations where more structure is available for an evaluation (in the form of a zero value for an objective), it should be ignored and only interval scales used (with a base point datum con guration). This conclusion is drawn from a purely psychological reason, not from the underlying mathematics. Mathematically the methods have no dierence in information content of the result, the relative separations are the same. The dierence must be compared in a psychological frame, and I have found the interval approach more comprehensible for product development.

4.4 Example: Importance

Another example of using a ratio scale involves the common product development task of establishing the relative importance of a single objective among multiple real-valued objectives. Importance can be represented using normalized weighting coecients , in such methods as goal programming [8] or weighted sums [29]. While there are many arguments against using weighting factors in any form [29], it remains true that they are commonly used in product development practice. Sound methods of construction should be developed. Rating importance must involve a ratio scale. Importance can be considered an objective to be evaluated across the option set  of multiple objectives. Zero importance is a well understood concept as an objective with zero importance is not relevant to the decision. A unit of importance, however, is not well understood. Thus, importance falls into the ratio scale domain. A procedure for determining importance is to apply the ratio scale construction question. This is the well known marginal rate of substitution question [29]. One objective f1 must be selected as a metric case, either arbitrarily or perhaps by the one that is \most understood."

17

Then for all of the other objectives fi ask: \By how much should fi be increased to compensate for a loss of one unit in f1 ?"

(Q6)

This de nes relative amounts of importance
i for each fi. Notice the inherent dependence on the amount
1 = 1 used. A problem with weights is that the results can change with smaller or larger
1 . In any case, the results can be scaled by the sum to de ne absolute importance wi = P

i : j

For example, in the concept selection charts discussed previously, the importance ranges used (Table 4) can be constructed using this approach. Similarly, with QFD and small sample sizes, the customer requirement importance factors can be determined with this question. Notice that this implies that importance ratings can only be constructed after the individual ratings are completed (Table 3), so that units of each objective are determined.

4.5 Other Examples of Ratio Scales

Other examples of ratio scales in design can be found. For example, in his early undergraduate design text, Siddall [27] discussed a graphical technique that, for every objective in a decision, starts by assigning a 10 to the best value and a 0 to the worst value. The remaining are graphically interpolated in-between, similar to the better/worse question for ratio scales. This construction is used so that one can add the results into a weighted sum overall gure-of-merit. This approach is commonly used for normalizing real-valued objectives. Freiheit and Rao [9], Mistree [3, 16, 33], Biegel and Pecht [5], Dlesk and Liebman [7] and many others, for example, all use a similar construction to normalize real-valued goals for subsequent incorporation into an overall gure-of-merit. Basically, for each real-valued goal fj a normalization is de ned as Fj (~x) = ffj (~x)??ffjl ju jl where fju is an upper metric value, and fjl is a lower base value. Typically, the lower base value fjl is either zero or a \threshold of acceptance" value. Mistree uses zero, Biegel and Pecht use a value that is known from a failed ~x. The upper metric value fju is de ned by either a best possible value, or by the highest achievable value. The best possible value diers from the highest achievable value in that the best possible may never be attainable (e.g., 100% eciency). The highest achievable value is the highest attained over the search space X (de ned by the design variables and perhaps other constraint functions ~g) when optimizing fj exclusively. For a minimization problem, this is

fju = inf ffj (~x) j ~x 2 X g:

(8)

An issue that arises is which normalization should be used. What value should be used for fju and fjl ? Using measurement theory, a ratio scale is implied if the zero and best possible value are used. However, it may be that no concepts ~x 2 X can actually produce these values. Thus, interpretation during actual practice becomes more dicult. The alternative is to use (8) and a \threshold" value to de ne the scale. An interval scale is implied if the threshold value is actually achievable, i.e. an ~xl 2 X exists such that 18

f (~xl ) = fjl . Using the interval normalization de nes a best practice for ease of designer interpretation . That is, the resultant Fj de nes the achievement on goal fj percentage-wise as compared with the dierence from what is minimally acceptable to what is maximally possible. On the other hand, it is more dicult to formulate the interval normalization than the ratio normalization. Interval normalization requires an up-front optimization as in (8) to formulate the scale. Finally, another decision-making method used in design is Saaty's Analytic Hierarchy Process (AHP) [26]. AHP is well suited to problems whose objectives have a hierarchical structure. For example, when cost can be broken down into material cost, manufacturing cost, development cost, etc . AHP has been successfully used in both design [15] and manufacturing [23]. AHP operates by formulating a weighted sum gure-of-merit for the overall decision using the highest considerations in the hierarchy. V (~y) =

X w V (y ) n

j =1

j j

j

There are two types of values which must be determined, the weights wj and the values Vj . AHP uses a ratio scale to determine the weighting factors wj [10]. The weighting factors wj for each objective j are determined using relative comparison factors aij by comparing to every other related objective i using an absolute scale f1; : : : ; 9g. Each Vj is determined by breaking down j into components and then locally at each level repeating the relative comparisons, which are combined into the parent hierarchy. A relative comparison factor of 1 means both objectives are equally important. A 9 means one objective is vastly more important than the other. Thus, there is a base and a metric value. As before, determining weight requires a ratio scale. Values in between signify the level of grayness in relative importance. These relative comparison factors aij are then normalized into a weighting wj in [0; 1]. The values Vj on each objective yj = fj (~x) are determined by expanding each objective into component objectives, which can be recursively repeated for multiple levels in the hierarchy. At the leaf nodes a value f must be assigned. AHP can operate with any numerical scale f to determine value on the leaf node objectives, the mathematics of AHP operates with the dierences in these values. If these objectives are measured on a realvalued scale complete with units then the approach is clear. However, if the objective has no such measuring system then a numerical scale must be constructed, and again the issue of the type of scale arises. A feature of AHP is that numerically it does not matter. Both a ratio or an interval scale can be used, and in theory there will be no dierence in results. This is due to the same reason that the results of Table 5 and Table 6 were invariant (the same solution was evaluated as highest), namely the evaluation depends only on the dierences, not the absolute numerical values. This means that to construct the leaf node AHP rating, either a base point design or a base value can be used, and either a metric design or a metric value can be used. The relative separations as elicited from the designer should always be proportionally the same, and that is what the AHP results depend upon. On the other hand, the real issue is whether a designer can realistically provide numbers. As previously argued, using con gurations as datums for comparison provides better results. So far the discussion has centered upon constructing scales for objectives which have no structure. The next section will discuss objectives which come complete with \units." 19

5 Extensively Measurable Scales In many cases, the appropriate scale for rating dierent product options is apparent and given: the set X of elements in question has additional structure that makes the scale clear. Methods of constructing scales such as the better/worse method are not required. For example, designs that are already represented with real numbers can use the real numbers to order the designs. Cost, for example, has a base point of no cost and a metric of a dollar, if the design is rated to the nearest dollar. Likewise, length has zero length as a base point, and may have a metric of a thousandth of a inch. What these sets have in common is they can be extensively measured: they have a base value, a metric value, and a well de ned concatenation operation [14]. There is a well de ned operation over the set, generally called concatenation, that can be used to \add" the metric value (called a unit of measure in this case) recursively to the base value to produce any other value in the set. The unit of measure de nes the fundamental \units" as commonly understood in engineering practice.

5.1 Example: Optimization Methods

Numerical optimization methods are commonly used in engineering design [21]. The typical optimization problem involves formulating the problem into the standard null form, described by a search space, objective function, and constraint functions: min f (~x); ~x 2 X subject to

~g(~x)  ~0 ~h(~x) = ~0 where X is the search space of design con gurations, f is the objective function, ~g are inequality constraints, and ~h are equality constraints. Analyzing this formulation in measurement theory terms, it is clear that the real valued order  on the range of f forms an extensively measurable scale. The value 0 2 R is the base value, and the metric value is the smallest value  discernible from zero. The concatenation operation is real valued addition. Thus, an optimization formulation implicitly de nes a measurement of the designs forming the search space: designs with lower f are better. For multi-objective problems, dierent scales are used. This presents a problem; either the dierent scales must be combined into an overall gure-of-merit or they must be tradedo by the designer interactively. This question concerns the combination of dierent scales, and is beyond the scope of the paper. However, it is worth noting that all methods that combine multiple objectives into a single gure-of-merit do so by reducing the extensively measurable scale into a ratio or interval scale. That is, some form of (8) is always used to normalize each extensively measurable scale into a scale that can be relatively compared by the designer across the dierent objectives, or can be multiplied by a weighting factor. This holds true whether operating with utility theory [10, 30], imprecision [18, 19, 20], goal programming [3, 9], weighted sums [7], or any method that uses an overall gure-of-merit over real-valued objectives. The discussion is now complete for numerical evaluations. There are times, however, when evaluations are made without numbers. The next section will discuss these methods. 20

6 Non-Numeric Scales So far the discussion has been restricted to constructing numerical ranks. The extension to more general non-numeric image sets can be considered. For example, performance metrics might be represented with simple statements of \feel" from customers, rather than with numerical ranks. Customer questionnaires typically use non-numerical scales, since contexts need to be supplied for meaningful answers [12]. To use such non-numerical ranks to evaluate options, sucient structure to provide an order of the options must be known among the statements. That is, the non-numerical ranks must themselves have an order among them to place an order over the options. This simply says that among the statements a customer provides, knowledge of how the statements re ect the performance must be known. Without this order, the statements cannot be used to induce an order onto the options (evaluate the options). Mistree [3], for example, notes this in his discussion on the ratio scale construction required to use his \Decision Support Problem Technique." He states that ordinal scales are appropriate for objectives that can only be modeled in words, and not with numbers. They are minimally equivalent (though in fact not always, when the words provide an indication of relative separation). Any extension to such non-numeric sets is for convenience only. Non-numerical methods of evaluation oers no more theoretical generality than R. It is provable that, given a weak order over a set, one can always construct a real valued map t which preserves this ordering [14]. Thus, non-numeric ranks provide no more expressiveness or generality. For example, in QFD the symbol set (0 0 ; 4; ; ) is non-numeric. But it also has a total ordering, which was re ected by the map t of Section 4.2. Such maps t are guaranteed to exist when an ordering over the symbol set exists. Further, when the ordering over the symbols set does not exist, then the symbols cannot be used to order the design con gurations. Thus using statements or symbols rather than numbers provides no additional generality. They may, however, provide more convenience in their construction, which is usually a very big concern with evaluations from third parties such as customers. For the actual design evaluation, the non-numeric ranks can be converted directly into numerical equivalents. This approach has strong appeal because it can leave the imprecision in the evaluation. Mistree [16], for example, present a hierarchical method using fuzzy sets to convert non-numeric evaluations into numbers.

6.1 Example: Research Models

Measurement also plays a role in many current research paths in engineering design. Language translators, grammatical approaches, and feature based design, for example, all use measurement as a fundamental aspect. Language translator models have been proposed for converting between dierent descriptions of designs, using, for example, features [25, 34]. Features in one design description are converted into descriptions of another kind. High level functional descriptions might be converted into catalog part numbers [34]. Feature based CAD database descriptions might be converted into manufacturing assembly descriptions [25]. Measurement plays a role in these research initiatives. When high level functional descriptions are converted into part descriptions, measurement of the satisfaction a design provides in the original functional descriptions must be made. Objectives must be de ned using the part descriptions that correspond to the high 21

level functional descriptions. These objectives must be measurable, so that an ordering between candidate components can be established, and a best component selected. When the design descriptions are converted into manufacturing descriptions, then again measurements are used. Manufacturing features within CAD drawings must be recognized. Pattern matching is a proposed mode for this recognition operation [25], and the degree of t is a measurement.

7 Conclusions Measurement theory oers a rigorous framework to choose among dierent design evaluation methods. This discussion clari ed many traditional engineering design evaluation methods, and de ned best practices. For example, the proper interpretation of the gure-of-merit in Pugh's method is the number of objectives. A base point con guration should always be used with ordinal evaluation. Two con gurations must be considered to form a numerical rating, (the base point and metric con guration). Base values of any objective should be ignored in a concept evaluation, and instead a base point con guration used. QFD without negative assignments must be interpreted as impact without direction. Multi-objective optimization can be normalized with either a ratio or an interval approach. The interval normalization is better for ease of interpretation. Determining the relative importance of objectives necessitates a ratio scale. Multi-objective optimization metrics always reduce the scale on each objective to at most a ratio scale (even if an extensively measurable scale exists). Finally, this discussion has also clari ed requirements of research in design. The models presented by researchers should strive to more clearly identify the sets used, the discriminating objectives considered and the measurement systems. Doing so will foster better communication, understanding, and results.

Acknowledgements

This work was made possible in part by a Sloan initialization grant from the Department of Mechanical Engineering at the Massachusetts Institute of Technology, funded in part by the DuPont Corporation. The author also wishes to acknowledge the support of the Leaders for Manufacturing Program, a collaboration between MIT and U.S. industry. Any opinions, ndings, conclusions, or recommendations are those of the author and do not necessarily re ect the views of the sponsors.

References [1] Y. Akao. Quality Function Deployment: Integrating Customer Requirements into Product Design. Productivity Press, Cambridge, 1990. [2] M. Asimow. Introduction to Design. Prentice-Hall, Inc., Englewood Clis, N.J., 1962. [3] E. Bascaran, R. Bannerot, and F. Mistree. Hierarchical selection decision support problems in conceptual design. Engineering Optimization, 14:207{238, 1989. [4] P. Bernays. Axiomatic Set Theory. North-Holland, New York, 1968. 22

[5] P. Biegel and M. Pecht. Design trade-os made easy. Concurrent Engineering, 1(3):29{ 40, May/June 1991. [6] M. Davis. Computability and Unsolvability. McGraw-Hill, 1958. [7] D. C. Dlesk and J. S. Liebman. Multiple objective engineering optimization. Engineering Optimization, 6:161{175, 1983. [8] H. Eschenauer, J. Koski, and A. Osyczka, editors. Multicriteria Design Optimization. Springer-Verlag, Berlin, 1990. [9] T. Freiheit and S. S. Rao. A modi ed game theory approach to multi-objective optimization. In S. S. Rao, editor, Advances in Design Automation - 1988, volume DE-14, pages 107{114, New York, September 1988. ASME. [10] S. French. Decision Theory: An Introduction to the Mathematics of Rationality. Ellis Horwood, Chichester, 1988. [11] J. Hauser and D. Clausing. The house of quality. Harvard Business Review, pages 63{73, May/June 1988. [12] B. Hayes. Measuring Customer Satisfaction: Development and Use of Questionnaires. ASQC Quality Press, Milwaukee Wis., 1992. [13] R. Howard and J. Matheson, editors. The Principles and Applications of Decision Analysis. Strategic Decision Group, 1984. [14] D. Krantz, R. Luce, P. Suppes, and A. Tversky. Foundations of Measurement, volume I. Academic Press, New York, 1971. [15] E. Marsh, A. Slocum, and K. Otto. Hierarchical decision making in mechanical design. Engineering Design Laboratory Technical Report 93-6, Massachusetts Institute of Technology, Cambridge, MA, 1993. [16] F. Mistree et al. Fuzzy compromise: An eective way to solve hierarchical design problems. Structural Optimization, 4:115{120, 1992. [17] D. Nicklaus, S. Tong, and C. Russo. Engineous: A knowledge directed computer aided design shell. Proceedings of the Third IEEE Conference on AI Applications, February 1987. [18] K. N. Otto and E. K. Antonsson. Trade-O Strategies in Engineering Design. Research in Engineering Design, 3(2):87{104, 1991. [19] K. N. Otto and E. K. Antonsson. Modeling imprecision for product design. In Proceedings of the 1994 FUZZ-IEEE Conference, pages 340{251, Orlando, FL, June 1994. IEEE. [20] K. N. Otto, A. D. Lewis, and E. K. Antonsson. Determining optimal preference points with dependent variables. Fuzzy Sets and Systems, 60(1), November 1993. [21] P. Papalambros and D. Wilde. Principles of Optimal Design. Cambridge University Press, New York, 1988. 23

[22] S. Pugh. Total Design. Addison-Wesley, New York, 1990. [23] R. Putrus. Non-traditional approach in justifying computer integrated manufacturing systems. In Autofact '89 Conference Proceedings, pages 8.1{8.26, Detroit, MI, October 1989. SME. [24] H. Raia and R. Keeney. Decisions with Multiple Objectives: Preferences and Value Tradeos. Wiley, New York, 1976. [25] D. Rosen and J. Dixon. A methodology for conversions of feature-based representations. In L. Stauer, editor, Proceedings of the ASME Design Theory and Methodology Conference, pages 45{51, Miami, 1991. ASME. [26] T. Saaty. The Analytic Hierarchy Process. McGraw Hill, New York, 1980. [27] J. N. Siddall. Analytical Decision Making in Engineering Design. Prentice-Hall, 1972. [28] H. Simon. The Sciences of the Arti cial. MIT Press, Cambridge, MA, 2 edition, 1981. [29] R. Steuer. Multiple Criteria Optimization: Theory, Computation, and Application. J. Wiley, New York, 1986. [30] D. Thurston. A formal method for subjective design evaluation with multiple attributes. Research in Engineering Design, 3(2):105{122, 1991. [31] D. G. Ullman. The Mechanical Design Process. McGraw Hill, New York, 1992. [32] K. Ulrich and S. Eppinger. Product Design and Development. McGraw-Hill, New York, 1995. [33] S. Vadde, R. Krishnamachi, F. Mistree, and J. Allen. The bayesian compromise decision support problem for hierarchical design involving uncertainty. In G. Gabriele, editor, Advances in Design Automation, pages 209{217. ASME, 1991. Volume DE-32-1. [34] A. C. Ward and W. P. Seering. Extending the constraint propagation of intervals. Arti cial Intelligence in Engineering Design and Manufacturing, 4(1):47{51, 1990.

24