Measuring the performance of electronic ... - Semantic Scholar

Decision Support Systems 54 (2013) 1065–1075

Contents lists available at SciVerse ScienceDirect

Decision Support Systems journal homepage: www.elsevier.com/locate/dss

Measuring the performance of electronic marketplaces: An external goal approach study Sabine Matook The University of Queensland, UQ Business School, Brisbane, QLD 4072 Australia

a r t i c l e

i n f o

Article history: Received 17 November 2011 Received in revised form 7 October 2012 Accepted 21 October 2012 Available online 26 October 2012 Keywords: Electronic marketplace Performance Organizational goal Goal achievement Success Failure

a b s t r a c t Electronic Marketplaces (EMPs) are virtual intermediaries that facilitate exchanges of products between buying and selling organizations. Prior research has examined factors that explain the establishment of EMPs, but scant knowledge exists on how to determine their performance. This study draws on organizational theory to measure EMP performance, thereby utilizing the concept of goals and goal achievement as criteria for success or failure. In this empirical work, we classify EMP as successful or unsuccessful and compare this classification with data collected eight years later. Our research identifies critical EMP goals and promotes the use of goals to determine performance. © 2012 Elsevier B.V. All rights reserved.

1. Introduction The advent of the Internet has resulted in new ways of doing business as well as new business forms – so-called information technology (IT)-centric businesses [100]. Electronic marketplaces (EMPs) are one representation of an IT-centric business because of the role that technology plays in their foundation and continuing existence. EMPs are virtual, technology-enabled trading spaces that facilitate the exchange of information, goods, services, and payments among multiple buyers and sellers [63]. ChemConnect, cc-hubwoo and Quadrem are successful examples of business-to-business (B2B) EMPs that not only survived the dot.com crash in 2001 [25] but have established themselves as leading trading platforms [97]. In fact, in their recent meta-review of EMP research, Standing et al. [97] report that EMPs “have a significant role to play in business”, and that research on EMPs has been and “continue[s] to be a vibrant research topic” (p.41). Much of the research on EMPs to date has focused on understanding what is required to establish these IT-centric businesses [7,63,90,96]. This knowledge of EMP design factors was needed initially because of the intertwined and symbiotic nature of the business and the technology. Today, EMPs have manifested themselves in the e-commerce landscape as an accepted business form that must ensure its viability and performance in the same way as any traditional organization [61]. For an EMP, information on its performance is important because it allows the organization to understand the extent to which EMP is efficient in

E-mail address: [email protected]. 0167-9236/$ – see front matter © 2012 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.dss.2012.10.032

undertaking its tasks and meeting set goals, while simultaneously continuing to innovate and grow [71]. Thus, EMPs measure performance to obtain reliable and valid information about an organization's current performance [102]. With this information, an EMP is able to evaluate, compare, motivate, promote and manage performance to eventually improve its performance [10]. Having knowledge of an EMP's performance also allows for improved planning of the organization's strategic direction and supports more informed decision making regarding allocation of resources and budgeting. However, only a few studies have been directed towards understanding and measuring the performance of an EMP [see, for example, 36,70,94]. Additionally, these studies are either qualitative in nature or measure performance with a focus on either technology aspects or business factors. Consequently, the symbiotic relationship of the two perspectives has been overlooked and, as a result, EMP performance has not been captured comprehensively. For decades, measuring the performance of an organization has been an important research agenda in management and organizational theory [83]. Over time, numerous models of organizational performance have been proposed [see for example, 13,31,45,79,111,112]. Among the various models, an approach that utilizes organizational goals to determine performance is most notable because it is a common and most effective way to measure performance [99]. In a nutshell, a goal-based approach measures performance as a function of goal achievement. An organization uses its pre-defined goals to assess the degree of goal attainment and then to determine its performance based on the attainment degrees. By pre-defined goal, the literature means “a desired state of affairs which the organization attempts to realize” [31]. Using goals to

1066

S. Matook / Decision Support Systems 54 (2013) 1065–1075

measure performance instead of accounting measures (e.g., cash flow or sales) provides the benefit of determining performance at times when objective measures are not available — a common aspect for IT-centric businesses [21,92]. Against this backdrop, this study draws on organizational theory and, in particular, the external goal approach to determining the performance of an EMP. To tailor the goal approach to the domain of EMPs, we first determine EMP's goals. These goals are then mapped into four views of Matook's and Vessey's [63] multi-layered EMP model to represent the different (i.e., IT and business) perspectives of an EMP. In an initial proof-of-concept study, we illustrate the application of the external goal approach to 20 B2B EMPs. Our findings are threefold: (1) we identify EMP goals; (2) we apply the external goal approach as a theory-grounded approach to evaluate EMP performance; and finally, (3) we suggest that by using the strategic goals of an EMP, its performance can be determined. This research makes several contributions to theory and has a number of practical implications. With this study we contribute to the body of literatures on EMP e-commerce and organizational theory by providing knowledge on EMP goals and their importance in regard to performance measurement. Although performance of organizations has previously been determined using goals, for example in marketing and management research [2,5], this study is the first to apply the external goal approach to the information systems field. Findings of the study have a number of implications for practitioners. First, EMPs can use our list of identified EMP goals to compare and reflect on the set of goals they currently pursue. Second, we demonstrate how to use the performance measurement approach and thus, EMP operators can use it to measure their performance. Any other firm needs to identify the goals relevant to their domain before the approach can be used. Third, with this study we illustrate how the approach can be used not only for performance assessments, but also for goal comparisons and adjustments. The remainder of the paper is as follow: First, we present prior research on EMPs and performance measurement approaches. Then, we derive from the literature a set of 16 goals that are then mapped into four views of the EMP model. The empirical study is then presented and results and findings are discussed. The paper concludes with contributions, limitations, and suggestions for future research.

achievement strategy may further contribute to EMP failures [16]. Consequently, research needs to focus on the measurement of EMP performance to facilitate the evaluation of current performance and enable improvements to this performance. Indeed, regular performance measurement provides EMPs with the possibility that the goal achievement outcomes will be monitored closely to allow for adjustments and detection of areas of low goal achievement. Prior research on EMP performance has drawn mainly on the resource-based view [8,111] which is a specific input-focused organizational performance model. While using such a model, however, little to no attention has been paid to the performance outcomes of utilizing those resources [8]. The resource-based model evaluates performance as the ability of an organization to obtain, integrate, and manage scarce and valued resources [107]. For example, Laseter and Bodily [52] developed a conceptual model for evaluating the financial performance of EMPs that builds on the resource-based view and transaction economics. Soh et al. [94] extend their work on the resource-based view with a strategic alignment theory that bridges the resource-based view and Porter's theory of competitive advantage [77]. The use of a resource-based view, then, appears to be appropriate in industries in which resources are scarce. It may not apply, however, to the majority of businesses where throughput or outcomes are more central to performance. In particular, it appears unlikely to apply to an EMP where suppliers, and not the EMP, choose the products that will be offered for sale. A noteworthy exception to the narrow use of the resource-based view is the study by Gengatharen and Standing [36] who used a number of additional theoretical foundations, such as ‘innovation diffusion models’, the ‘technology acceptance model’, and the ‘unified theory of acceptance and use of technology’ to create an integrated framework that guides their qualitative study on performance of governmentsupported EMPs. Although their study provides valuable insights into the complex interplay of technical and economic factors in relation to EMP performance, Gengatharen and Standing [36] conclude that their results “do not answer the question of how success of a governmentsupported SME-REM can be measured” (p.431). Consequently, this current study aims to answer the question of how to measure performance to determine success.

2. Background

2.2. Goal-based performance measurement approaches

2.1. Electronic marketplaces: Performance and success of IT-centric businesses

Organizational performance has been described as the most important dependent variable of interest in the research fields of management and organizational theory. Thus, it comes as no surprise that numerous models to measure organizational performance have been proposed in the literature [for a detailed discussion on various performance models in accounting, finance, and management, see, 83]. Organizations seek to determine the performance that will improve it, and therefore rely on measurement approaches to provide them with reliable and valid information about the organization's current performance [102]. Therefore, goals of an organization serve as a key foundational element in the goal-based performance measurement approaches. Organizational goals represent planned positions or results to be achieved by an organization [84]. Thus, the degree of goal fulfillment serves as a measure of performance [69]. The greater the degree of goal achievement, the higher the performance of the organization [78]. And as Lindley and Wheeler [57] state: “An organization's goals are fundamental to success; an organization without goals (equally an individual without goals) is unlikely to compete effectively.” (p. 361). The process of goal formulation, identification and definition takes places at different organizational levels and express varying goal directions, viz. strategic, tactical, and operational [12]. The latter two are concerned with local performance and, thus, have a limited focus on an organization (i.e., work unit, departments, and regions). Strategic goals capture goals of the entire organization and represent

EMPs bring together a large number of organizational buyers and sellers to facilitate the exchange of information, products, services, and payments [7,98,116], engage in continuous trading, and have no physical or geographical boundaries [76]. In the future, the importance of EMPs will continue to grow. For example, a recent Forrester Research prediction of e-commerce sales in the U.S., including via EMPs, estimates a compound annual growth rate of 10% from 2010 to 2015, to reach U.S. $278.9 billion in 2015 [34]. Prior research has studied EMP with a particular focus on design factors that impact on the operation and establishment of these IT-centric businesses [see for a detailed discussion on the factors, 63]. A limited number of EMP research has focused on EMP performance aimed at understanding how to operate an EMP successfully. This overall lack of research into EMPs is surprising given the high failure rates of EMPs. After a large initial growth in 2000, a major shakeout took place that resulted in consolidation (mergers and acquisitions) as well as the failure of many EMPs [see, for example, 25,67,108]. As of May 2012, only 758 EMPs are listed worldwide compared to 1500 EMPs in 2000 (www. emarketservices.com). EMP failures can be attributed to a lack of clearly defined EMP goals, the pursuit of ‘unrealistic’ goals, or even that the goals pursued had less relevance for EMP success and survival [17,114]. The misalignment of the pre-defined goals with the goal

S. Matook / Decision Support Systems 54 (2013) 1065–1075

an organization's objectives in terms of its internal aspects (i.e., strength and weaknesses) and external influences. Strategic goals are translated into tactical and operational goals outlining how to design day-to-day activities in a specific work unit or department. This distinction corresponds with official and operative goal as described by Perrow [75]. Official goals express the visions and general purposes of an organization, while operative goals articulate the specific content of official goals. Hence, for this study, our focus is on strategic EMP goals because they represent the EMP comprehensively, including its purpose. It is widely accepted in management and organizational research that organizations have multiple goals and, thus, pursue a goal bundle rather than a single goal [29]. The organizational goals differ based on goal content, goal origin (that is, which level in the organization formulated the goal), and the organization's environment [93]. Clearly, goals differ based on the type of product under consideration; for example, one would expect the goals of manufacturing organizations to differ from those of service organizations. The multiplicity of goals that an organization aims to achieve at the same time means that, among the goals, relations exists that can be complementary, neutral and conflicting [1,68]. Goal complementary is the effect of the achievement of one goal positively impacting on the achievement of another goal. Goal hierarchies, in which goals are represented as means-end chains, have been used to capture the supportive interrelations among goals [for a detailed account and illustrative examples of means-end chains for organizational goals, see, for example 37]. Goal neutrality, although rarely present in an organization, can be found if two goals have no overlap in content, origin and outcome focus. Goal conflict occurs because goals may compete with one another or are incompatible as the achievement of one goal negatively impacts the achievement of the other goal [38]. Organizations address issues of goal conflict by assigning priorities to their goals and creating a rank order of their goals [89]. Although the multiplicity and interrelations of goals represent challenges in measuring performance as a function of an organization's goal, if multiple goals are ignored, performance measurement may risk imposing performance measures on a firm that are too narrow, and neither relevant nor applicable [87]. Consequently, an effective performance evaluation measure should include not only the extent to which a certain goal was actually achieved, but also how important a goal was for the organization. Goal-based performance measurement approaches are the balanced scorecard, which has an internal control focus [45] and the external goal model, both approaches originating from the field of organizational theory [13,31]. The balanced scorecard is a comprehensive management control approach that is based on the four perspectives of financial performance: customer service, internal business processes, learning, and growth [45]. The approach is focused on the internal efficiency of a firm with a focus on control to correct a firm's actions [73]. The approach addresses the fact that managers want a balanced overview of their financial and non-financial measures at any point in time [45]. The principal objective is to help managers to focus on the indicators that define the success of their organization over time, as defined by the organization's vision and strategy [23]. Yet, an internal focus on the balanced scorecard is of concern when the external performance of a firm is measured [15]. Bremser and Chung [15] state that the balanced scorecard “is for most part internally focused”, but later conclude that a performance measurement approach “should help the firm [to] look outside” (p. 399). Hence, the balanced scorecard is not, per se, appropriate for assessing external performance. Other concerns with the balanced scorecard are related to their empirical grounding and a lack of theoretical foundation for the approach [113]. Consequently, we believe that the balanced scorecard is not suitable to measure EMP performance. Research in management and organizational theory has created the external goal model as a generic approach to measure the performance of a firm [31,78]. The goal approach is an output-oriented approach that

1067

understands a firm as a goal-seeking entity [13]. The approach allows a firm to focus on external opportunities while it maintains a competitive position in the marketplace [99]. The goal approach measures performance by focusing on the attainment of the organization's goals [31], that is, the degree to which the firm's goals are met. The assumption is that an organization's performance depends on the effective pursuit of its identified goals [69]. Consequently, the greater the degree of goal achievement, the higher the performance [78]. The goal approach allows for an integrated, more holistic view of organizational performance because it considers multiple performance perspectives by including multiple goals [54]. In sum, we conclude that the external goal model can be applied to determine the performance of EMPs. The measurement can be undertaken in three steps: 1) Establishing organizational focus: Identification of the businessspecific goals, specification of management level at which the goals sit, and allocation of organizational priorities for each goal 2) Determining goal attainment: Assessment of the degree of attainment for each of the goals 3) Calculating performance: Measurement of the firm's performance using goal priority and goal achievement degree. 2.3. Identification of goals for electronic marketplaces The organizational goals of a firm are specific to the type of business [see, for example, 93,105]. Thus, goals of EMPs need to reflect the symbiotic nature of the business and the IT. Prior research on organizational performance suggested that the goals of an organization be determined either through a literature search, an empirical study, or a combination of both [69]. The goal identification process can be supplemented with corporate documents, such as company websites, mission statements [49], and corporate plans [14]. Accordingly, in this study, to identify the EMP goals, we chose a multi-step process that alternated between the literature and feedback from experts on EMPs. We commenced the process with a search for prior studies that comprehensively identified EMP goals aiming to identify goals for a particular organization in a similar way that Ambler and Kokanaki identified goals in marketing and advertisement firms [2], and At-Twaijr et al. [5] for major industries in Saudi Arabia. However, we did not find such studies, and decided to widen the literature search. In the second step, we searched books and journals published in German and English from the fields of management, strategy, organizational theory, information systems, and e-commerce. We used different search strategies to identify relevant publications (e.g., Google Scholar, the library catalog OPAC from the author's university to find books in English and German, and the “Arts & Sciences” database by JSTOR, the ACM Digital Library, and the “Periodicals Archive Online” by ProQuest). Over the course of six months, a list of more than 125 goals was created. The list included a variety of goals [for the details on the process and the list see, 115] ranging from goals of family businesses [101], social goals an organization aims to achieve [65], and goals an environmentally conscious organization pursues [80]. From the list of 125 goals, we selected goals for further consideration only if they were mentioned in at least two studies. The rationale for this was that we wanted to avoid goals which are unique to one single organization and, thus, very specific and potentially not relevant to a variety of organizations, and particularly not for EMPs. The second step resulted in a list of 41 goals [115]. In the third step, we sought feedback on the relevance and appropriateness of the 41 goals as strategic goals for EMP from four academics and three practitioners. The academics were members of an e-Commerce competence center at the first author's university. The practitioners were operators of business-to-customer EMPs and software vendors of EMP software solutions. These individuals were asked to indicate which of the 41 goals are relevant for EMPs. The outcome was a list of 16 strategic EMP goals that were selected by these academics and practitioners as

1068

S. Matook / Decision Support Systems 54 (2013) 1065–1075

most relevant for an EMP. We concluded our goal identification process with workshop meetings in which we sought further feedback and agreement on the selection of the 16 strategic EMP goals (see Table 1). 2.4. Mapping goals in the domain-specific model for EMPs While all of these 16 goals are of relevance for an EMP, they cover different perspective and strategic areas of an EMP. These differences lead to increased complexity in the managerial decision processes and, thus, it would be desirable to condense the perspective into a smaller number of dimensions. The reduction may be done via a meta-model that on the one hand enables the goals to be grouped, but on the other hand also represents the domain on an EMP. Matook and Vessey [63] developed a multi-layered model for the domain of EMPs that serves as our meta-model. Their EMP model builds on the reference model for electronic markets by Schmid and Lindemann [88]. The EMP model theorizes four views as essential for EMPs: (1) business view, (2) transaction view, (3) market service view, and (4) infrastructure view [63,88]. These views comprehensively cover the different perspectives of EMPs; given this, the 16 EMP goals can be mapped into one of the views. The mapping was done by two researchers who were guided by the definitions of the goals from Table 1 and the meaning of the views as outlined by Matook and Vessey [63]. The outcome is summarized in Table 2. During the process, each researcher carried out the mapping independently, after that the two met and compared notes. In a discussion, the mapping was explored and continued until consensus was reached. Next, we briefly describe the mapping process. Mapping goals into the first view: In the business view, the EMP's business model is captured and the business rules are internally as well as externally reflected. Factors such as pricing model, type and number of EMP participants, and ownership structure are instantiated for an EMP [63]. In this view, the EMP is concerned with its financial and accounting goals to sustain long-term business operations. Closely related to the attainment of the EMP's financial goals are goals that relate to the EMP participants. Thus, we allocated into the business view the goals: (1) profit; (2) growth; (3) sustainability; and (4) number of buyers. Mapping goals into the second view: The transaction view focuses on the technology-enabled transactions through the harmonization of the business model [63]. Hence, the EMP aims to achieve in this

Table 2 Mapping of the 16 EMP goals into the four views of the multi-layered EMP domain model [63,88]. EMP model view

Allocated goals

1. Business view

1. Profit 2. Growth 3. Sustainability 4. Number of buyers 1.Transaction volume 2. Transaction cost 3. Number of transaction 4. Transaction quality 1. Customer service 2. Awareness level 3. Customer satisfaction 4. Customer loyalty 1. Accessibility 2. User friendliness 3. Software quality 4. Infrastructure quality

2. Transaction view

3. Market service view

4. Infrastructure view

(P) (G) (S) (NoB) (TV) (TC) (NoT) (TQ) (CSe) (AL) (CSa) (CL) (A) (UF) (SQ) (IQ)

view goals that relate to EMP transactions. Important factors of EMP transactions are the volume [109], the cost [46], and the number and quality of the transaction [4,104]. These factors can be translated into goals. Thus, we allocated these four goals to the transaction view: (1) transaction volume; (2) transaction cost; (3) number of transaction; and (4) transaction quality. Mapping goals into the third view: The market service view specifies the services offered to the EMP participants including logistics and tracking services, electronic payment services, and user identification management [28,50]. These services impact on external customer satisfaction and customer loyalty [35]. Hence, EMPs set goals to achieve high levels of satisfaction, loyalty and good customer service. Additional factors in this view address ways to attract participants to the EMP [63]. Thus, the EMP defines marketing-driven goals [43] to increase its awareness to potential customers in and across an industry [58,85]. In sum, the four goals: (1) customer service; (2) customer loyalty; (3) customer satisfaction; and (4) awareness level have been allocated to the market service view. Mapping goals into the fourth view: The infrastructure view provides the information communication technologies (ICT) for a smooth

Table 1 Overview about the EMP goals (in alphabetic order). Goal

Definition

Reference

Accessibility Awareness level Customer loyalty Customer satisfaction

The availability and speed of access of the EMP The degree to which the EMP is known and has established recognition among its current and potential buyers and sellers The positive attitude of a buyer towards the EMP that leads to repeated purchasing behavior over the EMP Positive emotional reaction of an external EMPs customer to the overall EMP transaction experience with a product or service Communication processes centered on products and/or other EMP offering between of an EMP representative with an EMP buyer to meet the customer expectation The creation of value and the alignment of the expectations of current EMP stakeholders without restricting the need satisfaction of future EMP stakeholders. Percentage of change in EMP sales and/or assets The degree to which a EMP participant perceives that the EMP hardware and other technical equipment possess the desired characteristics and meets his or her composite expectations The total number of buyers registered on the EMP who actively transact over the EMP The total number of transactions an EMP participant undertakes in a period The difference between total revenue minus total cost The degree to which a EMP participant perceives that the entire transaction carried out via the EMP possess the desired characteristics and meets his or her composite expectations The degree to which a EMP participant perceives that the EMP software applications possess the desired characteristics and meets his or her composite expectations The sum of all costs involved in the transaction process, such as search cost, comparison cost, examination cost, negotiation cost, payment cost, delivery cost, post-service cost The total value amount or quantity of transactions within a timeframe The degree to which the EMP as a whole is easy to use even by novice users and transaction can be carried out free of effort.

[22] [41,110] [48,95] [64,71]

Customer service Sustainability Growth Infrastructure quality Number of buyers Number of transactions Profit Quality of transactions Software quality Transaction cost Transaction volume User friendliness

[33,81] [30,82] [11,106] [35,40] [24,109] [6] [19] [20] [9,26] [55,86] [3,18] [56,74]

S. Matook / Decision Support Systems 54 (2013) 1065–1075

operation of the EMP [63]. The infrastructure view illustrates the symbiotic nature of EMPs as IT-centric businesses and highlights that ICT underpins all aspects of an EMP. To achieve high levels of EMP accessibility and user friendliness, EMPs specify these aspects as infrastructure goals [74]. Further, to ensure continuous information and transaction support for all EMP stakeholders, high degrees of software quality and infrastructure quality are also important technology goals [35]. Consequently, we allocated the goals: (1) user friendliness; (2) accessibility; (3) infrastructure quality; and (4) software quality in this view. 3. Application of the goal approach for EMP performance measurement 3.1. Survey methodology We now present our exploratory investigation into the performance of EMPs. Our survey sought both demographic information regarding the EMPs, as well as responses to questions on the EMP goals we had theoretically identified. 3.1.1. Participants We selected EMPs for this study based on the following criteria: First, the marketplace headquarters were located in Germany. We focused on German EMPs because at the time of the study Germany had the largest concentration of EMP headquarters in Europe [53]. Second, the marketplaces supported both multiple buyers and multiple sellers. This criterion ensured that we could distinguish EMPs from either sell or buy side dominated trading platforms [51]. Third, the marketplace supported all stages of the business transaction online. In using this criterion, we could distinguish EMPs from other Internet sites such as portals or search engines [51]. We accessed a number of sources to obtain the information needed to identify a comprehensive set of EMPs for participation in our study: 1) the Berlecon Research e-Market Directory (www.berlecon.de/research/); 2) databases of Internet Providers (www.b2b-web-germany.de and www.b2b-link.de); and 3) the Marketplace Guide of the German Association of Materials Management, Purchasing and Logistics (www.b2b-marktplaetze.de/). The selection process took place from July to October 2002. Altogether, we identified 100 EMPs of which 85 EMPs agreed to participate after an initial contact. The 15 EMPs which declined expressed reasons for not participating as “having no time” (8 EMPs), “done so many surveys before and never did get anything back” (3 EMPs), and “never participate in any survey or market research” (4 EMPs). The sample included EMPs operating in a variety of industries including building and construction, chemicals, food, health care products and tools, electrical, automotive, insurance, energy, and raw materials, as well as a number of multi-industry EMPs. 3.1.2. Measures First, we determined that the EMPs pursued the 16 strategic goals we had theoretically identified. Thereafter, we asked the EMP respondents to indicate using a nominal scale (yes/no) whether the goal was pursued or not. Next, degree of priority and level of attainment for each goal were measured by asking the EMPs: 1) to rate the importance of the goal to their firm at this point in time on a 5-point scale; and 2) to rate the level of attainment for each goal in the same timeframe again on a 5-point scale. Second, goal-based performance was calculated building on multi-attribute decision making research [103] and the organizational theory literature on cooperation success [66,87]. Prior research, for example, Schaan [87], and Mellewigt and Matiaske [66] used a similar measure when they determined the performance of joint-ventures. Formula 1 presented below calculates EMP performance by first multiplying the goal priorities with the attainment values, and then normalizing the sum of the weighted attainment values using the sum of the priorities. The calculation results in a single, overall measure of the relative performance

1069

of an EMP measured on a 5-point Likert scale (1: low, 3: medium, and 5: high) [54,79]. n  X

EMP Performance ¼

i¼1



EMP Goal Priorityi EMP Goal Attainmenti n X



EMP Goal Priorityi

i¼1

i ¼ Number of EMP Goals Formula 1 : Calculation of organizational performance of EMP The use of Formula 1 is illustrated with an example as follows: It can be assumed that an EMP has three goals that are assessed on a 5-point Likert scale. The priorities of those goals are 5, 4, and 4, while attainment levels are 3, 5, and 2. The sum of the products of priority and attainment is 43, while the sum of the priorities is 13, leading to an EMP performance score of 3.31. Third, we examined the ability of our approach for evaluating performance of EMPs by using the measure of survival. Use of survival as a dependent variable has its roots in population ecology [39]. We measured survival as the ongoing presence of an EMP [83]. To determine which EMPs survived, we used an eight-year window (from the initial data collection in 2003 to 2011). We determined the current status of our EMPs by examining their websites and using information available to us, which included press releases, business reports, and white papers. 3.1.3. Data collection process We conducted a pilot study, followed by the main study. The survey was piloted as a paper-based questionnaire in November to December 2002. We requested nine EMP software vendors to evaluate the comprehensibility of our questions, and we made changes to the survey instrument to address their concerns. Data were gathered over a six-month period from January to June 2003. We targeted the CIO of each EMP as the appropriate person to respond to the wide variety of issues addressed in the questionnaire, because the CIO is a major player who would be intimately involved with all aspects of the business, as well as the IT. In an attempt to maximize the response rate, we first contacted either the CIO or, failing that, the marketing manager, by telephone from January to March 2003, using information from the EMP's websites. We then followed up with email reminders. During April and May 2003, we made two further telephone calls to encourage participants to finalize their responses. We obtained 35 responses from a total sample of 85 EMPs, of which only 20 responses were completed and as such used for this study. This corresponded to a response rate of 23.5%, which compares favorably with other online surveys [see, for example, 47,91]. 3.2. Results This section presents the results. In examining the 20 EMPs, we first focus on the types of goals that they pursued and performance on those goals. We then report on a series of analyses related to the relationship between performance and survival. 3.2.1. Electronic marketplace demographics The EMPs in our sample were founded mostly in 2000. This means that our sample includes EMPs that survived the shakeout phase that began in early 2001 [25]. The EMPs come from a variety of industries (e.g., energy, oil and fuel, food, healthcare and pharmacy, agriculture, machinery, vehicles, equipment, building and construction) and 65% of them are horizontally oriented. While all EMPs had multiple buyers and sellers, most of them are owned by independent operators. Finally, the products traded on the EMPs are equally balanced across the three types of A-, B-, and C-products. According to the managerial approach “ABC-Analysis” [32,60], A-products are defined as

1070

S. Matook / Decision Support Systems 54 (2013) 1065–1075

Table 3 EMP demographics reference [63,115]. EMP # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Table 5 Importance and performance per goal overall EMPs.

Foundation year

Ownership structure

Sources of revenue

Types of products

Industry orientation

1999 2000 2000 1994 1999 2000 2000 2001 1999 1992 2001 2000 2001 2000 1998 1999 1995 2002 2001 1995

Independent Independent Independent Independent Independent Independent Buyer Supplier Independent Independent Independent Independent Independent Independent Independent Independent Supplier Independent Supplier Independent

1 >=2 >=2 – 1 1 – 1 1 >=2 1 1 >=2 1 1 – – 1 1 1

C A, B C C A, B, C B B, C C C A, C A, B, C C C A, B, C A, B, C A A, B A, B, C A A

Horizontal Vertical Horizontal Vertical Horizontal Vertical Horizontal Horizontal Horizontal Horizontal Horizontal Vertical Horizontal Horizontal Horizontal Vertical Horizontal Horizontal Vertical Vertical

expensive, highly complex and strategically-important products which are purchased infrequently. In contrast, C-products also known as maintenance, repair, and operation (MRO) goods are inexpensive, less complex products that are purchased frequently. B-products are in the middle between A- and C-products which means that they are neither expensive nor complex but also not cheap and are purchased in moderate frequency. Table 3 provides a summary of the EMP's demographic. 3.2.2. Electronic marketplace goals: Priorities and performance The EMPs pursued all of the 16 strategic goals, albeit to varying extents as Table 4 illustrates. Hence, the frequency with which a goal was rated as high priority (recorded as 4 or 5 on the 5-point Likert scale) reflected the importance of the goals to the EMPs. We included in our analysis only responses to those goals that EMPs judged to be of high priority, because our initial analysis of performance evaluation in the organizational literature emphasized the importance of focusing on the fulfillment of an organization's most important goals. As Table 4 shows, the number of high priority goals pursued by the EMPs ranged

EMP goals Awareness level Growth Sustainability Profit Number of buyers Customer satisfaction Accessibility Transaction volume Transaction cost Customer loyalty Customer service Number of transactions Software quality Transaction quality User friendliness Infrastructure quality Overall mean per goal

# EMPs

Goal importance

Performance

(AL) (G) (S) (P) (NoB) (CSa) (A) (TV) (TC) (CL) (CSe) (NoT)

16 15 14 14 14 14 11 10 10 9 8 7

Six most important goals

3.30 4.02 3.77 3.09 2.73 3.14 4.20 3.65 3.64 2.76 4.17 3.75

(SQ) (TQ) (UF) (IQ)

7 6 5 3 10.19 (51.0%)

Six least important goals

3.87 3.19 4.00 3.62 3.56

from 13 goals (out of a total of 16 goals) to only 3 goals, with EMPs being relatively evenly distributed within that range. Table 4 also presents the performance for each EMP differentiated by model views. The performance means were 3.22 for business goals, 3.58 for transaction goals, 3.18 for market service goals, and 4.20 for infrastructure goals. Table 5 shows the number of EMPs that viewed each of the goals as high priority and the performance of all EMPs on each of the 16 goals. Overall, EMPs regarded a total of 163 goals out of potentially 320 goals (16 goals * 20 EMPs) as being of high priority (i.e., 51.0% of all goals). The most important goals are (see Table 5): the level of awareness of the EMP (16 EMPs); growth (15 EMPs); sustainability (14 EMPs); profit (14 EMPs); number of buyers (14 EMPs); and customer satisfaction (14 EMPs), while the least important goals are: infrastructure quality (3 EMPs); user friendliness (5 EMPs); quality of transactions (6EMPs); software quality (7 EMPs); and number of transactions (7 EMPs). Table 5 also lists the performance of all EMPs for each goal, which indicates how well the EMPs on average achieved a particular high-priority goal. We found that the EMPs achieved the highest performance for the goals of accessibility (4.20), followed by the goal customer service (4.17), and the goal growth (4.02). When the goals are grouped in the

Table 4 Overview about high priority EMP goals and performance per view. EMP

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Sum/Means

Business goals

Transaction goals

P

G

S

NoB

Performance

X X X

X X X X X X

X X X X X X

X

3.25 3.62 3.67 3.00 3.57 3.53 1.00 3.18 2.00 3.79 3.62 2.62 3.67 4.36 4.00 4.50 1.78 3.78 3.00 2.50 3.22

X X X X X X X X X X X 14

X

X

X X X X X

X X X

X x X X X 15

X X 14

X X X X X X X X X X X X X 14

TV

TC

NoT

X X

X

X X

X X X

X X X

Performance

X

4.00 2.00 5.00 – 2.57 4.00 3.75 – – 3.00 5.00 1.67 – 4.00 4.00 3.00 – 3.77 4.00 4.00 3.58

X X

X X X

X X X

X

X

X

X

X

X

X X X

10

X X 10

X X 7

Market service goals

TQ

6

CSe

X X X

X

X X X X

8

AL

CSa

X X

X X

X X X X X X X X X X X

X X X X

X X X 16

CL

X X

X X X X X X X X 14

X X X X X X X 9

Infrastructure goals Performance

A

2.44 3.50 – – – 3.75 3.00 3.78 3.00 3.00 2.92 4.00 3.33 4.00 3.50 3.95 1.56 4.08 1.89 2.44 3.18

X X X

X X X X

UF

X X

SQ

X X X

IQ

X

X X X X X

X X X

X 11

5

X X

X

X 7

3

Sum of goal Performance 5.00 4.00 4.00 – – 3.71 4.00 3.53 5.00 4.00 5.00 – 5.00 4.00 3.62 4.00 – – – 4.00 4.20

8 8 7 3 6 13 10 12 4 8 10 7 7 13 11 9 4 10 5 8 163

S. Matook / Decision Support Systems 54 (2013) 1065–1075

four views of the EMP model, we found that 57 of the 163 high-priority goals (35%) were in the business view, 33 goals (20.2%) in the transaction view, 47 goal (28.8%) in the market service view, and 26 goals (16%) in the infrastructure view. 3.2.3. Electronic marketplace success Using the goal priorities (only high priority goals) and goal attainment degrees, we calculated the performance of our 20 EMPs (see Table 6). Then, we determined the current status of our EMPs via information available to us by examining the EMP's websites. Subsequently, we divided the EMPs into two groups: those that were still operating as EMPs and those that were not. We understand the EMPs that are in operation as being successful and those that failed as being unsuccessful. Note that the successful EMPs made only small adjustments to their business model in the intervening eight years, presumably to manage EMP evolution; for example, EMP 18 included further logistic partners and adjusted the amount buyers had to pay for participation. The group of unsuccessful EMPs either did not survive their first few years as EMPs, or went through revolutionary change [see, also, 25]. Such changes include, for example, converting from a trading to a consulting company focused on e-commerce strategies (EMP 9) and from an EMP to an information portal (EMP 20), as well as going out of business entirely (e.g., EMPs 1, 10, 2, and 19). Despite the fact that Day et al. [25] identified a group of EMPs that had “ceased independent operation due to merger and acquisition,” (p.148) we identified only one EMP (EMP 3) that underwent a later merger. As Table 6 shows, we could determine unequivocally what happened to all but 3 of our 20 EMPs (see “no active link”); however, checking the other sources of information suggested that these EMPs were no longer operational. Table 6 shows that just 9 of our 20 EMPs (45%) were still in business after eight years and whereas 11 EMPs were no longer operational. Because we were able to consistently classify 18 out of 20 EMPs (90% accuracy) as successful and unsuccessful, we suggest that goals may be an effective way of explaining longer-term success of EMPs. Note, however, that there is no way of determining what the cut-off for success will be in a given instance because of the difficulty in anchoring perceptual measures. For example, in this study, the cut-off point is at 3.5. Thus, EMP performance needed to be “above average” during this timeframe for an EMP to have a chance of surviving. Having established that performance of high priority EMP goals can be used to determine the success of EMPs, we now seek to find out what drives the differences between the groups of operational

Table 6 EMP performance in 2003 and EMP status eight years later. Operational — successful EMPs

Non-operational — unsuccessful EMPs

EMP #

Performance EMP#

Performance Reason

3

4.29

13

3.71

14 11 16 18

4.08 4.02 4.00 3.86

10 1 2 9

3.48 3.38 3.17 3.16

15

3.72

5

3.07

6

3.71

4

3.00

8 7

3.50 3.25

20 19 12

2.83 2.78 2.47

Sum Mean 9 EMPs 3.83

17 1.67 Sum Mean 11 EMPs 2.92

Now develops software for EMPs No active link Now an auction platform No active link Now an e-commerce consulting firm Now a procurement consulting firm Now a service provider for digital sales management Now only an information portal No active link Closure announcement on web page Now only an information portal

1071

Table 7 Performance and importance of goals and types of goals based on EMP success. Operational — successful EMPs

Non-operational — unsuccessful EMPs

Performance Percent of EMPs (#)

Performance Percent of EMPs (#)

Business goals Sustainability Growth Profit Number of buyers Means

4.26 4.00 3.48 3.39 3.78

77.8 66.7 77.8 66.7 81.3

(7) (6) (7) (6) (6.50)

3.29 3.89 2.78 2.27 3.06

63.6 (7) 72.7 (8) 45.5 (5) 63.6 (7) 61.4 (6.75)

Transaction goals Transaction volume Transaction cost Number of transaction Transaction quality Means

4.63 4.26 4.32 3.56 4.19

55.6 55.6 44.4 22.2 50.0

(5) (5) (4) (2) (4.0)

2.50 2.56 2.44 3.00 2.63

36.4 (4) 36.4 (4) 18.2 (2) 27.3 (3) 34.1 (3.25)

Market service goals Customer service Awareness level Customer satisfaction Customer loyalty Means

4.37 4.00 3.33 3.00 3.68

77.8 77.8 77.8 55.6 81.3

(7) (7) (7) (5) (6.50)

3.16 2.36 1.64 2.39

63.6 (7) 45.5 (5) 27.3 (3) 29.5 (3.75)

Infrastructure goals Accessibility User friendliness Software quality Infrastructure quality Means Overall means

4.04 4.00 4.00 3.00 3.76 3.85

66.7 55.6 44.4 22.2 53.1 66.4

(6) (5) (4) (2) (4.25) (5.25)

4.53 – 4.00 – 4.27 3.08

36.4 (4) – 18.2 (2) – 13.6 (1.5) 34.7 (3.8)

EMP goals

and non-operational EMPs. Although our results are similar whether we include the EMPs that appear to be misclassified or not, we dropped them from each of the operational and non-operational groups (EMPs 13 and 7). As such, we examined eight EMPs in the operational group and ten EMPs in the non-operational group. Table 7 presents the analyses in terms of the four views. Overall, 66.4% of the operational EMPs regarded the 16 goals as being of high priority; just 34.7% of the non-operational EMPs regarded these goals as being of high priority. This pattern is repeated in each of the views. We then sought to gain insights into the validity of our conceptualization of EMPs as IT-centric businesses by comparing how various sub-sets of goals predict EMP survival. We examined whether business goals alone as a reflection of traditional, accounting-related approaches [44], or the types of goals specific to EMPs, that is, the transaction and infrastructure goals (as a group), predict survival better than our composite of all four types of goals. Table 8 presents both our original findings for survival based on all four types of goals as well as the performance on the sub-sets of goals outlined above. Perusal of Table 8 shows that analyzing survival using performance on sub-sets of the goals destroys the cohesive pattern that results from using the four types of goals specific to electronic markets. The four types of goals combined best predict survival, followed by business goals, and, lastly followed by transaction and infrastructure goals. Performance on transaction and infrastructure goals, in particular, bears little relationship to survival, showing the futility of using goals related to the IT artifact [72] alone to measure the performance of EMPs. 4. Discussion In this study, we used the external goal approach to measure EMP performance. To evaluate EMP performance, we defined a set of 16 strategic goals and mapped them into the multi-layered, domainspecific model for EMPs [63,115]. Our evaluation of EMP performance sought to characterize both the goals that EMPs seek and the EMPs

1072

S. Matook / Decision Support Systems 54 (2013) 1065–1075

Table 8 Relationship between EMP performance and survival on different groups of goals. Overall performance

Business

Transaction and infrastructure

EMP

All

Outcome

EMP

Business Outcome

EMP

Trans. Outcome & IFS

3 14 11 16 18 15 6 13 8 10 1 7 2 9 5 4 20 19 12 17 Mean

4.29 4.08 4.02 4.00 3.86 3.72 3.71 3.71 3.50 3.48 3.38 3.25 3.17 3.16 3.07 3.00 2.83 2.78 2.47 1.67 3.36

Operat. Operat. Operat. Operat. Operat. Operat. Operat. Non-oper. Operat. Non-oper. Non-oper. Operat. Non-oper. Non-oper. Non-oper. Non-oper. Non-oper. Non-oper. Non-oper. Non-oper.

16 14 15 10 18 13 3 2 11 5 6 1 8 20 4 12 20 9 17 7 Mean

4.50 4.36 4.00 3.79 3.78 3.67 3.67 3.62 3.62 3.57 3.53 3.25 3.18 3.00 3.00 2.62 2.50 2.00 1.78 1.00 3.22

13 11 9 3 1 14 19 20 7 6 18 15 16 8 10 2 5 12 64 17 Mean

5.00 5.00 5.00 4.79 4.50 4.00 4.00 4.00 3.84 3.82 3.77 3.72 3.71 3.53 3.24 2.57 2.57 1.67 – – 3.82

Operat. Operat. Operat. Non-oper. Operat. Non-oper. Operat. Non-oper. Operat. Non-oper. Operat. Non-oper. Operat. Non-oper. Non-oper. Non-oper. Non-oper. Non-oper. Non-oper. Operat.

Non-oper. Operat. Non-oper. Operat. Non-oper. Operat. Non-oper. Non-oper. Operat. Operat. Operat. Operat. Operat. Operat. Non-oper. Non-oper. Non-oper. Non-oper. Non-oper. Non-oper.

themselves. We also examined the ability of our performance measure to predict survival by using data collected eight years after the original data collection. 4.1. Goals and performance of electronic marketplaces Our study findings provide information on appropriateness, importance, and attainment of strategic goals for 20 EMPs. We found that, in general, these 20 EMPs did not particularly well define and prioritize their goals. Our results suggest that business goals were the best defined and most highly prioritized goals, followed in sequence by market service goals, transaction goals, and infrastructure goals. Hence, EMPs deemed goals in the business and market service views to be more important than goals in the transaction and infrastructure views (see Table 4). Interestingly, performance on the goals in each view was inversely proportional to their perceived importance, that is, performance on transaction and infrastructure goals was perceived to be better than on the business and market service goals. This result suggests that the 20 EMPs performed well on technology-related aspects, which ensure that the EMP is available to its customers, while most attention was paid to the business aspects. There, EMP operators felt they have not achieved everything they aimed for and that potential for improvements in performance at the business view and market service view exists. Based on the findings from a series of analyses, we believe that EMPs state goals based on a combination of two factors: 1) whether they consider a goal as being crucial to their operations; and 2) they are uncertain about whether their performance on a particular goal is sufficiently good. We also examined if our performance figures could be used to draw conclusions about the success of EMPs over time. On revisiting our EMPs eight years after the initial evaluation, we found that with the external goal approach, we correctly classified 90% of the EMPs as successful–unsuccessful. The results support the notion that IT-centric businesses depend on both business and IT for their survival. They also support our choice of the domain-specific EMP model to reduce the dimension of EMP goals. Differences among the successful and unsuccessful EMPs emerged in regards to their priorities in terms of strategic goals and, hence, indicated how important certain goals were for the EMPs. While successful EMPs regarded 66.4% of goals as being of high priority, unsuccessful EMPs regarded just 34.7% of

goals as being of high priority (see Table 7). This pattern is repeated in each of the views of the reference model. The greatest difference between the high priority goals of the successful and unsuccessful EMPs was on the market service goals. While successful EMPs paid significant attention to them, unsuccessful EMPs paid relatively little attention. As an example (see Table 7), unsuccessful EMPs prioritized how to market their EMPs lower (29.5%) compared to successful EMPs (81.3%). Perhaps rather strangely given the current emphasis on customer service in the business world, no unsuccessful EMP regarded customer service as being of high priority, compared with 77.8% of successful EMPs (with the second highest performance figure of 4.37). Hence, lack of customer service appears to be a major oversight on the part of the unsuccessful EMPs in our sample, and may well have played a significant role in the failure of those EMPs. Similarly, our findings suggest that EMPs need to pay attention to the transactions carried out on their platform. We found in our study that there was also a large difference in performance of the two groups on the transaction volume goal. Successful EMPs achieved a much higher performance figure of the transaction volume goal (4.63), than the unsuccessful EMP (2.50) (see Table 7). An EMP with low transaction volume would find it difficult to survive long-term. In sum, our findings for the 20 EMPs suggest that goals need to be defined for different strategic area, and that the EMPs should assign high priorities in their various strategic goals. Beyond the specific findings for the 20 EMPs, we can derive more general findings. Our results provide support for using attainment of high priority goals as the foundation for evaluating performance. Indeed, this research highlights the utility and value of strategic goals to determine an organizations' performance. Our findings on successful and unsuccessful EMPs suggest that these strategic goals need to be carefully specified as they indicate the strategic direction of an EMP [57]. Furthermore, we were able to show that EMPs as IT-centric businesses should not prefer one group of goals over another; rather a balanced focus on a diverse set of strategic goals is recommended. Without this goal diversity, EMPs might not comprehensively capture all critical facets of the organization. Specifically, an organization which operates in virtual space faces more complex management challenges through the need for stronger IT dependency than does a traditional brick-and-mortar company. Fig. 1 summarizes the building blocks of our EMP performance measurement. The multi-layered EMP model suggests four views in which an EMP should map its strategic goals. Once the mapping is completed, priorities can be assigned to express the importance of the goal for the EMP. After that, the degree of goal attainment is determined as a means to measure the ongoing and long-term performance of the EMP. Although externally focused, the measurement can be seen as a closed system in which the EMPs first identify targets that are important for them, define strategies to achieve these targets, and then utilize attainment levels to determine the effectiveness of the chosen strategy. This approach would allow the strategic choices made by EMPs to be evaluated (see [62] for details on such a method). 4.2. Contributions to theory and implications for practice This research makes several theoretical contributions. First, it contributes to the literature on EMPs and e-commerce. We employed an easy-to-use model for performance measurement and success-failure analysis that enabled us to predict which EMP would be successful and which would not be. Thus, this study expands our understanding on organizational issues of EMPs by highlighting the importance of strategic goals as performance targets of EMP. This study demonstrates theoretically the impact of organizational goal on EMP success. Second, we contribute to the organizational performance literature. The external goal-based performance measurement approach is the most traditional way to measure performance, and reminds a firm of the importance of clearly defined and pursued goals. With this

S. Matook / Decision Support Systems 54 (2013) 1065–1075

1073

Summary of the goal-based performance measurement approach EMP Goals in Four Views

Business View

Profit

Transaction View

Transaction Volume

Market Service View

Customer Loyalty

Infrastructure View

User Friendliness

Goal Priority

Sustainability

Number of Buyers

Transaction Cost

Number of Transaction

Transaction Quality

Customer Service

Awareness Level

Growth

Software Quality

Accessibility

Customer Satisfaction

Infrastructure Quality

}

High Goal Priorities (i.e., importance rating of 4 and 5) for EMP goals in all four views

Goal Attainment

EMP Success

on lin e

EMP Model Views

=> High Degrees of Goal Attainment for EMP goals in all four views

Fig. 1. Summary of the goal-based performance measurement approach.

study, we provide support for the use of goals to measure performance when tailored to the application domain under study. Third, this work adds to the body of knowledge on organizational goals. Previous studies identified organizational goals for a variety of firms (e.g., family-owned, green sustainable focused, profit and non-profit firms), but prior research is scant on goals of IT-centric businesses and the value goals have for measuring performance. This study extends the relevant body of knowledge by presenting goals that are relevant for EMPs. This research also has a number of implications for practice. First, our study emphasizes the importance of a firm's organizational goals. Decision makers can use the goal-related aspects of our performance measurement approach for the definition, clarification, and prioritization of their goals. We suggest that EMP decision makers make use of the 16 strategic goals identified in this study to compare their current bundle of goals with the goals of the study's EMPs—either to identify why they have missed goals, or to identify which additional goals they could pursue. Thus, our work can assist EMPs in their efforts to reflect on the goals in regard to importance and appropriateness. Indeed, clearly defined goals are fundamental to the success of an organization and, in particular, for the 20 EMPs in this study. Mapping goals into the four perspectives of the multi-layered EMP model allows EMP operators to classify their goals, and achieve a balance among the different strategic goals. For example, EMPs that pursue primarily business goals and infrastructure goals should recognize the importance of having high priority goals in the transaction and market service view. Second, with this study, we demonstrated how a performance measurement method can be used as a strategic instrument that helps organizations to understand their current position in the marketplace, define their desired future position, and determine necessary actions to achieve that position. Hence, the organizational goals represent final end states to be achieved, and a firm's strategies can be understood as the pathway to these end states. In addition, the performance measurement approach allows EMPs to decide on the effectiveness of their strategies. Thus, the EMPs can manage performance and identify ways to improve performance by repeatedly measuring goal achievement. 4.3. Limitations and future research Although we believe that we have carried out this study following good academic practice, we need to acknowledge some limitations. First, we conducted our analyses with a small sample of 20 EMPs. And while we believe this number is sufficiently large, we acknowledge the limited generalizability of our results due to the abundance of statistical tests for significance. Note, however, that our sample size was sufficient to demonstrate that our measure of performance differentiated between EMPs that survived the following eight years and those that

did not. Indeed, our sample size is comparable to prior research studies on EMP success; for example Soh et al. [94] based their data and findings on only 19 EMPs. Consequently, with this sample, we cannot conclude what the ‘right or wrong’ EMP goals are, what strategies the EMPs should have followed to realize high degrees of goal attainment, nor why the EMPs failed. Yet, we can conclude that those EMPs which failed made decisions in regards to their strategic goals that led to low EMP performance. Furthermore, our data shows that low performance is related to the abatement of a firm's operation and that the strategic goals can be used to continuously monitor performance. Second, our study focused on German-based EMPs. At the time of our data collection (2003), Germany was the European nation most actively pursuing the development of EMPs [53]. We have, however, no reason to suspect bias in the goals of German EMPs in relation to the goals of EMPs in any other nation. Third, our analysis suffers from the fact that there is no way of determining what the cut-off for success will be in a given instance because of the difficulty in anchoring perceptual measures. Finally, the data we used in this study are subjective in nature. Although Dess and Robinson [27] strongly advocated that subjective data are not “worse” than objective data, our choice of data restricts us from comparing the subjective figures with any objective data. For example, we cannot compare accounting measures (e.g., profit or growth figures) with the goal achievement degrees for the goals profit and growth. However, these aspects open avenues for future research. From the perspective of future research, we call for other researchers to conduct more in-depth studies. We have shown that organizational goals, represented in four broad perspectives of an EMP, can be used to measure EMP performance. We have outlined three steps that EMP can undertake when measuring their performance. A more detailed account of the development of a method for performance measurement that builds on our work would be a valuable future research topic. Such a method development study should consider the issues of the degree of flexibility and adaptability of a highly generic instrument that can be tailored to different measurement foci. Issues such as the interrelations of goals and different management levels should then be considered in such a method development study. We also suggest that future work should use larger data sets, collect both subjective and objective data, and include further dependent variables besides performance-survival. Future research may also like to develop a taxonomy to characterize the various types of IT-centric businesses (e.g., social media [42] or online collaboration platforms [59]). In doing so, future research could then determine appropriate application domain models, which would facilitate the use of our approach for a broad range of IT-centric businesses. In the same light, organizations in other domains may possess different goals and, hence, future research may determine other goals or distinguish the goals at a more fine-grained level. A reflection on the goals would also allow for expansion of the goal meanings and contents to consider their interrelations (i.e., conflicting, neutral, and

1074

S. Matook / Decision Support Systems 54 (2013) 1065–1075

complementary), their focus at a particular management level (i.e., strategic, tactical, and operational), and broaden goals such as customer satisfaction to consider EMP staff satisfaction and customer-staff satisfaction. 5. Conclusion With the advent of the Internet, new businesses, as well as new ways of doing business, have emerged — most notably electronic marketplaces (EMPs). Their symbiotic relationship between business and IT has created challenges for operators and participants alike, including how to measure the performance of this new business form. In this study, we have drawn on the external goal approach from organizational theory literature as a foundation to determine success or failure of EMPs. Because managerial goals are the core of this approach, we identified 16 strategic goals of relevance for an EMP, and mapped these goals into the domain model for EMPs. Our empirical work reveals that goals from all perspectives are important for an EMP to be successful. Thus, this study encourages EMP operators to carefully define their strategic goals and monitor the achievement of them to allow for adjustments to ensure long-term success. Acknowledgements The author would like to thank Werner Esswein, Viswanath Venkatesh, Iris Vessey, and research workshop participants at the Australian National University, University of Queensland, and the Technische Universität Dresden for their valuable inputs and helpful comments on earlier versions of the manuscript. References [1] R.S. Achrol, M.J. Etzel, The structure of reseller goals and performance in marketing channels, Journal of the Academy of Marketing Science 31 (2003) 146–163. [2] T. Ambler, F. Kokkinaki, Measures of marketing success, Journal of Marketing Management 13 (1997) 665–678. [3] R. Amit, C. Zott, Value creation in E-business, Strategic Management Journal 22 (2001) 493–520. [4] U. Arnold, New dimensions of outsourcing: a combination of transaction cost economics and the core competencies concept, European Journal of Purchasing & Supply Management 6 (2000) 23–29. [5] M.I. At-Twaijri, S.M. Al-Ghamdi, M. Luqmani, Prioritization of corporate goals in Saudi Arabia: an exploratory investigation, International Journal of Value-Based Management 9 (1996) 259–270. [6] J.P. Bailey, Y. Bakos, An exploratory study of the emerging role of electronic intermediaries, International Journal of Electronic Commerce 1 (1997) 7–20. [7] Y. Bakos, The emerging role of electronic marketplaces on the Internet, Communications of the ACM 41 (1998) 35–42. [8] J. Barney, Firm resources and sustained competitive advantage, Journal of Management 17 (1991) 99–120. [9] G. Bazzana, O. Andersen, T. Jokela, ISO 9126 and ISO 9000: friends or foes? in: Proceedings of the Software Engineering Standards Symposium, Brighton, UK, 1993. [10] R.D. Behn, Why measure performance? Different purposes require different measures, Public Administration Review 63 (2003) 586–606. [11] C.H. Berry, Corporate growth and diversification, Journal of Law and Economics 14 (1971) 371–383. [12] K. Bleicher, Normatives Management: Politik, Verfassung und Philosophie des Unternehmens, Campus, Frankfurt aM, 1994. [13] A. Bluedorn, Cutting the Gordian knot: a critique of the effectiveness tradition in organization research, Sociology and Social Research 64 (1980) 477–496. [14] R. Bozec, Boards of directors, market discipline and firm performance, Journal of Business Finance & Accounting 32 (2005) 1921–1960. [15] W.G. Bremser, Q. Chung, A framework for performance measurement in the e-business environment, Electronic Commerce Research and Applications 4 (2005) 395–412. [16] P. Brunn, M. Jensen, J. Skovgaard, e-Marketplaces: crafting a winning strategy, European Management Journal 20 (2002) 286–298. [17] G. Büyüközkan, A success index to evaluate e-marketplaces, Production Planning & Control 15 (2004) 761–774. [18] S.T. Chou, Migrating to the web: a web financial information system server, Decision Support Systems 23 (1998) 29–40. [19] D.H. Ciscel, T.M. Carroll, The determinants of executive salaries: an econometric survey, The Review of Economics and Statistics 62 (1980) 7–13. [20] J.E. Collier, C.C. Bienstock, Measuring service quality in e-retailing, Journal of Service Research 8 (2006) 260–275.

[21] T.R. Coltman, T.M. Devinney, D.F. Midgley, e-Business strategy and firm performance: a latent class assessment of the drivers and impediments to success, Journal of Information Technology 22 (2006) 87–101. [22] J. Cox, B.G. Dale, Service quality and e-commerce: an exploratory analysis, Managing Service Quality 11 (2001) 121–131. [23] R.L. Daft, Organization Theory and Design, South-Western Pub., Cinncinati, OH, 2009. [24] Q. Dai, R.J. Kauffman, Partnering for perfection: an economics perspective on B2B electronic market strategic alliances, in: Advances in Information Systems and Economics, Idea Group Publishing, Harrisburg, PA, 2005, pp. 43–79. [25] G.S. Day, A.J. Fein, G. Ruppersberger, Shakeouts in digital markets: lessons from B2B exchanges, IEEE Engineering Management Review 31 (2003) 24–35. [26] W.H. DeLone, E.R. McLean, Measuring e-commerce success: applying the DeLone & McLean information systems success model, International Journal of Electronic Commerce 9 (2004) 31–47. [27] G.G. Dess, R.B. Robinson, Measuring organizational performance in the absence of objective measures: the case of the privately held firm and conglomerate business unit, Strategic Management Journal 5 (1984) 265–273. [28] W. Dou, D.C. Chou, A structural analysis of business-to-business digital markets* 1, Industrial Marketing Management 31 (2002) 165–176. [29] P.F. Drucker, The Practice of Management, Harper & Brothers, New York, NJ, 1954. [30] T. Dyllick, K. Hockerts, Beyond the business case for corporate sustainability, Business Strategy and the Environment 11 (2002) 130–141. [31] A. Etzioni, Modern Organization, Prentice-Hall, Englewood Cliffs, NJ, 1964. [32] B.E. Flores, D.C. Whybark, Multiple criteria ABC analysis, International Journal of Operations & Production Management 6 (1986) 38–46. [33] W.Z. Ford, Customer expectations for interactions with service providers: relationship versus encounter orientation and personalized service communication, Journal of Applied Communication Research 29 (2001) 1–29. [34] Forrester, US Online Retail Forecast 2010 To 2015: eCommerce Growth Accelerates Following “The Great Recession”, 2011. [35] W. Fritz, Market orientation and corporate success: findings from Germany, European Journal of Marketing 30 (1996) 59–74. [36] D.E. Gengatharen, C. Standing, A framework to assess the factors affecting success or failure of the implementation of government-supported regional e-marketplaces for SMEs, European Journal of Information Systems 14 (2005) 417–433. [37] J. Gutman, Means-end chains as goal hierarchies, Psychology and Marketing 14 (1997) 545–560. [38] R.H. Hall, Organizations: Structures, Processes, and Outcomes, Prentice-Hall, Upper Saddle River, NJ, 2002. [39] M.T. Hannan, J. Freeman, The population ecology of organizations, The American Journal of Sociology 82 (1977) 929–964. [40] R.A. Hirschheim, H.K. Klein, K. Lyytinen, Information Systems Development and Data Modeling: Conceptual and Philosophical Foundations, Cambridge Univ. Press, 1995. [41] W.D. Hoyer, S.P. Brown, Effects of brand awareness on choice for a common, repeat-purchase product, Journal of Consumer Research 17 (1990) 141–148. [42] I. Jahnke, M. Koch, Web 2.0 Goes Academia: Does Web 2.0 Make a Difference? International Journal of Web Based Communities 5 (2009) 484–500. [43] K. Kalyanam, S. McIntyre, The e-marketing mix: a contribution of the e-tailing wars, Journal of the Academy of Marketing Science 30 (2002) 487. [44] R.S. Kaplan, Management accounting for advanced technological environments, Science 245 (1989) 819. [45] R.S. Kaplan, D.P. Norton, The balanced scorecard–measures that drive performance, Harvard Business Review 70 (1992) 71–79. [46] S. Kaplan, M. Sawhney, E-hubs: the new B2B marketplaces, Harvard Business Review 78 (2000) 97–106. [47] M.D. Kaplowitz, T.D. Hadlock, R. Levine, A comparison of web and mail survey response rates, Public Opinion Quarterly 68 (2004) 94. [48] N.M. Kassim, N.A. Abdullah, Customer loyalty in e-commerce settings: an empirical study, Electronic Markets 18 (2008) 275–290. [49] S. Kemp, L. Dwyer, Mission statements of international airlines: a content analysis, Tourism Management 24 (2003) 635–653. [50] M. Koch, K.M. Moeslein, Identities management for e-commerce and collaboration applications, International Journal of Electronic Commerce 9 (2005) 11–29. [51] L.W. Lam, L.J. Harrison-Walker, Toward an objective-based typology of e-business models, Business Horizons 46 (2003) 17–26. [52] T.M. Laseter, S.E. Bodily, Strategic indicators of B2B e-marketplace financial performance, Electronic Markets 14 (2004) 322–332. [53] M. Lenz, H.D. Zimmermann, M. Heitmann, Strategic partnerships and competitiveness of business-to-business e-marketplaces: preliminary evidence from Europe, Electronic Markets 12 (2001) 100–111. [54] A.Y. Lewin, J.W. Minton, Determining organizational effectiveness: another look, and an agenda for research, Management Science 32 (1986) 514–538. [55] T.P. Liang, J.S. Huang, An empirical study on consumer acceptance of products in electronic markets: a transaction cost model, Decision Support Systems 24 (1998) 29–43. [56] Z. Liao, M.T. Cheung, Internet-based e-banking and consumer attitudes: an empirical study, Information Management 39 (2002) 283–295. [57] E. Lindley, F.P. Wheeler, The learning square: four domains that impact on strategy, British Journal of Management 11 (2000) 357–364. [58] M. Madlberger, Applications of Internet-based marketing instruments, in: S. Krishnamurthy (Ed.), Contemporary Research in E-Marketing, Idea Group Inc., Hershey, PA, 2005.

S. Matook / Decision Support Systems 54 (2013) 1065–1075 [59] M. Madlberger, What drives firms to engage in interorganizational information sharing in supply chain management? International Journal of e-Collaboration 5 (2009) 18–42. [60] T.W. Malone, J. Yates, R.I. Benjamin, Electronic markets and electronic hierarchies, Communications of the ACM 30 (1987) 484–497. [61] B. Marr, A. Neely, Organisational performance measurement in the emerging digital age, International Journal of Business Performance Management 3 (2001) 191–215. [62] S. Matook, A method for evaluating performance of IT-centric businesses applied to the domain of electronic marketplaces, Journal of Computer Information Systems (in press). [63] S. Matook, I. Vessey, Types of business-to-business e-marketplaces: the role of a theory-based, domain-specific model, Journal of Electronic Commerce Research 9 (2008) 260–279. [64] V. McKinney, K. Yoon, M.Z. Fatemeh, The measurement of web-customer satisfaction: an expectation and disconfirmation approach, Information Systems Research 13 (2002) 296–315. [65] A. McWilliams, D. Siegel, Corporate social responsibility: a theory of the firm perspective, Academy of Management Review (2001) 117–127. [66] T. Mellewigt, W. Matiaske, Zur Messung des Unternehmens- und Kooperationserfolges, Die Betriebswirtschaft 60 (2000) 125–128. [67] J. Miller, Technology: lessons from the e-marketplace shakeout, Pharmaceutical Technology 25 (2001) 52–54. [68] H. Mintzberg, Power In and Around Organizations, Prentice Hall, New York, 1983. [69] L.B. Mohr, The concept of organizational goal, The American Political Science Review 67 (1973) 470–481. [70] P. O'Reilly, P. Finnegan, Intermediaries in inter-organisational networks: building a theory of electronic marketplace performance, European Journal of Information Systems 19 (2010) 462–480. [71] R.L. Oliver, R.T. Rust, S. Varki, Customer delight: foundations, findings, and managerial insight, Journal of Retailing 73 (1997) 311–336. [72] W.J. Orlikowski, C.S. Iacono, Research commentary: desperately seeking the “IT” in IT research, Information Systems Research 12 (2001) 121–134. [73] D. Otley, Management control and performance management: whence and whither? The British Accounting Review 35 (2003) 309–326. [74] J.W. Palmer, Web site usability, design, and performance metrics, Information Systems Research 13 (2003) 151–167. [75] C. Perrow, The analysis of goals in complex organizations, American Sociological Review 26 (1961) 854–866. [76] S. Poon, P. Swatman, An exploratory study of small business Internet commerce issues, Information Management 35 (1999) 9–18. [77] M.E. Porter, Competitive Advantage — Creating and Sustaining Superior Performance, Free Press, London, 1985. [78] J.L. Price, The study of organizational effectiveness, The Sociological Quarterly 13 (1972) 3–15. [79] R.E. Quinn, J. Rohrbaugh, A competing values approach to organizational effectiveness, Public Productivity Review 5 (1981) 122–140. [80] P. Ransom, D.J. Lober, Why do firms set environmental performance goals?: some evidence from organizational theory, Business Strategy and the Environment 8 (1999) 1–13. [81] G. Ray, W.A. Muhanna, J.B. Barney, Information technology and the performance of the customer service process: a resource-based analysis, MIS Quarterly 29 (2005) 625–652. [82] F. Reinhardt, Sustainability and the firm, Interfaces 30 (2000) 26–41. [83] P.J. Richard, T.M. Devinney, G.S. Yip, G. Johnson, Measuring organizational performance: towards methodological best practice, Journal of Management 35 (2009) 718–804. [84] M.D.V. Richards, Organizational Goal Structures, West Publishing, St. Paul, MI, 1978. [85] K. Roth, D.A. Ricks, Goal configuration in a global industry context, Strategic Management Journal 15 (1994) 103–120. [86] M. Sarkar, B. Butler, C. Steinfield, Cybermediaries in electronic marketspace: toward theory building, Journal of Business Research 41 (1998) 215–221. [87] J.L. Schaan, International joint venture success measurement: Mexican evidence, in: Proceedings of the 3rd Annual Meeting of EIBA, Antwerp, Belgium, 1987. [88] B.F. Schmid, M.A. Lindemann, Elements of a reference model for electronic markets, in: Proceedings of the 31st Hawaii International Conference on System Sciences Hawaii, 1998. [89] S.E. Seashore, Criteria of organizational effectiveness, Michigan Business Review 17 (1965) 26–30. [90] A. Segev, J. Gebauer, F. Färber, Internet-based electronic markets, Electronic Markets 9 (1999) 138–146. [91] K.B. Sheehan, E mail survey response rates: a review, Journal of Computer-Mediated Communication 6 (2001) 1–10. [92] Y.P. Sheng, P.P. Mykytyn Jr., C.R. Litecky, Competitor analysis and its defenses in the e-marketplace, Communications of the ACM 48 (2005) 107–112.

1075

[93] Y. Shetty, New look at corporate goals, California Management Review 22 (1979) 71–79. [94] C. Soh, M.L. Markus, K.H. Go, Electronic marketplaces and price transparency: strategy, information technology, and success, MIS Quarterly 30 (2006) 705–723. [95] S.S. Srinivasan, R. Anderson, K. Ponnavolu, Customer loyalty in e-commerce: an exploration of its antecedents and consequences, Journal of Retailing 78 (2002) 41–50. [96] C. Standing, C. Lin, Organizational evaluation of the benefits, constraints, and satisfaction of business-to-business electronic commerce, International Journal of Electronic Commerce 11 (2007) 107–134. [97] S. Standing, C. Standing, P.E.D. Love, A review of research on e-marketplaces 1997–2008, Decision Support Systems 49 (2010) 41–51. [98] R. Stockdale, C. Standing, A framework for the selection of electronic marketplaces: a content analysis approach, Internet Research 12 (2002) 221–234. [99] S. Strasser, J. Eveland, G. Cummins, O.L. Deniston, J.H. Romani, Conceptualizing the goal and system models of organizational effectiveness: implications for comparative evaluation research, Journal of Management Studies 18 (1981) 321–340. [100] D.W. Straub, R. Watson, Transformation issues in researching IS and net-enabled organizations, Information Systems Research 12 (2001) 337–345. [101] R. Tagiuri, J.A. Davis, On the goals of successful family companies, Family Business Review 5 (1992) 43–62. [102] J. Theurer, Seven pitfalls to avoid when establishing performance measures, Public Management 80 (1998) 21–24. [103] E. Triantaphyllou, Multi-criteria Decision Making Methods: A Comparative Study, Springer, Netherlands, 2000. [104] A. Umar, P. Missier, A framework for analyzing virtual enterprise infrastructure, in: Proceedings of the 9th International Workshop on Research Issues on Data Engineering: Information Technology for Virtual Enterprises, 1999. [105] N. Venkatraman, V. Ramanujam, Measurement of business performance in strategy research: a comparison of approaches, Academy of Management Review 11 (1986) 801–814. [106] L.G. Weinzimmer, P.C. Nystrom, S.J. Freeman, Measuring organizational growth: issues, consequences and guidelines, Journal of Management 24 (1998) 235–262. [107] B. Wernerfelt, A resource-based view of the firm, Strategic Management Journal 5 (1984) 171–180. [108] A. White, E. Daniel, Electronic marketplace-to-marketplace alliances: emerging trends and strategic rationales, in: Proceedings of the International Conference on Electronic Commerce, Pittsburgh, 2003. [109] R. Wise, D. Morrison, Beyond the exchange–the future of B2B, Harvard Business Review 78 (2000) 86–96. [110] Y. Xu, H.-W. Kim, Order effect and vendor inspection in online comparison shopping, Journal of Retailing 84 (2008) 477–486. [111] E. Yuchtman, S.E. Seashore, A system resource approach to organizational effectiveness, American Sociological Review 32 (1967) 891–903. [112] R.F. Zammuto, A comparison of multiple constituency models of organizational effectiveness, Academy of Management Review 9 (1984) 606–616. [113] Y. Zhuang, A.L. Lederer, An instrument for measuring the business benefits of e-commerce retailing, International Journal of Electronic Commerce 7 (2003) 65–99. [114] S. Zumpe, Goal dependent Success Strategies for Marketplace Operators in different Roles, in: Proceedings of the Eleventh Research Symposium on Emerging Electronic Markets, Dublin, Ireland, 2004. [115] S. Zumpe, Zielabhängige Erfolgsanalyse virtueller B2B-Marktplätze: Eine empirische Untersuchung, Dissertation, Technische Universität Dresden, 2004. [116] V. Zwass, Electronic commerce and organizational innovation: aspects and opportunities, International Journal of Electronic Commerce 7 (2003) 7–37.

Sabine Matook, Ph.D., is a senior lecturer in Information Systems at the UQ Business School, University of Queensland. She received her doctoral degree from the Technische Universität (TU) Dresden, Germany. Her research interests are on the IT artifact and in the areas of IT strategy, goal-related IS use and adoption, social media, and agile IS development. Her work has appeared in Decision Support Systems, European Journal of Information Systems, Journal of Strategic Information Systems, and the International Journal of Operations & Production Management. Dr. Matook has presented research papers at a variety of international conferences, including the International Conference on Information Systems.