Exploring the use of the Delphi method in accounting information ...

ACCINF-00277; No of Pages 16 International Journal of Accounting Information Systems xxx (2012) xxx–xxx

Contents lists available at SciVerse ScienceDirect

International Journal of Accounting Information Systems

Exploring the use of the Delphi method in accounting information systems research James L. Worrell a,⁎, Paul M. Di Gangi b, 1, Ashley A. Bush c, 2 a b c

University of Alabama at Birmingham, School of Business, BEC311A, 1530 3rd Ave S., Birmingham, AL 35294, United States Loyola University Maryland, Sellinger School of Business, Sellinger Hall 305, 4501 N. Charles St, Baltimore, MD 21210, United States Florida State University, College of Business, RBA510, Tallahassee, FL 32306, United States

a r t i c l e

i n f o

Article history: Received 16 January 2012 Received in revised form 5 March 2012 Accepted 6 March 2012 Available online xxxx Keywords: Delphi method Expert panel IT risk Methods in accounting research Interacting groups

a b s t r a c t Recent focus on the diversity of research methodologies available to accounting information systems (AIS) scholars has led researchers to suggest the Delphi method has reached the limits of its usefulness. Using a review of the accounting and information systems literature, we suggest such a finding is premature for the AIS discipline. The Delphi method is especially useful in reducing ambiguity through the use of expert panels of both practitioners and experts and informing relevant and timely issues facing organizations. In essence, the Delphi method has potential to provide both rigor and relevance to AIS researchers. Our purpose is to review the prior literature on the use of the Delphi method and discuss potential areas of research within the AIS discipline where the method might add value. Based on this review, we develop a series of guidelines on how to properly develop, administer, and assess panel responses and then use an illustrative study example that explores IT risks in operations. We conclude with a discussion of the value of the Delphi method and provide insight into its limitations. © 2012 Elsevier Inc. All rights reserved.

1. Introduction As a field of inquiry, accounting information systems (AIS) draws heavily from a wide variety of referent disciplines, including management information systems (MIS), organizational behavior, psychology, computer science and economics. Given this rich milieu, the AIS literature has diversity in phenomena, theoretical perspectives and methodological approaches. In a recent editorial on methodologies in AIS ⁎ Corresponding author. Tel.: + 1 205 934 8873. E-mail addresses: [email protected] (J.L. Worrell), [email protected] (P.M. Di Gangi), [email protected] (A.A. Bush). 1 Tel.: + 1 410 617 5521. 2 Tel.: + 1 850 644 2779. 1467-0895/$ – see front matter © 2012 Elsevier Inc. All rights reserved. doi:10.1016/j.accinf.2012.03.003

Please cite this article as: Worrell JL, et al, Exploring the use of the Delphi method in accounting information systems research, Int J Account Inf Syst (2012), doi:10.1016/j.accinf.2012.03.003

2

J.L. Worrell et al. / International Journal of Accounting Information Systems xxx (2012) xxx–xxx

research, Nicolaou (2011) advocated a broad view of the AIS discipline and encouraged a conversation on alternative methods and theoretical approaches that have potential for informing AIS research. We wish to contribute to this conversation by providing an overview and discussion of the Delphi method, and how it might be used for exploratory research, theory building and forecasting. The Delphi method is “intended for systematically soliciting, organizing and structuring judgments and opinions on a particularly complex subject matter from a panel of anonymous experts until a consensus is reached on the topic or until it becomes evident that further convergence is not possible” (Anderson et al., 1994, p478). The Delphi method is especially well-suited for exploratory, theory-building research efforts which involve complex, multi-disciplinary issues, especially if analyses of new or future trends are the focus of the research (Meredith et al., 1989; Neely, 1993; Akkermans et al., 1999, 2003; Daniel and White, 2005). Although the Delphi method has not seen widespread adoption in the AIS literature, this method does have distinct benefits when effectively paired with the appropriate research question and focal topic. For example, the Delphi method has been demonstrated to provide more accurate decisions than other group decision techniques, such as focus groups and nominal group technique (Rowe and Wright, 1999; Daniel and White, 2005). Our intention is not to suggest that one method is better than the other, but rather to provide an objective view of the method's strengths and weaknesses so that the researcher can make an informed decision on whether or not the Delphi method is the appropriate approach for her current inquiry. This manuscript will unfold as follows. First, we will provide an overview of the Delphi method, to include its history and purpose. We will examine the use of the Delphi method in accounting and information systems research. Second, we will provide guidance on designing and executing Delphi studies, to include design considerations, expert panel selection, and quantitative and qualitative analyses. Third, we will present an example of a seeded, ranking-type Delphi study to illustrate the use of the method. This example explores the perceptions of IT risk among three key stakeholder groups: IT auditors, business managers, and IT managers. Fourth, we will discuss the strengths of the Delphi method and its value in theory building and exploratory analysis. Finally, we will conclude with the limitations of the Delphi method.

2. The Delphi method: an overview 2.1. History and characteristics The Delphi method originated in the early 1950s at the RAND Corporation, a California-based think-tank (Dalkey and Helmer, 1963). Named for the famed Oracle at Delphi, there have been numerous implementations and variations on the original classical Delphi method and all share four core characteristics (Linstone and Turoff, 1975; Turoff and Hiltz, 1995). First, all employ a panel or group of panels composed of knowledgeable experts. Rather than attempting to assemble a statistically representative sample, the Delphi technique utilizes a purposely selected panel of experts to opine on a problem or situation. The rationale for this design choice is that a non-representative sample of experts is more apt to arrive at a correct decision that is a representative sample of non-experts (Rowe and Wright, 1999; Okoli and Pawlowski, 2004). Second, all members of the expert panel remain unknown to each other throughout the execution of the study (Linstone and Turoff, 1975; Turoff and Hiltz, 1995). Anonymity among the panelists was (and remains) crucial as a means of guarding against the effects of individual biases, personal influences and groupthink on the ability to reach consensus. Anonymity allows panelists to freely offer alternatives and expertise without fear of reprisals or judgment. In multiple round approaches, anonymity allows panelists to alter their opinions based on feedback from the panel without fear of losing credibility or status. Third, group communication is utilized to manage feedback and develop consensus among the expert panel (Linstone and Turoff, 1975; Turoff and Hiltz, 1995). In early implementations of the Delphi method, communication between the expert panelists involved mailing paper-based surveys. This increased the time necessary to complete a multi-round Delphi, as well as the cost to the researcher. Panelist fatigue (i.e., the panelist grows weary with the time commitment and discontinues participation) is a concern in Delphi studies, with reliance on traditional mail increasing the risk associated with this method. More recently, web-based survey software and other information Please cite this article as: Worrell JL, et al, Exploring the use of the Delphi method in accounting information systems research, Int J Account Inf Syst (2012), doi:10.1016/j.accinf.2012.03.003


3

communication technologies (ICTs) have significantly reduced the costs and time associated with this research method, thereby ameliorating the risk of panelist fatigue. Finally, Delphi studies are characterized by controlled feedback and iteration (Linstone and Turoff, 1975; Turoff and Hiltz, 1995). With each phase in a Delphi study, the expert panel receives feedback on the quality of the decision and the level of consensus among the panel. Feedback on the decision process, quality and consensus allow the expert panel to have a structured debate on the merits of the alternatives. Iteration, whereby the expert panel reviews and evaluates alternatives through several controlled phases, affords the panelists opportunities to reflect on opinions and assessments of their peers and to utilize these to shape and reinforce their opinions. 2.2. Use in AIS and MIS research, implementation and variation While a rich stream of research utilizing the Delphi method exists within the management, marketing, information systems and psychology disciplines, accounting researchers have been slow to adopt this method. Traditionally, studies employing the Delphi method fall into two camps: those that employ the method as a forecasting tool, and those that employ the method to evaluate the relative importance of factors and frameworks. Table 1 below provides an overview of selected Delphi studies from the AIS and MIS literatures, as well as details expert panel composition, implementation approach and summary findings. Traditionally, the Delphi method has consisted of four phases: (1) assemble expert panel, (2) brainstorm alternatives, (3) narrow alternatives, and (4) rank alternatives. These steps follow from the original purpose of identifying and ranking alternatives for forecasting. With the passage of time, there have been several innovations in the Delphi method which have resulted in variations in its implementation. Two variations in particular warrant further discussion, as they have the most potential and applicability AIS research. One variation has been to modify the second phase (brainstorm alternatives) to allow for a seed of factors. The purpose of the seed is to account for factors from theory or prior literature which have been demonstrated to be relevant to the problem at hand. In using the seed as a starting point for narrowing alternatives, most Delphi studies utilizing this approach do not eliminate brainstorming, but rather use brainstorming to augment the seed derived from prior literature, frameworks or other sources. The use of a seed need not be limited to extending prior research or in building a research stream. Many times, the seed represents a starting point based on factors drawn from the literature. Prior literature, frameworks and factors identified in practitioner journals have been used as the basis for seeds in Delphi studies (Doke and Swanson, 1995; Greenstein and Hamilton, 1997; Fomin et al., 2008). A second variation has been to modify the first phase (selection and composition of the expert panel) to include multiple expert panels. The emergence of multi-panel Delphi studies can be explained by the need for researchers to account for multiple perspectives in complex, multi-dimensional problems. Prior literature has identified the risks and issues associated with exploring organizational concerns from a single perspective, and the deleterious outcomes when rival perspectives are not accounted for (Bassellier et al., 2001; Schmidt et al., 2001; Keil et al., 2002). Recognizing this as a limitation, a growing number of Delphi studies have incorporated a multi-panel design, whereby two or more expert panels are utilized in the evaluation and ranking of factors or alternatives (Schmidt et al., 2001; Keil et al., 2002; Liu et al., 2010). These studies (and the one briefly described in Section 4) allow researchers to more fully explore the differences between stakeholder groups as well as identify where different stakeholder groups agree on important factors. While multi-panel designs are gaining popularity, there are numerous studies that have employed the traditional single panel design with diverse expert composition. For example, Daniels and White's (2005) study on the future of interorganizational system linkages utilized a single panel design composed of experts with specializations in a variety of technology and interorganizational system areas, with diverse roles and managerial levels. Similarly, De Haes and Van Grembergen's (2009) study on the implementation and alignment of IT governance employed a panel with expertise in business management, IT assurance, IT management and IT consulting. Although one might question whether such a diverse expert panel can reach consensus on key issues, results from these studies (and others, see Fomin et al., 2008; Holsapple and Joshi, 2001; McCubbrey, 1999; McFadzeen et al., 2011) with diverse panel composition have successfully reached consensus in addressing their focal topic. The appeal of the multi-panel design, Please cite this article as: Worrell JL, et al, Exploring the use of the Delphi method in accounting information systems research, Int J Account Inf Syst (2012), doi:10.1016/j.accinf.2012.03.003

4


Table 1 Selected Delphi studies from AIS and MIS literatures. Citation Theme: forecasting Akkermans et al. (2003)

Baldwin and Trinkle (2011)

Focus/motivation

Panel composition (size)

To examine the future effects of enterprise resource planning systems (ERPs) on supply chain management (SCM) To predict the future impacts of extensible business reporting language (XBRL) on financial reporting

Dutch SCM executives Brainstorming-style (23) Delphi, using group decision support systems to facilitate faceto-face anonymous interaction Brainstorming-style Accounting faculty, Delphi XBRL users and developers (9)

Baldwin-Morgan (1993)

To predict the potential Accounting faculty effects of expert systems and public accounting auditors (18) on delivering audit services

Brancheau and Wetherbe (1987)

To predict critical IS issues facing IS executives in the short term (circa 1985)

Chief IS executives, IS department managers and IS consultants (54) General managers (12)

Brancheau et al. (1996)

To predict critical IS issues facing IS executives in the short term (circa 1996)

Society for Information Management (SIM) institutional and board members (83)

Daniel and White (2005)

To explore the nature of future interorganizational system (IOS) linkages

Experts in IT and IOS with various roles from diverse geographic locations (15)

McCubbrey (1999)

To predict the effects of electronic commerce technologies on the air travel distribution industry

Prominent experts in travel agency industry and air travel industry associations (17)

Theme: factors and frameworks Bonson et al. (2009) To identify factors influencing North American companies to early adopt XBRL De Haes and Van To examine how Grembergen (2009) organizations are implementing IT governance and to explore the relationship between IT governance and business/IT alignment Doke and Swanson To identify factors used (1995) by IS managers when

Accounting faculty, XBRL developers and researchers (29)

Business, audit, IT management and IT consulting professionals (22)

IS managers employed by

Implementation

Findings

Predicted a modest role for ERP in improving SCM effectiveness, suggesting that 1st generation ERPs are inadequate in the new economy Predicted that XBRL adoption will benefit financial reporting, regulatory compliance, access to financial reporting Predicted that expert Ranking-style Delphi systems were likely to used to evaluate the improve documentation, likelihood of a series of propositional statements distribution of expertise and enhance decision occurring in the future quality (by 2001) Predicted the top ten Brainstorming-style critical issues facing IS Delphi used to identify executives in the coming 19 distinct challenges three to five years expected to confront IS executives in the short term Predicted the top ten Brainstorming-style critical issues facing IS Delphi used to identify executives in the coming 23 distinct challenges three to five years expected to confront IS executives in the short term Predicted that IOS deBrainstorming-style Delphi using four key IOS velopment will focus on themes to guide elicita- widespread adoption of tion of key directions in (1) ERPs, (2) web services, (3) e-hub and (4) technologies enterprise portals Identification of channel Predicted that the airline travel will experience players and market significant changes due segments by industry experts, with evaluation to cybermediaries, airline direct services by the expert panel and increasing fees

Ranking-style Delphi with seed informed by literature, option for experts to provide alternative factors Ranking-style Delphi with seed informed by literature, option for experts to provide additional factors

Rankings suggest that XBRL early adopters do so to gain knowledge of the technology and to be viewed as pioneers Provided a list and ranking of factors related to IT governance along the following dimensions: structures processes, relational mechanisms

Ranking-style Delphi with seed informed by



5

Table 1 (continued) Citation

Fomin et al. (2008)

Greenstein and Hamilton (1997)

Iden and Langeland (2010)

Kasi et al. (2008)

Keil et al. (2002)

Liu et al. (2010)

Focus/motivation

Panel composition (size)

Implementation

Findings

deciding to include prototyping in software development projects To examine the role of standards in developing and governing public information communication technologies infrastructures

organizations on Computerworld Premier 100 list (27) Diverse group of professionals, government officials and academics with experience in IT (13)

literature, option for experts to provide additional factors Ranking-style Delphi with seed informed by literature, option for experts to provide additional factors, focused on looking for areas of dissensus

To identify factors affecting public accounting firms' client acceptance/continuance decisions To identify the factors necessary for a successful ITIL implementation

Accounting professionals (16)

Ranking-style Delphi with seed informed by literature and review of firm policies and interviews with experts Brainstorming-style Delphi used to identify and rank critical success factors

Provided list and ranking of factors influencing prototyping decisions Rankings suggest that economics of standards, public good and compliance, and intellectual property rights are most important considerations Rankings suggest a list of 18 troublesome factors that influence client acceptance/continuance decisions Identified factors and associated rankings suggest those related to senior management are most important; provided list of 65 success factors for ITIL implementations Results suggest that organizations are ill equipped to learn from their mistakes on subsequent projects Identified zones of concordance and discordance between IT project managers and users of IT

Norwegian Armed Forces IT managers with experience in ITIL v2/v3 (15)

Practitioners with expertise in conducting postmortem examinations (23) To identify IT project risk Users of IT, IT project factors that are salient to managers (15) for IT project managers and user panel, IT project manager panel data users pulled from Schmidt et al. (2001) To identify and compare IT project managers IT project risk percep(34) Senior managers tions of IT project man- (30) agers and senior managers in Asian culture To identify the barriers of conducting postmortem analysis for IT projects

Ranking-style Delphi with seed informed by literature, option for experts to provide additional factors Ranking-style Delphi with seed informed by literature, option for experts to provide additional factors Ranking-style Delphi with seed informed by literature, option for experts to provide additional factors

Compared results to prior IT project risk studies, noting that culture affects perceptions; identified approaches for remediating IT project risk in collectivist societies

however, is its ability to isolate diverse groups (be they cultural, social, hierarchical or functional) and more effectively tease out their similarities and differences. 3. Guidelines for research using Delphi method The Delphi method has the potential to provide deep understanding of current issues facing both research and practice. From an operational perspective, the Delphi method is relatively simple to administer; however, design choices made before administering the Delphi questionnaire directly impact the rigor and relevance of the results. Failure to take into consideration such elemental items as proper research question specification, panel composition, method design, and analysis for interpretation, can compromise the results and their generalizability. Table 2 provides an overview of design elements to be considered, as well as general implementation guidance. Please cite this article as: Worrell JL, et al, Exploring the use of the Delphi method in accounting information systems research, Int J Account Inf Syst (2012), doi:10.1016/j.accinf.2012.03.003

6


Table 2 Delphi Study design elements and guidelines. Design elements Participants How are expert panelists picked?

How many panelists are needed?

Brainstorming vs. seeded list Should the Delphi begin with brainstorming a list? Should the Delphi begin with a seeded list? Where does the seeded list come from? How many items are on the initial seeded list?

General guidelines • Convenience sample based on the researcher's knowledge of experts in the area (see Brancheau and Wetherbe, 1987; Baldwin-Morgan, 1993; McCubbrey, 1999; Schmidt et al., 2001; Keil et al., 2002) • KRNW approach (see Okoli and Pawlowski, 2004) • Panel as small as 4 experts under ideal circumstances (see Brockhoff, 1975; Delbecq et al., 1975) • Panel between 10 and 30 experts under typical circumstances (see Baldwin-Morgan, 1993; Doke and Swanson, 1995; Keil et al., 2002; Akkermans et al., 2003; Daniel and White, 2005; Kasi et al., 2008; De Haes and Van Grembergen, 2009; Baldwin and Trinkle, 2011)

• Objective of study is to identify factors pertinent to a topic or domain (see Brancheau and Wetherbe, 1987; Akkermans et al., 2003; Daniel and White, 2005; Baldwin and Trinkle, 2011) • Objective of study is to rank or prioritize an established list of factors or issues (see Greenstein and Hamilton, 1997; Keil et al., 2002; Kasi et al., 2008; Liu et al., 2010) • Seed drawn from prior literature and/or existing frameworks (see Greenstein and Hamilton, 1997; Holsapple and Joshi, 2000; Keil et al., 2002; Kasi et al., 2008; Liu et al., 2010) • 20 to 25 items (see Okoli and Pawlowski, 2004)

Rounds How many rounds?

• Iterate until (a) consensus is reached or (b) plateau in consensus is reached (see Schmidt, 1997) o Strong consensus reached at Kendall's W = 0.7 (see Schmidt, 1997) o General guideline is 3 rounds before panelist fatigue becomes issue (see Okoli and Pawlowski, 2004) o Subsequent rounds do not result in a change in consensus (see Okoli and Pawlowski, 2004) What criteria are used for keeping • Initial winnowing accomplished by carrying forward all factors which were selected items at each round? by a simple majority (≥50%) of experts (see Schmidt, 1997; Schmidt et al., 2001; Keil et al., 2002) What statistics are reported for each • Delphi studies using Kendall's W for assessing consensus (see Schmidt, 1997; Schmidt round? et al., 2001; Keil et al., 2002; Kasi et al., 2008) o Mean ranks of factors o Kendall's W for each round • Delphi studies using reduction in standard deviation for assessing consensus (see Brancheau and Wetherbe, 1987; Brancheau et al., 1996) o Mean ranks of factors o Standard deviation for each round

3.1. Aligning research method and question Given the unique characteristics of the Delphi method, pairing research method and approach is critical. The choice of research question guides the development of the research effort, namely through supporting the formulation of a theoretical perspective and the subsequent methodology used to explore a phenomenon (Creswell, 2003). For studies employing the Delphi method, appropriate research questions include those that reflect two conditions: 1) sufficient ambiguity from prior research or the lack of pre-existing information to reach a solution and 2) a need to provide order or the assignment of relative importance of a set of items for research and/or practice. From a researcher's perspective, literature highlighting divergent findings or contradictions among similar studies would indicate a lack of broad consensus and support the need for expert judgment to forge a path forward. Please cite this article as: Worrell JL, et al, Exploring the use of the Delphi method in accounting information systems research, Int J Account Inf Syst (2012), doi:10.1016/j.accinf.2012.03.003


7

3.2. Panel design considerations An expert panel in a Delphi study is a group of individuals deemed experts by either objective (e.g., job title, work experience, organizational affiliation) or subjective (e.g., reference triangulation) factors. It is this expertise that suggests the individual can make an informed opinion on the focal topic. Researchers must carefully consider expert panel composition as part of their design considerations, as a suitable expert panel is essential to assuring the validity of the findings and satisfying subsequent scrutiny of reviewers for publication consideration. When considering how to form a panel, researchers should pay particular attention to panel composition and panel size. While expert panels are often assembled based on a convenience sample, a more objective and methodical approach is to utilize a Knowledge Resource Nomination Worksheet (KRNW). Using this approach, the researcher first identifies the relevant disciplines or skills that are applicable to the panel topic with a KRNW. Once these experts are identified, a methodical selection process ensues (see Fig. 1), resulting in the appropriately composed expert panel (Okoli and Pawlowski, 2004). As an alternative to using convenience samples for expert panel selection, the use of social networks and network analysis to identify relevant individuals within an embedded social structure (such as an expertise domain) can be a valuable method for identifying the pool of potential panelists (Worrell et al., 2012). Using keyword analysis on professional social networks (e.g., LinkedIn), the researcher can identify an individual's work experience, professional references, and key contact information that can be used to just the subject matter expert's ranking as a potential panel member. Social network technologies also provide researchers with the ability to identify additional members via connections and friendships visible on profile pages. While panel composition is important, researchers often struggle with what is considered an acceptable expert panel size for Delphi studies. A review of Delphi studies published in AIS and MIS journals reveals that most studies utilize between 10 and 30 expert participants (Baldwin-Morgan, 1993; Doke and Swanson, 1995; Greenstein and Hamilton, 1997; McCubbrey, 1999; Schmidt et al., 2001; Keil et al., 2002; Akkermans et al., 2003; Daniel and White, 2005; Nevo and Chan, 2007; Fomin et al., 2008; Kasi et al., 2008; Bonson et al., 2009; De Haes and Van Grembergen, 2009; Iden and Langeland, 2010; Baldwin and Trinkle, 2011). However, an expert panel as small as four is appropriate, if panelists demonstrate deep understanding of the subject matter or the focal topic requires a unique set of conditions where only few panelists are capable of contributing towards a solution (Brockhoff, 1975; Delbecq et al., 1975). Studies examining the relationship between panel size and effectiveness in decision making found no consistent relationship between the two (Brockhoff, 1975; Boje and Murnighan, 1982). In effect, panel size is dependent upon the requirements identified in the panel composition process as well as the characteristics of the individual panel members.

Identify skills and literature • Identify disciplines and skills relevant to current study • Identify relevant literature that informs topic of interest • Identify organizations and entities tied to the topic of interest

Link skills to experts

Snowball expert list

• Populate KRNW with names of experts associated with • discipline • literature • organizations • entities • social networks • electronic communities

• Contact experts from KRNW and request contact information for other (previously unidentified) experts

Rank experts • Establish objective criteria for evaluating experts • qualifications • certifications • job title • years in industry • Rank experts based on objective criteria

• Identify social networks and other electronic communities with expertise relevant to current study

Fig. 1. Procedure for selecting panelists. Adapted from Okoli and Pawlowski, 2004.


8


3.3. Delphi design and administration considerations Design and administration considerations generally center on whether to use brainstorming or seeding to begin the Delphi study, how to execute the ranking of items or factors, and how to evaluate the outcome of the Delphi study. Each of these is discussed below. As mentioned in Section 2.2, Delphi studies employ either a brainstorming or a seeded approach as the beginning point for their ranking or forecasting exercise. This choice is dependent upon the initial starting point of the Delphi study as a condition of a properly specified research question. The first approach, the brainstorming approach, is best used when there is no pre-existing literature that contains items that can be rank ordered by the panel. The second approach, the seeded approach, is best when there is preexisting literature that the researcher wishes to leverage or extend for the current study. Oftentimes, the seeded approach will use a brainstorming component to allow expert panelists to provide additional factors not covered in the extant literature. Irrespective of the approach used to establish a starting list of factors, these items or factors will be ranked by the expert panel. Oftentimes, the initial list or seed will include a large number of items. In many cases, an initial constraint can be placed on the panel to select a maximum number of items to encourage a winnowing effect (e.g., select the ten most important items without assigning ranks to the ten items). Following this initial round, the researcher will then analyze the results and identify all items that were selected by the majority of all panelists creating the finalized item list to be used in the subsequent ranking rounds. The list that enters the ranking portion of the Delphi study should be a manageable size. Using this winnowed list, the expert panel begins the ranking process, which focuses on achieving consensus among the panelists concerning the rank ordering of each of the individual items. Each panelist is asked to independently rank each item from most important to least important and provide justification for their decisions. The justification will be used as part of the structured feedback process in subsequent ranking exercises, thereby enabling the expert panel to consider other opinions and rationales. As the objective of the Delphi method is to achieve consensus among a panel of experts, having an objective approach to knowing when consensus is achieved is important. While earlier Delphi studies used a reduction in the standard deviation as a measure of consensus from round to round (Brancheau and Wetherbe, 1987; Brancheau et al., 1996), this approach has been demonstrated to be suboptimal and has generally been discounted (Schmidt, 1997). More recently, the use of Kendall's coefficient of concordance (W) has been advocated as a more accurate manner to calculate consensus among a panel of experts (Schmidt, 1997; Keil et al., 1998; Schmidt et al., 2001; Keil et al., 2002; Kasi et al., 2008). This non-parametric statistics is evaluated on a scale of 0, indicating no agreement, to 1, indicating full agreement among panelists (Kendall and Babbington Smith, 1939). Kendall's W has also been used to evaluate agreement among experts in studies investigating factors that affect the quality of audit processes (Sutton, 1993; Lampe and Sutton, 1994). Schmidt (1997) provides guidance on how to interpret Kendall's W (i.e., a value range anchored by no consensus to very strong consensus, along with the amount of confidence in the mean rankings for each value range). Achieving strong agreement (W ≥ 0.7) has traditionally been used as an ideal target for Delphi studies. As the ranking exercise proceeds through the various rounds, feedback is invaluable in assisting the expert panel in their deliberations. This feedback takes on a variety of forms. As mentioned previously, if justification is solicited from the expert panel on why they selected certain items over others, this should be provided to the expert panel in subsequent rounds. Additionally, some have suggested that each panelist has the ability to see how he or she ranked an item in the previous round (in addition to the mean rank from the panel as a whole) as well as an indication of the current agreement of the panel concerning the ranking of items (Okoli and Pawlowski, 2004). Thus, an ideal approach for providing feedback to the expert panel after each round is 1) to re-order the list of items based on mean rank from the previous round, 2) to present a summary of the justifications for rankings by other panel members, 3) to disclose to each panelist information concerning how the panelist ranked each item in the previous round, and 4) to indicate current agreement among panel members (i.e., an assessment of consensus based on an interpretation of Kendall's W). This will allow panelists to assess whether they fully examined an item (in comparison to justifications made by other panelists) and adjust their rankings if needed. Studies have shown that providing panelists with ‘reasons’ feedback that explains rationales and justifications for Please cite this article as: Worrell JL, et al, Exploring the use of the Delphi method in accounting information systems research, Int J Account Inf Syst (2012), doi:10.1016/j.accinf.2012.03.003


9

why panelists selected or ranked items improves the decision quality of the Delphi method (Best, 1974; Rowe and Wright, 1999). It is important to recall that a fundamental part of the Delphi method is the complete anonymity of all panel members. Consensus must be derived through the independent subjective judgments of the individual panel members. While the qualitative comments are included as an information resource, the panelists should be reminded that their ranking decisions are determined by their individual judgments. While achieving strong consensus is certainly the goal, there are times (as in the example provided in Section 3) where the expert panel is unable to reach strong consensus. The question for the researcher then becomes when to conclude the study. Terminating a panel can occur due to a variety of positive and negative reasons. The ideal scenario for termination is when a panel adjourns because consensus has been reached. A panel can also terminate because it has reached a pre-specified number of iterations and continuation would result in panelist fatigue. Lastly, a panel can terminate if the subsequent ranking rounds do not result in a statistically significant difference in consensus (using the McNemar test for repeated measures of a respondent's rankings; Hollander and Wolfe, 1999; Okoli and Pawlowski, 2004). However, we recommend that a final round be conducted to ensure the panel has reached a plateau of consensus and that the previous ranking was not an anomaly, if possible (Schmidt, 1997). Additionally, researchers should consider including a satisfaction measure to determine whether the panel has reached consensus rather than exhaustion due to the time intensive process of the Delphi method (Rowe and Wright, 1999). For multi-panel designs, the researcher needs to resolve how interpretation of consensus and dissensus among the panels will be addressed. For past studies, one approach has been to use a Venn diagram or some other representation showing the factors or items that all groups agreed upon, thereby highlighting similarities and differences (Schmidt et al., 2001; Keil et al., 2002; Liu et al., 2010). While this approach is useful, it places the burden on the researcher to theorize why there were agreements and disagreements among the expert panels. To ameliorate this concern and strengthen the validity and generalizability of the findings, we recommend the final rankings be distributed to each expert panel with the following information: 1) rationale for multiple panels, 2) results from each panel, and 3) any justifications for rankings provided by members of the other expert panels. With this feedback, panelists may then be asked to provide additional commentary concerning their individual rankings in comparison to the rankings of the different panels. By soliciting feedback in this manner, the researcher is able to gather valuable insights into why the different panels may have reached different conclusions, thereby removing much of the guesswork in explaining study outcomes. 3

4. Information technology risk Delphi: an illustration Information Technology (IT) can be an organization's most strategic asset when competing; however, IT also holds the potential of being its greatest threat. News media frequently highlight instances where an organization suffers from the consequences of an IT glitch or security breach. DataLossDB, an open source research project that catalogues information security breaches and data loss events related to IT, reported 895 incidents in 2011 and 140 for January and February of 2012. 4 Staples, an office supply company, recently experienced a data theft from an employee with estimated losses of $181,800 from 50 stolen customer credit card numbers. 5 These failures and countless others illustrate the need for a coherent organizational strategy to manage the risks associated with IT within the production environment. We define IT risks as the risk that an organization's information systems will not adequately support the organization in achieving its business objectives, sufficiently safeguard its information resources, or deliver accurate and complete information to its users. Since IT risks impact both the technical infrastructure of an organization as well as a variety of business processes and managerial areas, organizations must consider and mitigate IT risks from multiple stakeholder perspectives. 3 We would like to acknowledge the thoughtful comment from an anonymous reviewer during the development of this research for suggesting this approach to mitigate a potential weakness in the findings of our Delphi study. 4 http://datalossdb.org/statistics (accessed 02/24/2012). 5 http://www.thedailyharrison.com/news/mamaroneck-staples-cashier-charged-181 k-fraud (accessed 02/24/2012).


10


A strategy for IT risks requires input from business professionals (who leverage IT for executing business processes), IT professionals (who develop and maintain an organization's technical infrastructure at the operational level), and auditors (who assess whether sufficient controls and procedures are in place to protect the assets of an organization). However, empirical evidence suggests a disconnect between these stakeholders regarding decision making and risk identification/mitigation (Bassellier et al., 2001; Schmidt et al., 2001; Keil et al., 2002; Bassellier and Benbasat, 2004). Thus, our study focuses on examining IT risks in operations, where information systems in the production environment affect both business and IT professionals. Motivated by these concerns, the purpose of our study was to address the following research question: How do each of the major stakeholder groups within organizations (representing both strategic and operational levels) conceptualize the risks associated with IT in operations? While identifying IT risks, we examined the literature from both the AIS and MIS disciplines to determine the depth and breadth of IT risk factors affecting business today. From this analysis, the literature suggests that each field views IT risks through somewhat related, but different lenses. AIS scholars view IT risks as representing threats to the reliability of financial reporting, while MIS scholars perceive IT risks from the security vulnerabilities created from specific IT resources in operation. For instance, the MIS literature suggests a variety of risks that can disrupt the technical infrastructure of an organization ranging from physical damage to unauthorized access or inadvertent damage from authorized personnel (Rainer et al., 1991; Loch et al., 1992; Straub and Welke, 1998). Thus, the first step in creating a cohesive strategy around IT risks requires an understanding of how each stakeholder group views the risks facing an organization via IT. We conducted a multi-panel, seeded, ranking-type Delphi study to identify IT risks and compare these risks across stakeholder groups to identify common elements and areas of divergence. This variation of the Delphi method was selected for several reasons. First, the research question deals with a complex topic that is multi-disciplinary in nature, with a dearth of model-based statistical methods available for use. Second, this research effort requires evaluating the judgments of diverse groups of experts, with sometimes competing and conflicting priorities, possessing multiple perspectives and knowledge domains; allowing for this requirement suggests a multi-panel design. Finally, the existence of prior literature on IT risk factors in operations provides a starting point for evaluating other factors that might contribute to risks of IT in operations. Three expert panels were assembled, representing practicing IT auditors (n = 17), and business (n = 15) and IT (n = 12) managers primarily from Fortune 1000 companies. Table 3 shows the key demographics from each panel. Prior to soliciting panel member participation, we created an initial list of 22 risk factors related to IT in operations. This seed was developed based on Sherer and Alter's (2004) review of the IT risk literature. Once compiled, we invited panelists to participate in the Delphi study via a web-based survey which randomized the list of IT risk factors with their associated definitions. Independently, each panel was asked to select the ten most important IT risk factors. Additionally, panelists were asked to identify any additional IT risk factors not included in the initial seeded list but deemed important based on their expert opinion. While several offered additional risk factors, these were determined to either overlap with existing factors or were outside the scope of the study. 6 Each panel's list was narrowed by dropping any IT risk factors that failed to receive a simple majority vote (more than 50%) for that panel. This resulted in the IT auditors' IT risk factors list containing eight items, the business managers' list containing nine items, and the IT managers' list containing ten items. The subsequent list was then distributed to each panel and given the instructions to rank order the IT risk factors from most important to least important as well as provide qualitative justification behind their rankings. Once this task was completed, the mean ranks for each IT risk factor were calculated, as well as Kendall's W to determine the degree of consensus among each expert panel on the ranking. Subsequent iterations presented the panelists with the IT risk factors ordered by their mean ranks, the justification provided in the previous step for experts selecting their most important IT risk factor, and feedback indicating the degree of consensus among their panel relative to the IT risk factor rankings. This

6 Several experts suggested IT risk factors that were associated with IT project risks and software development risks. Because our study was focused on IT in operations, these suggested risk factors were deemed outside the scope of the current study.



11

Table 3 Individual demographics. IT auditors (n = 17)

Business managers (n = 15)

IT managers (n = 12)

Educational level Associate Bachelors Masters

– 44% 56%

– 57% 43%

27% 64% 9%

Years in field Mean St. dev.

7.69 2.73

14.5 7.29

16.45 7.19

Tenure in organization Mean St. dev.

4.67 2.75

4.43 2.82

6.09 4.21

continued until either each panel independently reached consensus or demonstrated signs of panelist fatigue. Fig. 2 highlights Kendall's W statistics for each of the three panels. All three panels reached marginal consensus (IT auditors panel: W = 0.40, business managers panel: W = 0.30, IT managers panel: W = 0.37). The IT auditors panel achieved moderate consensus and further rounds were deemed untenable due to a flattening of the consensus in the two preceding rounds (W = .37 and .40 in rounds 2 and 3 of the rankings). In contrast, both the business and IT manager panels demonstrated a decrease in consensus over time indicating that the maximum level of consensus had already been reached and no further rounds would produce fruitful observations. Thus, we chose to terminate the Delphi panels based on achieving consensus and panelist fatigue, respectively. Table 4 presents the IT risk factor, final mean rank, and the definitions for each of the IT risk factors. Given the diverging interests of the three stakeholder groups, it was not surprising that we find clear differences in terms of the most important IT risk factors and only small convergence on IT risk factors that span all stakeholder groups. Of the IT risk factors identified across all three expert panels, only three were common across all panels: lack of organizational alignment between business and IT, interdependencies between systems, and technical complexity. The IT auditors panel consistently ranked issues related to IT governance as most important, while the business and IT managers panels consistently ranked issues related to technology security and configuration as most important.

Fig. 2. Delphi results: Kendall's W.


12


Table 4 IT risk factors, rankings and associated definitions. IT risk factor

IT-A BM IT-M Definition

Lack of organizational alignment between business and IT Unauthorized information access

1

Information quality

3

1

Interdependencies between systems Weak change management

4

4

5

8

Lack of IS participation in business initiatives Resource insufficiency

6 7

7

Technical complexity

8

9

9

Problematic interfaces between systems Difficulty integrating software from vendors and subcontractors Malicious software

2

4

5

5

Software errors/bugs

3

Unauthorized physical access to hardware and processing environment

6

2

2 3

1

7

6 8 10

Failure to align the IT infrastructure and applications with business needs Firm's information or information systems are not adequately secured against unauthorized logical access Failure in management decision-making resulting from irrelevant, incorrect, or insufficient information provided by the information system The need for systems to share data with other applications or systems, either internal or external to the organization Weak policies and procedures governing changes to applications or technical infrastructure and other information system components Failure to aggressively leverage or engage IT enablers in business initiatives Insufficient resources are available to carry out or execute an IS initiative or plan, such as insufficient personnel or budgeting Information system or application is comprised of multiple components that combine to yield a complex system Information and data exchanges between systems do not occur completely, accurately, or in a timely manner Integration of packages from multiple vendors hampered by incompatibility and lack of cooperation Software written to produce an undesirable effect to the system, user, or organization Programming problems typically resulting from oversights of programmers and/or analysts Weak, ineffective, or inadequate physical control over access to the processing environment

Further examination of qualitative data (justifications for ranking an IT risk factor as most important and follow-up interviews with several experts) suggests that the IT auditors panel was able to reach consensus based on a common background and world view; all were experts in various risk management/ internal control frameworks and managed a diverse client portfolio which likely insulated them from being influenced by issues at a single organization. Overall, our results suggest that stakeholders from business, IT and audit communities often have difficulty in both identifying those IT risk factors that merit attention, as well as assessing their relative importance. Moreover, our results suggest that IT risk is situational, and that managers who consistently work in the same organization often find it difficult to look beyond the daily challenges that their organization faces. However, a word of caution should be taken when interpreting the results of this study. Specifically, we relied on our observation and interpretation of the results as authors conducting the study, and did not conduct a final integrated round soliciting feedback from the panel on comparison findings across panels. Future research should consider this a vital step in validating that each panel viewed the factors similarly and can provide qualitative feedback as to the motivations or underlying causes of these divergent findings. The purpose of this section was to provide an overview of a seeded, multi-panel, ranking-type Delphi study. As this summary was necessarily brief, researchers wishing to utilize this method will require additional guidance. For researchers seeking more in-depth guidance on planning and performing Delphi studies, we encourage them to review Linstone and Turoff's (1975) work on the Delphi method, which is widely considered the seminal piece on this method. For researchers seeking more guidance on selecting and managing expert panels, we encourage them to review Okoli and Pawlowski's (2004) work on panel selection and composition. Finally, for researchers seeking more guidance on statistical methods for calculating and interpreting consensus, we encourage them to review Schmidt's (1997) work on using nonparametric statistics in Delphi studies. Please cite this article as: Worrell JL, et al, Exploring the use of the Delphi method in accounting information systems research, Int J Account Inf Syst (2012), doi:10.1016/j.accinf.2012.03.003


13

5. Benefits of the Delphi method The benefits of the Delphi method are fourfold: 1) applied research tightly linked to both research and practice, 2) research agendas for scholarship, 3) exploratory analysis for theory building, and 4) rigor and relevance for business scholars. The Delphi method has been used as a tool to uncover underlying issues and to derive consensus among a panel of experts. Recent studies using the Delphi method have addressed diverse topics such as post mortem evaluation of failed projects (Kasi et al., 2008), information assurance (McFadzeen et al., 2011), offshoring and outsourcing (Nakatsu and Iacovou, 2009), impact and adoption of XBRL (Bonson et al., 2009; Baldwin and Trinkle, 2011), and the adoption of expert systems in auditing (BaldwinMorgan, 1993). Common across these studies is that the focal topics were directly applicable to pressing research and industry concerns. Furthermore, the use of experts, both academic and practitioner, to reach consensus suggests that an applied solution towards identifying issues facing scholars and industry can provide valuable insight to both. Using the method in this fashion creates an opportunity to provide scholars with a roadmap or research agenda on how to approach a specific phenomenon which can enable and direct scholars in their research efforts. The Delphi method employs an abductive approach to theory development in that theory is derived from the outcome of a discussion and results from experts with direct experience of a focal topic. The brainstorming phase along with the refinement in producing the finalized list of issues represents an abductive approach toward construct reduction which is vital to creating a parsimonious theoretical model (Greenstein and Hamilton, 1997; Dubois and Gadde, 2002). The strength of the Delphi method is that researchers do not begin a priori with a set of expectations about the underlying causes or drivers of a particular phenomenon. Instead, the Delphi method allows the researcher to uncover the reasons driving the phenomenon, and then use those insights to inform further inquiry much like a grounded theory approach to theory development (Okoli and Pawlowski, 2004). Furthermore, the qualitative comments and justifications made by panelists provide researchers with the ability to judge causality which is crucial for theory development. Thus, the Delphi method can be an excellent approach towards theory development and refinement, and a stepping stone for theory testing (for examples, see De Haes and Van Grembergen, 2009; McFadzeen et al., 2011; Nevo and Chan, 2007). Finally, the Delphi method provides a unique opportunity to balance the much debated topic of rigor versus relevance (Benbasat and Zmud, 1999; Lee, 1999; Glass, 2001; Gulati, 2007; Straub and Ang, 2008). The Delphi method affords researchers an opportunity to directly engage practitioners on timely issues and pressing concerns, while using a set of standardized statistical techniques to assess when consensus is achieved. This rigorous process ensures that research informs pressing needs facing practitioners and organizations — thus bringing academia and practice closer together. Given these benefits and considering the historical uses of the Delphi method as a tool for forecasting, issue analysis and framework specification, this method has the potential to make significant contributions to the broader AIS literature. As a discipline, AIS tends to first focus its research efforts on understanding technologies that may be leveraged within the accounting domain, and subsequently works to better understand those technologies that have a demonstrated effect on the reporting, governance, risk and control aspects of the organization. In the early stages of a technology's life cycle, the Delphi method might prove useful in identifying those strategic and emerging technologies that have potential for impacting the accounting and decision-making processes within organizations. This approach has been used to forecast the use and adoption of audit expert systems in accounting firms (Baldwin-Morgan, 1993; Greenstein and Hamilton, 1997), as well as to predict the benefits and challenges of XBRL adoption (Bonson et al., 2009; Baldwin and Trinkle, 2011). In taking a forecasting approach to new technologies, the method can also be useful for identifying potential pitfalls that require remediation. As a technology matures and is adopted within organizations, the focus of the method and its contribution to AIS research shifts from prediction to issue analysis and framework development. For those technologies pertinent to AIS research, the method can be useful in identifying and prioritizing issues related to the technology's integration into organizations, as well as its impacts on reporting, governance, risk and internal control. Once the technology is in place and widely accepted, the method can also be useful in developing and extending frameworks for research and practice. Please cite this article as: Worrell JL, et al, Exploring the use of the Delphi method in accounting information systems research, Int J Account Inf Syst (2012), doi:10.1016/j.accinf.2012.03.003

14


6. Limitations As with any method, the Delphi method is not without its limitations. The limitations of this method include administrative effort, panelist satisfaction, generalizability and validation. Each of these is discussed below. One of the more onerous limitations of the Delphi method is the amount of administrative effort required to effectively execute the study (Van de Ven and Delbecq, 1974; Delbecq et al., 1975; Malhotra et al., 1994). The Delphi method requires a significant time commitment from both the researcher and the expert panel. Most Delphi studies continue for three or more rounds (initial round for brainstorming, with subsequent rounds for narrowing and ranking items or factors), requiring the researcher to manage and modify subsequent surveys with feedback (justifications, mean ranks, degree of consensus) from the expert panel. On the panelist's side, while each survey may require a nominal amount of time to complete, the time between subsequent rounds is often 2 to 3 weeks, thereby requiring a significant commitment. On the researcher's side, each round requires synthesis of the previous round's feedback, calculation of factor mean ranks and consensus of panel, and construction of a new survey reflecting the previous round's insights. It is not unreasonable for the brainstorming and ranking portion of the Delphi study to last 2 to 3 months (three rounds at 2 to 3 weeks per round). Panel satisfaction represents another potential limitation of this method. Although one of the strengths of the Delphi method is its use of anonymous expert panels, this has been viewed by some as an inherent limitation of the method. In their review of nominal, Delphi and interacting group techniques, Van De Ven and Delbecq (1974) noted that panelists reported lower social–emotional rewards when participating in Delphi studies compared to nominal and interacting groups. The lack of face-to-face interaction left panelists feeling detached from the problem solving effort, with some feeling as though differences of opinion were not effectively resolved. Researchers must also be cognizant of panelist fatigue. As the study proceeds, with oftentimes weeks between rounds, panelists may become less committed to continuing their participation in the study. This presents a unique challenge to the researcher, as the question of how to report and manage attrition in data analysis and subsequent rounds becomes an issue. Generalizability and validation can also be a concern with this method. By definition, the Delphi method utilizes a non-representative sample of experts to opine on complex, multi-disciplinary problems. Generalizing the opinions and estimations of a non-representative group to a larger population can be problematic at best. The researcher should consider, however, that one of the strengths of the method is that it leverages the knowledge of experts in a specific domain. These experts are viewed to have insights above and beyond a representative group, suggesting the results obtained from such a panel may produce fruitful benefits for research and practice. Validation of the outcome can be problematic as well, especially if the purpose of the Delphi study was to forecast future events. In the short-term, assessing the accuracy of these predictions is difficult at best. However, studies have shown that the Delphi method has been effective at predicting the adoption of emerging technologies and industry trends (Cuhls, 2001; McCubbrey and Taylor, 2005). When reporting on forecasting studies, caution should be taken to highlight the composition of the panel clearly such that readers can assess the validity of the panel's qualifications. Taken collectively, researchers must be sure to address each potential limitation through the initial stages of the Delphi design process (i.e., initial stages of research question alignment, panel, and design considerations). These decisions must be thoughtfully considered and justified within the methodology section of a Delphi study to ensure a valid Delphi study was conducted. 7. Conclusions The purpose of this manuscript was to highlight the advantages and disadvantages of the Delphi method and discuss its potential value to AIS scholars. In reviewing its use in AIS and MIS literatures, we outlined the history of the Delphi method as well as research opportunities for AIS research. We then provided an illustrative example of the Delphi method and subsequently offered guidelines on how to design and administer a Delphi study. As with all research methods, the Delphi method has strengths and weaknesses, and is more appropriate in some settings than others. The challenge is to effectively align phenomenon, research question and method so that the researcher and study can make a Please cite this article as: Worrell JL, et al, Exploring the use of the Delphi method in accounting information systems research, Int J Account Inf Syst (2012), doi:10.1016/j.accinf.2012.03.003


15

contribution to the broader literature. It is our hope that more AIS researchers will view the Delphi method favorably and add it to their toolkit.

References Akkermans H, Bogerd P, Vos B. Virtuous and vicious cycles on the road towards international supply chain management. Int J Oper Prod Manage 1999;19:565–81. Akkermans HA, Bogerd P, Yücesan E, van Wassenhove LN. The impact of ERP on supply chain management: exploratory findings from a European Delphi study. Eur J Oper Res 2003;146:284. Anderson J, Rungtusanatham M, Schroeder R. A theory of quality management underlying the Deming management method. Acad Manage Rev 1994;19:472–509. Baldwin-Morgan A. The impact of expert system audit tools on auditing firms in the year 2001: a Delphi investigation. J Inf Syst 1993;7:16–34. Baldwin A, Trinkle B. The impact of XBRL: a Delphi investigation. Int J Digit Acc Res 2011;11:1-24. Bassellier G, Benbasat I. Business competence of information technology professionals: conceptual development and influence on IT-business partnerships. MIS Q 2004;28:673–94. Bassellier G, Reich B, Benbasat I. Information technology competence of business managers: a definition and research model. J Manage Info Syst 2001;17:159–82. Benbasat I, Zmud R. Empirical research in information systems: the practice of relevance. MIS Q 1999;23:3-16. Best J. An experiment in Delphi estimation in marketing decision making. J Mark Res 1974;11:448–52. Boje D, Murnighan J. Group confidence pressures in iterative decisions. Manage Sci 1982;28:1187–96. Bonson E, Cortijo V, Excobar T. A Delphi investigation to explain the voluntary adoption of XBXRL. Int J Digit Acc Res 2009;9:193–205. Brancheau J, Janz B, Wetherbe J. Key issues in information systems management: 1994–1995 SIM Delphi results. MIS Q 1996;20: 225–42. Brancheau JC, Wetherbe JC. Key issues in information systems management. MIS Q 1987;11:22. Brockhoff K. The performance of forecasting groups in computer dialogue and face to face discussions. In: Linstone H, Turoff M, editors. The Delphi method: techniques and applications. London: Addison-Wesley; 1975. Creswell J. Research design: qualitative, quantitative, and mixed method approaches. Thousand Oaks: SAGE Publications; 2003. Cuhls K. Foresight with Delphi studies in Japan. Technol Anal Strateg Manage 2001;13:555–69. Dalkey N, Helmer O. An experimental application of the Delphi method to the use of experts. Manage Sci 1963;9:458–67. Daniel E, White A. The future of inter-organisational system linkages: findings of an international Delphi study. Eur J Info Syst 2005;14:188–203. De Haes S, Van Grembergen W. An exploratory study into IT governance implementations and its impact on business/IT alignment. Info Syst Manage 2009;26:123–37. Delbecq A, VanddeVen A, Gustafson D. Group techniques for program planning: a guide to nominal group and Delphi processes. Glenview: Scott, Foresman; 1975. Doke ER, Swanson NE. Decision variables for selecting prototyping in information systems development: a Delphi study of MIS managers. Inf Manage 1995;29:173–82. Dubois A, Gadde L-E. Systematic combining: an abductive approach to case research. J Bus Res 2002;55:553–60. Fomin V, Pedersen MK, de Vries H. Open standards and government policy: results of a Delphi study. Commun AIS 2008;22:459–84. Glass R. Rigor vs. relevance: a practitioner's eye view of an explosion of IS opinions. Commun AIS 2001;6. Greenstein MM, Hamilton D. Critical factors to consider in the development of an audit client engagement decision expert support system: a Delphi study of big six practicing auditors. Intelligent Systems in Accounting Finance and Management 1997;6: 215–34. Gulati R. Tent poles, tribalism, and boundary spanning: the rigor-relevance debate in management research. Acad Manage J 2007;50: 775–82. Hollander M, Wolfe D. Nonparametric statistical methods. New York: Wiley; 1999. Holsapple C, Joshi K. Organizational knowledge resources. Decis Support Syst 2001;31:39–50. Holsapple CW, Joshi KD. An investigation of factors that influence the management of knowledge in organizations. J Strateg Info Syst 2000;9:235–61. Iden J, Langeland L. Setting the stage for a successful ITIL adoption: a Delphi study of IT experts in the Norwegian armed forces. Info Syst Manage 2010;27:103–12. Kasi V, Keil M, Mathiassen L, Pedersen K. The post mortem paradox: a Delphi study of IT specialist perceptions. Eur J Info Syst 2008;17:62–78. Keil M, Cule P, Lyytinen K, Schmidt R. A framework for identifying software project risks. Commun ACM 1998;41:76–83. Keil M, Tiwana A, Bush AA. Reconciling user and project manager perceptions of IT project risk: a Delphi study. Info Syst J 2002;12: 103–19. Kendall M, Babington Smith B. The problem of m rankings. Ann Math Stat 1939;10:275–87. Lampe J, Sutton S. Evaluating the work of internal audit: a comparison of standards and empirical evidence. Acc Bus Res 1994;24: 335–48. Lee A. Rigor and relevance in MIS research: beyond the approach of positivism alone. MIS Q 1999;23:29–34. Linstone HA, Turoff M. The Delphi method: techniques and applications. Addison-Wesley Publishing Co; 1975. Liu S, Zhang J, Keil M, Chen T. Comparing senior executive and project manager perceptions of IT project risk: a Chinese Delphi study. Info Syst J 2010;20:319–55. Loch K, Carr H, Warkentin M. Threats to information systems: today's reality, yesterday's understanding. MIS Q 1992;16:275–85. Malhotra M, Steele D, Grover V. Important strategic and tactical manufacturing issues in the 1990s. Decis Sci 1994;25:189–214. McCubbrey D. Disintermediation and reintermediation in the U.S. air travel distribution industry: a Delphi study. Commun AIS 1999;1.


16


McCubbrey D, Taylor R. Disintermediation and reintermediation in the U.S. air travel distribution industry: a Delphi reprise. Commun AIS 2005;15:464–77. McFadzeen E, Ezingeard J-N, Birchall D. Information assurance and corporate strategy: a Delphi study of choices, challenges, and developments for the future. Info Syst Manage 2011;28:102–29. Meredith J, Raturi A, Amoako-Gyampah K, Kaplan K. Alternative research paradigms in operations. J Oper Manage 1989;8:297–327. Nakatsu R, Iacovou C. A comparative study of important risk factors involved in offshore and domestic outsourcing of software development projects: a two-panel Delphi study. Inf Manage 2009;46:57–68. Neely A. Production/operations management: research process and content during the 1980s. Int J Oper Prod Manage 1993;13:5-18. Nevo D, Chan Y. A Delphi study of knowledge management systems: scope and requirements. Inf Manage 2007;44:583–97. Nicolaou A. Editorial: a dialectic on methodologies in AIS research. Int J Acc Info Syst 2011;12:81–3. Okoli C, Pawlowski S. The Delphi method as a research tool: an example, design considerations and applications. Inf Manage 2004;42:15–29. Rainer R, Snyder C, Carr H. Risk analysis for information technology. J Manage Info Syst 1991;8:129–47. Rowe G, Wright G. The Delphi technique as a forecasting tool: issues and analysis. Int J Forecasting 1999;15:353–75. Schmidt R. Managing Delphi surveys using nonparametric statistical techniques. Decis Sci 1997;28:763–74. Schmidt R, Lyytinen K, Keil M, Cule P. Identifying software project risks: an international Delphi study. J Manage Info Syst 2001;17: 5-36. Sherer S, Alter S. Information system risks and risk factors: are they mostly about information systems? Commun AIS 2004;14: 29–64. Straub D, Ang S. Editor's comments: readability and the relevance versus rigor debate. MIS Q 2008;32. Straub D, Welke R. Coping with systems risk: security planning models for management decision making. MIS Q 1998;22:441–69. Sutton S. Toward an understanding of the factors affecting the quality of the audit process. Decis Sci 1993;24:88-104. Turoff M, Hiltz R. Computer based Delphi processes. In: Adler M, Ziglio E, editors. Gazing into the oracle: the Delphi method and its applications to social policy and public health. London: Kingsley Publishers; 1995. p. 55–88. Van de Ven A, Delbecq A. The effectiveness of nominal, Delphi, and interacting group decision making processes. Acad Manage J 1974;17:605–21. Worrell J, Wasko M, Johnston A. Social network analysis in accounting information systems research. Int J Acc Info Syst 2012;13.


Exploring the use of the Delphi method in accounting information ...

Exploring the use of the Delphi method in accounting information ...

Suggest Documents

The Delphi Method

The Delphi Method

The Delphi Method for Graduate Research - Journal of Information ...

Delphi method

Exploring Information Quality in Accounting ... - IBIMA Publishing

Evaluating the Intention to use Accounting Information

The use of accounting information systems in analytical procedures for ...

The use of Delphi method in the process of building a tourism

The use of Delphi method in the process of building a tourism ...

The Use of Accounting Information System for the ...

the use of the delphi approach in vet research

Is the information fit for use? Exploring teachers perceived information ...

Delphi Method Variants in Information Systems Research: Taxonomy

Exploring the development of the accounting

Accounting Craftspeople versus Accounting Seers: Exploring the ...

Using and Reporting the Delphi Method for

The Delphi Method: Techniques and Applications

The Application of a Combined Delphi-AHP Method in Maritime ...

An Experimental Application of the Delphi Method to the Use of ...

Exploring the Use of Target-Language Information ... - Semantic Scholar

Exploring the Use of Online Corporate Reporting Information

exploring the use of online corporate sustainability information - Core

Using Item Bank Information Exploring the Use of Item Bank ...

Exploring the Use of Online Corporate Sustainability Information (PDF ...