IMPROVING DECISION QUALITY: THE ROLE OF BUSINESS ...

 

IMPROVING DECISION QUALITY: THE ROLE OF BUSINESS INTELLIGENCE ABSTRACT This exploratory study provides a validated, parsimonious research model of antecedents of the perception of the quality of decisions made using business intelligence support. Findings provide insight into little investigated avenues such as the role of problem space complexity in perceived decision quality as well as indicate a more complex interplay among the antecedents of decision quality than heretofore examined. For example, results suggest that there may be a tipping point for which information quality and use of the system support higher perceived decision quality. In addition, these findings provide a direction for future research to generate deeper, more meaningful contributions in our collective understanding of how BI serves to improve the quality of decision making. Keywords: business intelligence, decision quality, problem space complexity, information quality, level of BI use INTRODUCTION Business intelligence (BI) provides decision makers with data, information, or knowledge to address decisions about problems specific to the individual decision maker’s needs, and that can be ‘rolled up’ to support broader organizational level decision making [11]. BI systems are primarily put in place to improve the quality of decisions and provide timely solutions to problems ranging from very structured to highly unstructured ones [13]. In today’s world of big data, in-memory databases and pervasive analytics, companies are increasingly grappling with the best ways to leverage data to make better decisions [28]. The expectation is that managerial experience enhanced with BI tools leads to better decision making [54]. Evidence, however, suggests that BI does not consistently live up to those expectations [63] [58]. Technologies continually evolve to allow companies to store volumes and varieties of data unheard of just a few years ago, yet companies often do so without a good plan for how to leverage this data for better decision making [28]. Research posits a variety of reasons for BI failures [62] [44], yet the focus is largely either on factors intrinsic to the BI, i.e., technical and data related issues, or on factors surrounding the organization in which BI operates, i.e., organizational readiness or alignment of BI with organizational goals [62] [45] [42]. BI has the potential to improve decision making, yet empirical research about BI success has largely overlooked the quality of decisions made using BI [11]. We argue that examining perceived decision quality in the BI context will help to bridge the gap in our understanding. The purpose of this paper, therefore, is to explore and examine the quality of decisions made in the context of BI. The research question it addresses is: what are the key antecedents of perceived decision quality for users of business intelligence systems?

LITERATURE REVIEW A key goal of management support systems is to improve the effectiveness of decision making [52]. The measurement of the extent to which this effectiveness is achieved is often operationalized though surrogates such as usage of information systems or user information satisfaction [22]. The difficulty in establishing direct measures of decision quality lies largely in 1   

 

the complexity of decision processes [57] [46] [32]. It is difficult to determine whether the BI actually improves decision quality. An organization may achieve high returns on investment from the implementation of a system, yet still be a long way from the quality of decisions that it could achieve under a different approach to managing or using the system [52]. We propose and test a model where perceived decision quality in the BI environment is impacted by several key factors. The organization of this section follows the relationships shown in the contextual model in Figure 1 which draws upon the theoretical model proposed by Clark [11] and his colleagues. Clark [11] provides a conceptual framework that integrates constructs identified across a large pool of research literature as key to the success of systems that support decision making, including BI. That model provides key relationships that interact through complex feedback mechanisms to produce behavior. Some constructs have received a great deal of research attention, while others such as decision quality have been largely ignored. Although Clark’s model applied a broad lens to MSS in general, we focus specifically on BI perceived decision quality and its antecedents. BI Related Factors   

Perceived Decision Quality

Level of BI use Problem Space Complexity Information quality

Figure 1: Contextual Model

PERCEIVED DECISION QUALITY Decision quality is a function of effectiveness and efficiency in the process of decisionmaking [11]. Decision making has been couched in terms of decision outcomes [23] [16], problem solving performance [60], expectancy of success [38], information processing performance [19], and decision maker risk preferences [30]. Others consider decision making in terms of how decisions are made and structured [12] [38]. Although these theories and perspectives encompass a variety of thought and depth of understanding, all use some definition of decision or decision maker performance as an indicator of decision quality. Decision quality outcomes are often measured using perceived decision maker satisfaction with the outcome as a surrogate for decision quality [19] [60] [36] [31]. We follow this approach and conceptualize decision quality as the decision maker’s perception of the outcome resulting from the decision making process. LEVEL OF BUSINESS INTELLIGENCE USE BI systems are primarily put in place to improve the quality of decisions and provide timely solutions to problems ranging from very structured to highly unstructured ones [64]. Although BI may also be used for other purposes such as to expand the knowledge of markets and learn how they function or to create new areas where new decisions might be made, improving decision quality is the purpose on which we focus in this study. It, therefore, is expected that the level of BI use, defined as the extent to which users employ and rely on the BI to make decisions [10], will result in a higher quality of decisions. Yet, the level of usage that occurs immediately upon rollout is often insufficient to achieve expected organizational benefits [10] [18] and BI use 2   

 

in many organizations has not yet reached the level of maturity that enable innovative and game changing decision making [28] [27]. As BI becomes more pervasive in an organization, users engage with a greater variety of applications across a broader scope of functionality. In today’s world of data that are characterized by ever greater volume, velocity, and variety, many decision makers in organizations are using and relying on BI more than ever before [40] [51]. Based on these discussions, we hypothesize the following: H1: The greater the level of BI use in the decision making process, the higher the perceived quality of the decision. PROBLEM SPACE COMPLEXITY The problem space is the context of the problem or situation about which a decision is made [11]. Problem space complexity can be defined by a variety of factors including the number of variables involved in a problem and the interaction among those factors, although contextual variables such as the time available when making decisions, and the tools available to solve the problem also influence problem space complexity [61] [11] [59] [47]. We define problem space complexity in terms of the number and variety of variables involved in the decision and interactions among them [37]. The problem space within which decisions are made is becoming increasingly complex [51]. Complex decisions require more information and more intellectual effort for analysing it [20] [9]. BI is used to support decision making about dynamic and complex problems where decision makers grapple with factors outside their control and increasingly rely on soft or unstructured data [1]. Therefore, the effectiveness of BI is often tied to how well it supports high complexity decisions [24]. Yet, there is evidence that organizations do not adequately understand the role of problem space complexity in the decisions for which BI is used [28], and that this is one reason that BI maturity is not progressing as quickly as it might [27]. A better understanding of problem space complexity and its role in influencing the quality of decisions is one key to more effectively leveraging BI. The direction of the relationship between problem space complexity and perceived decision quality is not intuitive. On one hand, decision makers presented with more complex situations may be more likely to put more thought into the decision making process, which would result in objectively better decisions. On the other hand, because of the inherent complexity of the problem, decision makers may have lower expectations about decision outcomes, and thus may be more satisfied with their decisions after the fact [48]. Finally, because in highly complex situations it is more difficult to determine, even after the fact, what would have been a more appropriate decision, decision makers may be more likely to perceive their decisions are of higher quality. Drawing on these discussions, we hypothesize the following: H2: The higher the problem space complexity, the higher the perceived quality of the decision made. INFORMATION QUALITY Information quality is key in the functioning and output of information systems [15] [21] [33]. One of the most powerful aspects of a BI system is its ability to harness and synthesize vast quantities of data into information [61]. Therefore, the quality of information in the BI is critical to the quality of decisions made based on the output of the BI satisfaction [8] [5]. BI helps augment the decision making process by providing decision makers with information in a way that they are 3   

 

generally unable to obtain without BI. Thus, information quality is a necessary antecedent of high quality decisions. Based on these discussions, we hypothesize the following: H3: The higher the quality of information in BI, the higher the perceived quality of the decision made using BI. We argue, however, that the importance of information quality in contributing to better decision making is contingent on the type of problem to be solved. If decision outcomes are influenced by only a few factors as is the case with low problem space complexity decision environments, it is critical to have high quality information regarding all of those factors. If the number of factors that influence decision outcomes is large, and the interaction effects of those factors are difficult to predict (as is the case in high problem space complexity environment), having higher quality information about each of the factors is less likely to lead to a better decision because of the limited information processing capacity of the decision maker. Similarly, because in high problem space complexity environments decision outcomes depend on large number of variables, perceived decision quality is less likely to be negatively influenced by poor information quality if it is limited to one or two variables. Therefore, we hypothesize that: H4: Problem space complexity moderates the effect of information quality on perceived decision quality. The effect of information on perceived decision quality is also likely to be moderated by the level of BI use. The quality of information underlying BI is key to the ability of the BI to facilitate decision making [53]. However, because the level of BI use may differ across different decision makers, and vary from one decision to another, the extent to which high quality information contributes to making a high quality decision may depend on the degree to which the BI system is used for that decision. Following this logic we hypothesize that: H5: The level of BI use moderates the effect of information quality on perceived decision quality. The research model is provided in Figure 2. Level of BI Use Problem Space Complexity Information Quality

H1 H2 H4

H5

H3

Figure 2: Research Model

4   

Perceived Decision Quality

 

METHODOLOGY For this study, the research design is a field study in which data were collected through a web-based survey. The constructs were operationalized and measurement items were developed based on a review of the literature. The items for measuring decision quality were developed and contextualized from existing literature [49] [14] [6] [55]. The problem space complexity items were developed based on the description of problem space complexity [11] [37]. The information quality items were developed by drawing on the instrument developed by Lee and his colleagues [39]. Much research about information quality focuses largely on intrinsic or contextual dimensions (e.g., accuracy, completeness). There are other dimensions of the construct, however [36] [65]. These include representational (e.g., understandability) and accessibility (e.g., ease of use) [36]. We draw on the measures in [46] that tap the latter two dimensions because they may be more quickly perceived by the decision maker and therefore be more influential on their perception of the quality of decisions made using the information. The items to measure the level of BI use were adapted from another prior study [29]. A 5 point Likert scale was used for all the items. The target population for this study includes decision makers who use a BI solution to support their decision making. Data were collected from 61 BI users across a range of industries and organizations within the United States. Several criteria were used to obtain participants that could be expected to have sufficient knowledge about how BI impacts their decision making. One criterion was that respondents have experience both with making decisions for which the BI is used (as opposed to routine execution of decisions made by others) and with using BI. Over onethird had worked at least 5 years in their area of decision making responsibility, and approximately 20% had at least 5 years of experience using BI in their organizations (Table 1). Over 25% had used BI prior to joining their current organization. The data in Table 1 indicates that we captured a range of BI and decision making experience.

Table 1: Decision Making and BI Experience Number of years making the types of decisions they % of are now using BI for total 1 year 2 years 3 years 4 years 5 years or more No response

23.3 14.7 13.3 11.5 36.0 1.6

Number of years using BI at their company 1 year 2 years 3 years 4 years

1.6 1.6 9.8 4.9

5   

% of those who responded to the question 23.3 15.0 13.4 11.7 36.6

4.6 4.6 22.2 13.6

 

5 years or more No response

18.9 63.9

25.0

Used BI prior to this organization Yes No No response

26.2 14.8 59.0

64.0 36.0

Another criterion was that they represent decision making at different organizational levels. Most who responded to this question make strategic decisions, followed by operational decisions, and tactical level decisions (Table 2). In addition, the respondents are well educated, with over one-half indicating at least 5 years of higher education, and over 20% had taken a class in BI. Table 2: Level of Decisions Made, Job Titles, and Education Level of Decisions % of total % of those Made who responded Operational 11.5 28.0 Tactical 6.6 16.0 Strategic 23.0 56.0 No response 59.0 Years of Education 3 to 4 years of college 5 to 7 years of college More than 7 years of college No response Taken a BI Class Yes No No response

6.5 42.6 11.3

10.3 66.6 20.7

1.6

21.3 42.6 36.1

33.33 66.67

Respondents represent a range of industries, with healthcare being the largest group, followed by finance, education, telecommunications, and consulting (Table 3).

6   

 

Table 3: Industry and BI Software Vendor Industry % of total % of those who responded Healthcare 9.84 24.0 Finance 6.56 16.0 Education 6.56 16.0 Telecommunications 4.92 12.0 Consulting 4.92 12.0 Manufacturing/Industrial 3.28 12.0 Other 4.92 12.0 No response 59.02 BI Vendor Business Objects Google Microsoft (only) Oracle (only) Pentaho SAP SAS McKesson Proprietary Multiple (including Microsoft, Oracle, etc.) Other No response

1.64 1.64 1.64 1.64 1.64 1.64 1.64 1.64 1.64 13.11

4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 32.0

9.84 59.02

24.0

Finally, a criterion was that a variety of BI vendors and software be represented. Table 3 shows that no single vendor dominated the group, and that a number of respondents used multiple vendors for their BI solutions. Thus, the 61 respondents represent a range of BI and decision making experience, across a variety of industries, and BI solutions. They also represent welleducated decision makers in a variety of positions across the organizational hierarchy. Email was sent to respondents that provided them with the URL for the on-line survey. Data were collected from two groups. For one group, the email was sent to graduate students at a large university in the U.S. Southwest who had recently used or were using BI in their corporate jobs. In the second group, the email was sent to other decision makers working in business organizations who were using BI in their jobs. Data were collected over a 6 week time period, with a follow-up email after 3 weeks. DATA ANALYSIS AND RESULTS Using the rule of thumb that there should be at least 10 times the number of observations than constructs, with no fewer than 50 observations [25], the minimum sample size for this study is 65. The sample size in this study meets that threshold with 60 respondents. The respondents consisted of 37 graduate students who were using or had used BI in corporate jobs and 28 other 7   

 

business BI users. The data sets from the two types of respondents were compared for differences in demographic data (age, gender, and education) as well as the independent, and dependent variables using t-tests and Chi-square tests to assess the feasibility of using this mixed set of respondents. The only significant difference between the two groups was for two of the independent variables, problem space complexity and user experience. We have also clean the data and replace some missing values with the average item value in order to avoid results alteration. The non-student respondents worked in greater problem space complexity and had greater experience with BI. This is not particularly surprising. Our model, however, posits questions about differences in problem space complexity and user experience. These differences, therefore, strengthen, rather than detract from the use of this mix of respondents. The other significant difference was in the age of respondents; the students were younger. However, because age is not a factor in our model, this is not deemed a substantive difference. Therefore, the responses for the two data sets were combined for subsequent analysis. The age and gender for the combined data set are provided in Table 4.

Age 20-25 26-31 32-37 38-43 44-49 50-55 56-61 No response

Table 4: Age and Gender of Respondents % of total 9.8 19.7 9.8 9.8 9.8 3.3 1.6 36.1

Gender Male Female No response

42.6 21.3 36.1

Exploratory factor analysis (EFA) was conducted in order to better understand the nature and structure of the variables used in testing the proposed model. EFA is appropriate to test whether theorized items converge together on the hypothesized number of factors and discriminate across multiple factors with minimal cross-loadings [25]. It is particularly useful for assessing newly developed measurement items and in situations where the items have not been used together before. The final solution was rotated using an Equimax method to facilitate the interpretation of the extracted factors. We have retained variables with loadings greater than 0.5 [25]. The results of the factor analysis are presented in Table 5.

8   

 

Table 5: Exploratory Factor Analysis Results

1 (PSC) .828

Component 2 3 4 (DQ) (IQ) (LBIU)

The outcome of the decision depends on the interaction of different factors The decision involve a large number of variables or elements .813 When making the decision I have to consider many different .804 factors The problems about which I make decisions are generally .773 complex The decision involve a high degree of interactions among the .740 variables or elements considered I believe I made a good decision .891 The decision that I made resulted in the desired outcome .868 I am satisfied with the outcomes of this decision .865 In retrospect I believe I made the right decision .844 The information my BI system provides is-Not overwhelming .789 The information my BI system provides is-Available when I .721 need it The information my BI system provides is-Easy to extract .720 The information my BI system provides is-Believable .701 The information my BI system provides is-Easy to interpret .675 The information my BI system provides is-Useable .644 I relied highly on BI functionality while making the decision .883 Using BI was critical in making the decision .846 I used various features of BI for making the decision .822 Cumulative Variance Explained (%) 19.56 38.62 57.58 73.20 Cronbach’s Alpha 0.890 0.913 0.868 0.888 Legend: PSC = Problem Space Complexity; DQ = Perceived decision quality; IQ = Information Quality; LBIU = Level of BI Use Discriminant validity was assessed examining the correlations among factors using Pearson’s correlation coefficient (Table 6). None of the correlations exceed the .50 threshold, which suggests that there is adequate discriminant validity of the measures [25] [35].

9   

 

Table 6: Discriminant Validity Assessment PSC IQ DQ LBIU ** Pearson Correlation 0.713 .347 .277* .475** PSC Sig. (2-tailed) .004 .024 .000 Pearson Correlation 0.656 .457** .526** IQ Sig. (2-tailed) .000 .000 Pearson Correlation 0.806 .198 DQ Sig. (2-tailed) .111 Pearson Correlation 0.815 LBIU Sig. (2-tailed) **. Correlation is significant at the 0.01 level (2-tailed). *. Correlation is significant at the 0.05 level (2-tailed).

Mean Std. Dev. 3.895

.676

3.704

.692

4.022

.678

3.484

1.052

Legend: PSC = Problem Space Complexity; DQ = Perceived decision quality; IQ = Information Quality; LBIU = Level of BI Use Hypotheses were tested using linear regression analysis using SPSS. The results of the regression analysis are shown in Tables 7a and 7b. Table 7a: ANOVA Results Model

Sum of Df Mean Square F Squares Regression 11.388 5 2.278 7.356 1 Residual 18.578 60 .310 Total 29.966 65 a. Dependent Variable: DQ b. Predictors: (Constant), LBIU_IQ, PSC, IQ, LBIU, PSC_IQ c. R-square = .38

Sig. .000b

t

.927 .454 .393 .299 .129 .107

.097 3.996 -3.255 3.402 -3.673 3.162

Sig. 95% Confidence Interval Lower Upper Bound Bound

Tolerance

.090 1.812 -1.279 1.018 -.475 .338

Std. Error

Observed Powera

Intercept PSC LBIU IQ PSC_IQ LBIU_IQ

B

Noncent. Parameter

Parameter

Partial Eta Squared

Table 7b: Regression Coefficients

VIF

.923 -1.764 .000 .905 .002 -2.065 .001 .420 .001 -.734 .002 .124

.000 .210 .150 .162 .184 .143

.097 3.996 3.255 3.402 3.673 3.162

.051 .976 .893 .917 .951 .875

.051 .028 .111 .022 .017

19.739 35.906 9.026 45.006 60.322

1.944 2.720 -.493 1.617 -.216 .552 10 

 

 

a. Computed using alpha = .05 b. Dependent Variable: DQ The results of the regression analysis suggest the significance of all the predictors included in the model. Also, the observed power and the confidence intervals for our estimated parameters support our findings. Reliability of these results was confirmed through a bootstrapping procedure in SPSS. The direct effect of the level of BI use on perceived decision quality was significant at α = 0.002, although in the opposite direction to the one hypothesized in the model. Thus Hypotheses 1 was contradicted. Problem space complexity has a significant positive effect of perceived decision quality at α = 0.001. Therefore, Hypothesis 2 is supported. Information quality has a significant direct positive effect on perceived decision quality at α = 0.001. This lends support to Hypothesis 3. Furthermore, both interaction terms were significant (H4 and H5). The negative interaction effect of information quality and problem space complexity on perceived decision quality was further examined by considering the effect of information quality on perceived decision quality at different levels of problems space complexity [3]. As evident from the difference in the slopes of regression lines in the interaction plot (see Figure 3a), the effect of information quality on decision quality is weaker for more complex problems, which lends support to Hypothesis 4. The positive interaction effect of information quality and the level of BI use on perceived decision quality suggest that the level of BI use has a stronger positive effect on perceived decision quality if the information quality is high, lending support to Hypothesis 5. This counters the direct negative effect of the level of BI use on decision quality, suggesting that the use of BI may only have a positive impact if the BI provides high quality information. The interaction plot in Figure 3b suggests that the effect of the level of BI use on perceived decision quality is negative for the low level of information quality, slightly positive for medium information quality, and positive for high information quality. The VIFs for some of the predictors were elevated due to the inclusion of interaction terms in the model, but were still lower than 100, which are considered acceptable for a model with interaction terms [13], although this should be interpreted in light of the caveat that it is difficult to establish a meaningful boundary to determine what should be considered high or low VIF values [4]. We also performed Ridge regression [26] [41] on the data set and the results were consistent with our initial outcomes. The summary of hypotheses testing is presented in Table 8.

11   

 

(a)

(b)

Figure 3: Interaction plots for moderation effects

Table 8: Results of hypotheses testing Hypothesis P-value H1: The greater the level of BI use in the decision making =0.002 process, the higher the perceived quality of the decision. H2: The higher the problem space complexity, the higher