Evaluating Visualisation Tools for Balanced Scorecards Brenda Scholtz
Clayton Burger
Malusi Gcakasi
Department of Computing Sciences Nelson Mandela Metropolitan University Port Elizabeth, South Africa
[email protected]
Department of Computing Sciences Nelson Mandela Metropolitan University Port Elizabeth, South Africa
[email protected]
Department of Computing Sciences Nelson Mandela Metropolitan University Port Elizabeth, South Africa
[email protected]
Abstract—The strength and flexibility of the Balanced Scorecard (BS) framework has made it immensely popular in a wide variety of industries. The BS combines an enterprise’s financial measures with non-financial, operational measures in order to visualise an enterprise’s strategic and operational goals and drive future financial success. Several vendors have developed software solutions to implement and support a BS. Software solutions range from well-known, pre-packaged spreadsheet applications such as Microsoft Excel, to lesserknown, custom made solutions specifically aligned to an enterprise’s requirements. A number of BS software solutions make use of visualisation techniques such as graphs and diagrams, including visual “strategy maps”. The use of visualisation techniques can provide several benefits to users. These benefits include allowing users to more easily comprehend large amounts of processed data, enabling users to perceive problems with data more easily and facilitating the understanding of large and small scale features of data with greater ease. Some software solutions do little more than provide window dressing for Microsoft Excel data. Others provide a wide array of features, making them complex, difficult to use and expensive. This paper identifies several criteria for evaluating a BS software solution, and reports on a usability study of two existing BS software solutions. Identified criteria are evaluated by means of a usability study. The usability study includes an analysis of the preferences of users regarding the various visualisation techniques provided by these tools. Keywords— Balanced Scorecard; Usability; Evaluation Criteria;
I.
INTRODUCTION
The Balanced Scorecard framework (BS), devised by Kaplan and Norton [4] in the early 1990s, has become enduringly popular as a performance management tool since its inception [1, 27]. The BS combines an enterprise’s financial measures with non-financial, operational measures to translate an enterprise’s vision into measurable strategic and operational goals and thus drive future financial success. The BS has been described as a strategic planning and management tool that presents managers with a combination of measures to evaluate how effectively an enterprise is achieving its strategic and visionary goals [16]. In other words, the BS provides top
managers with a tool by which they are able to measure, and thus improve, the performance of their enterprise [4]. The strength and flexibility of the BS framework has made it an immensely popular tool in many private, public and nonprofit enterprise’s [3]. Data collected by the Gartner group [1] suggests that more than half of the large enterprise’s in the US have already adopted the BS with others considering implementation of the BS as their performance management tool of choice [1, 4, 7, 19]. Similarly, a study of the use of the BS conducted in South Africa has also revealed that most South African enterprise’s perceive the BS as very valuable [17]. Enterprises are able to implement a simple BS using a variety of tools ranging from pen and paper to complex software tools [1]. The increasing role of Information Technology (IT) has not only made enterprise’s progressively more dependent on IT [2], but has opened the door for IT enterprise’s to develop BS software as a primary business offering [5]. The software developed by IT enterprises often plays a subsidiary role to enterprise strategy and enterprise architecture [30]. Top managers must, therefore, carefully select and align BS software with their respective enterprises. Similarly, as goals and strategies vary widely among enterprises, BS software often caters for requirements beyond the scope of the BS. The topic of evaluation criteria for BS software has not been deeply researched, with no explicit literature on this topic existing. This study provides a review of the literature of BSs (Section II) and provides some insight into aligning the BS with Enterprise Architecture (Section III). A small number of tools are provided which enable the visualisation of a BS (Section IV). An empirical study in the form of a usability of visualisation tools for BSs is reported on (Section V). The key contributions of this study are highlighted and provide some valuable insights to the management and research communities (Section VI). II.
THE BALANCED SCORECARD
The BS can be defined as a set of measures which can be used to give an enterprise’s top managers a fast, comprehensive view of their enterprise [4].
978-1-4673-6412-6/13/$31.00 ©2013 IEEE
The original BS proposed that enterprises choose a set of measures from four primary, high level groupings, or “perspectives”, to help design their own scorecards [1]. Traditionally, these are the Financial, Customer, Internal Process and the Learning and Growth perspectives (Fig. 1).
products must be made in order to meet financial objectives [7]. The learning and growth perspective thus defines how an enterprise must define new processes or improve on existing processes in order to achieve its goals [26]. D. The Learning and Growth Perspective The learning and growth perspective defines the basic components required by an enterprise to support a given strategy. Components include [3, 26]: The skills and capabilities required by employees; The technological infrastructure required by the enterprise; and The corporate climate needed to support the strategy. Kaplan [3] considers the learning and growth perspective to be the “foundation of any strategy.” From this, we may infer that an enterprise must be able to meet the goals set out in this perspective before it may actively pursue the goals laid out in the financial, customer and internal process perspectives respectively.
Fig. 1. Traditional Balanced Scorecard, using Four Primary Perspectives, adapted from Kaplan and Norton [4].
A. The Financial Perspective The Financial Perspective refers to that group of high level measures which indicate whether or not an enterprise’s strategy, implementation and execution are contributing to the enterprise’s financial success [4]. Increased adoption of the BS in the private, public and non-profit sectors has led to differences in how the financial perspective is implemented [3]. In profit-seeking private and public enterprise’s, the financial perspective may emphasise profit maximisation or the maximisation of shareholder wealth [3, 7]. Conversely, non-profit enterprise’s, which by definition do not have the maximisation of profit as a primary goal, may implement the financial perspective as a constraint, or may focus on goals such as increasing funding from external donors [6, 7]. B. The Customer Perspective The goal of customer satisfaction has been incorporated into many enterprise’s corporate missions’ [1, 7, 26]. In other words, how an enterprise performs from the perspective of its customers has become a priority to top management. The BS implements the Customer Perspective as the set of measures which indicate how accurately an enterprise’s performance is meeting customer needs in terms of time, quality, service and cost. C. The Internal Process Perspective Once the Customer and Financial Perspectives are clearly defined, an enterprise must determine which processes it must excel at in order to differentiate itself to customers and which
E. Perspectives and KPIs Kaplan and Norton [4] prescribe the four perspectives as a foundation upon which enterprises can build their own BSs. Enterprise’s build personalised BSs by adding their own strategic goals and measures to the existing perspectives prescribed by Kaplan and Norton [4, 7]. In order for measures to be effective, an enterprise must have clearly defined the target or objective it wishes to achieve. Targets defined by the enterprise should also be achievable given an enterprise’s context as defined by the learning and growth perspective. This study uses the term Key Performance Indicator (KPI) as an umbrella term encapsulating the measures and targets an enterprise wishes to achieve. In other words, KPIs are tools used to measure the progress of an enterprise’s goals over time. III.
ALIGNING THE BS TO THE ENTERPRISE
The four perspectives prescribed by the BS traditionally represent areas in which enterprise’s aim to distinguish themselves from their competition. To do this, top managers must carefully select goals and KPIs to align with the prescribed perspectives [4, 7]. As an enterprise’s measurement system strongly affects its behaviour [4], top managers should align measures to the enterprise’s existing goals and architecture [1]. A. Aligning the BS to Enterprise Architecture The ANSI/IEEE Std 1471-2000 defines architecture as the “fundamental organization of a system, embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution” [31]. Enterprise Architecture (EA) is described as the manner in which the components of an enterprise are fundamentally arranged and how they behave with regard to
each other and their outside environment. Top managers may choose to create EA models to describe the existing or desired components of their EA [30]. The complexity inherent in large EA models, however, usually implies a large number of components. As such, EA frameworks may choose to partition models into several layers to decrease the number of components per layer. According to Winter and Fischer [30], most architectures differentiate the following layers:
In spite of the benefits of visualisation, Microsoft Excel 1, a spreadsheet application which is primarily designed around text entry and not through visualisation artefacts is currently the most widely used tool to support BI and the BS [1, 9]. Excel offers numerous advantages for supporting the BS. Advantages often cited by proponents of Excel include [9, 20]: The multitude of knowledgeable users and consultants available;
The business architecture layer, which represents the fundamental arrangement of components from the enterprise’s viewpoint;
Excel’s low cost as part of the Microsoft Office suite; The many add-ons and plugins available to extend the application;
The process architecture layer, which represents the fundamental arrangement of service development, creation and distribution within an enterprise context; The integration architecture layer, which represents the arrangement of information system components within the enterprise’s context; The software architecture layer, which represents the fundamental arrangement of software components; and
The wide variety of calculations and formulae available to the average user; and Excel’s ubiquity in the market. Motivations against using Excel as a large enterprise BS solution include [9]: The difficulty in distributing, managing and controlling Excel applications when scaling to a large number of users;
The technology architecture layer, which represents the fundamental arrangement of computing and telecommunications hardware and networks. The abovementioned layers correspond strongly to the perspectives prescribed by the BS. The purpose of the technology architecture layer, for example, ties in directly with that of the learning and growth perspective since one of the components is the technological infrastructure [3, 26] Similarly aligning an enterprise’s EA layers to the perspectives of the BS, based on the purpose of the layers and perspectives, would allow an enterprise to measure the effectiveness of its EA model using the BS. Top managers are, thus, able to form a link between their EA and the performance of the enterprise. Furthermore, top managers are able to verify whether or not their enterprise has the correct EA elements in place to support a given strategy. IV.
BALANCED SCORECARD TOOLS
“Strategy Maps” are described as a technique for providing visual representations of the links between different components of an enterprise’s strategy [7, 18, 19]. In this way the four perspectives prescribed by Kaplan and Norton can provide a visual framework through which enterprises may define their corporate strategy. Ware [8] asserts that the benefits of using visualisation artefacts such as graphs or images include the ability to comprehend large amounts of data, the ability to identify errors in data and the ability to understand large-scale and small scale features of data. Kaplan and Norton [4] formulated the BS with the intent of giving an enterprise’s top managers a fast and comprehensive view of their enterprise. From this, we may infer that the benefits derived from using visualisation techniques such as graphs or images are directly aligned with the goals of the BS.
Excel also lacks the ability to track the origin of data, when, and who, made which changes and what business rules lie behind the formula’s in use; and A lack of referential integrity requires that Excel data, which may possibly become corrupted or contain other data anomalies, be assumed to be correct at face value. Though the disadvantages of Excel are widely known [1, 9], it continues to dominate the BS software field. As most BS systems make use of static data imported into Excel worksheets, however, drilling-down to view fine grained data or rolling up to view higher level data, also known as “cascading” down or up, requires the implementation of a BI solution. An internet search on BS software, at the time of this publication, will easily yield over thirty different enterprise’s offering BS solutions, with each claiming unique advantages for their particular product [29]. The cost of even a simple BS application, however, can be in the region of $100 000. Choosing the wrong BS software to suit an enterprise’s needs may, therefore, result in a significant waste of time, energy and money [1]. Top managers must, thus, ensure that BS software selected, closely suits the goals of their own enterprises. V.
USABILITY EVALUATION DESIGN
The risks involved in selecting the wrong BS application requires enterprise’s to establish strict criteria for assessing and selecting BS applications before making large capital investments [1].
1
Microsoft Excel: http://office.microsoft.com/en-za/excel/
The primary research question to be answered in this paper is, thus: Which criteria of BS applications most highly influence a user’s perception of the usability of the application? The usability of a system has been defined by the International Enterprise for Standardisation (ISO) as the “extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” [14]. This standard identifies the three criteria of usability as the effectiveness, efficiency and satisfaction with which users within the enterprise are able to accomplish their task before selecting a final solution [14]. These criteria are also included in the model of ten usability criteria proposed by Seffah, Donayee, Kline and Padda [25]. Tullis and Albert [10] describe effectiveness as those aspects relating to whether or not a user is able to complete a task. Efficiency encapsulates those aspects which describe the amount of effort required for a user to complete a task. Finally, satisfaction is described as the degree to which a software tool delivers a positive user experience. Efficiency measures include factors such as time taken to become proficient in a task (learnability) and user’s perceptions of the utility of an application. Satisfaction measures include the user’s opinions of the product being visually appealing or easy to use. The ISO9241-11 standard [14] states that the usability of any tool or system has to be viewed in terms of the context in which it is used, and its appropriateness to that context. The context of use of software tools includes [14, 24]: The users of the tool; The Tasks to be performed using the tool; The equipment or tools (hardware, software and materials) used with the tool; and
when evaluating a BS, but forms the basis upon which a BS application is chosen for enterpriseal use. A. Scenario There are currently over thirty different enterprise’s offering BS software solutions [29]. As software implementations differ in strengths and weaknesses, it is imperative that enterprise’s select or build a BS that fits their own mission, vision, strategy and success plan [1]. BS software solutions differ in complexity and features. In order for a solution to support the BS framework, it must adhere to the criteria prescribed in the original BS framework devised by Kaplan and Norton [4]. This implies that a BS solution must contain, at least, the following functionality: High level “perspective” groupings, which may be further subdivided into goals and measures related to the strategic plan; Metrics or KPIs used to measure the progress of goals and perspectives over time; and “Stoplight” indicators, which may be used to give quick progress information at a glance. In selecting the BS applications to be evaluated in this study, a feature comparison was conducted of a small number of existing BS applications. Two representative applications, namely ArcPlan Enterprise Software2 by ArcPlan (henceforth known as System A) (Table I) (Fig. 2) and KPI Scorecard by MAUS3 (henceforth known as System B) (Table I) (Fig. 3) were selected. System A and System B were selected because they represent two extremes of the BS application spectrum. System A makes use of click and drag functions to implement tables, buttons, images, graphs, cells and other visual artefacts to create and assess KPIs. System A also supports a large number of features and capabilities (Table I), including:
The physical and social environments in which a product is used. Preece, Rogers and Sharp [24] regard usability as ensuring that interactive products are easy to learn, effective to use, and enjoyable from the user’s perspective. In order to evaluate the usability of a system thoroughly, it is necessary to gain qualitative as well as quantitative data [22]. User preference questionnaires using Likert scales can provide a specific metric for the related usability attribute or metric [23]. The use of both quantitative and qualitative data can support the triangulation of usability problems. According to Marr and Neely [1], the purpose of BS software is to integrate the BS into an enterprise. We may thus infer that the BS software system that best integrates with an enterprise, is most likely to be selected. This study thus poses the theory that a BS application is evaluated primarily based on the proposed efficiency and effectiveness benefits of the software, followed by user satisfaction. This implies that the efficiency benefits asserted by Ware [8] are not only valid
Create dashboards or simple and complex scorecards; The ability to enforce referential integrity through backend databases; The ability to scale to a large number of people within an enterprise; and The ability to cascade into multiple levels of the BS. As System A supports the basic functionality required by the BS framework, it may be regarded as a BS application. However, System A can also be integrated into a number of existing systems in order to extend its functionality and better integrate with an enterprise’s existing strategy and architecture. 2
ArcPlan Enterprise v7.5.1: http://www.arcplan.com/en/products/enterprise/free-trialedition/ 3 MAUS KPI Scorecard: http://www.maus.com/kpi_scorecard/
System B requires the user to manually enter data into a spreadsheet based program, much like Excel to create and assess KPIs. System B supports a smaller range of features than System A, including (Table I): The ability to create custom reports; The ability to create simple scorecards; and The ability to assess KPIs over time. System B functions similarly to Excel, making it familiar to most users. The results of the feature comparison conducted between System A and System B showed that the standard criteria, namely: that criteria required to qualify as a BS application, were met in the two systems. (Table I) contains extended functionality present in System A and System B over and above the standard criteria. A great deal of overlap exists in the functionality of System A and System B. The two systems are, therefore, differentiated by the method in which their features are implemented. TABLE I.
Fig. 2. Complete KPI view of fictional budgted and actual sales figures created with System A.
COMPARISON OF EXTENDED FUNCTIONALITY OF SYSTEM A AND SYSTEM B System
Extended Functionality Custom Development Graphical Interface
ArcPlan Enterprise Software System A
MAUS KPI Scorecard System B
Report
✓
✓
User
✓
Create Scorecards
✓
Personalize User Interfaces Integrate with existing IT environments Run Queries against Data Warehouses and Operational Databases
✓
Web Browser Interface
✓
Chart Animation
✓
Access vis Mobile Devices Automatic Text and Graph Generation Actual vs Budget Assessment Ability to Compare KPIs over time
✓
✓
Fig. 3. Complete KPI View of budgetd and actual figures for a budget created for a fictional enterprise created with System B.
B. Study Goals Participants were asked to create simple Financial Perspective KPIs using both systems. As stated in Section III, the primary research question to be answered in this paper is: Which criteria of BS applications most highly influence a user’s perception of the usability of the application?
✓ ✓
A number of usability questions were posed as metrics against which collected data could be evaluated. These included [10]: ✓
✓
✓
✓
✓
The aim of the usability evaluation was to evaluate the usability of System A and B, and compare the criteria of effectiveness, efficiency and user satisfaction for the two systems. The vast difference in which both systems accomplish the goal of creating and assessing KPIs allows for the efficiency, effectiveness and satisfaction of both systems to evaluated and compared more easily.
1) Were users able to accomplish the tasks assigned to them? 2) Did users perceive and recover from mistakes easily? 3) How long did it take for users to become proficient with the system? 4) How satisfied were users with the system? 5) Which system did users prefer? A laboratory study comparing System A and System B was conducted with tasks for both systems being completed in succession. It should be noted that while the question of how long it took users to accomplish tasks with the two systems was considered, the vast differences between System A and System B means that this metric would not provide much useful data. As such, metrics such as learnability and time
taken to perceive and recover from mistakes were used to measure efficiency.
Technology Agnostic: The SUS can be used for a wide spectrum of technologies, including web-sites, desktop systems, cell phones and more; and
C. Research Method Tullis and Albert [10] argue that a small sample size of three to four participants is sufficient to identify major usability issues. The goal of this study was to identify only the criteria that held the greatest influence when evaluating a BS application. As such, a small sample of seven participants took part in the usability evaluation. All participants were given the same verbal briefing prior to the working through the tasks. This consisted of instructions for the test and an explanation of KPIs. Purposive sampling [28] methods were used to select participants that would reflect typical users of this application. All participants, therefore, are graduates in the Computer Science field qualified as Business Analysts. Participants have varying levels of knowledge and experience regarding KPI tracking and KPI tracking software.
Score: The SUS provides a simple, single score from 0 to 100, making it easy to understand.
Participants each evaluated System A and System B in succession, recording their perceptions of the system immediately after completing all tasks. The order in which participants evaluated System A and System B was alternated in order to cancel out the learning effect associated with completing the tasks pertaining to one system before another. A positive System Usability Scale (SUS) questionnaire [13] was administered after the use of each system in order to capture participant’s perception about both systems. The questionnaire was further augmented by adding two openended questions to more adequately record each participant’s perception of the system, namely:
In a study conducted by Bangor, Kortum and Miller [12], the SUS was shown to be highly reliable and useful over a wide range of interface types. The SUS is traditionally composed of ten statements, five of which are positive and five of which are negative [12]. Each question in the SUS includes a five point scale which ranges from Strongly Disagree to Strongly Agree. Sauro and Lewis [13] argue that the advantages of alternating question items include: The ability to control acquiescent response bias: Or the tendency to agree with all questions or to indicate positive connotations; and Protection against serial extreme responders, or participants who provide all high or low ratings. The potential disadvantages of alternating questions include [13]: Misinterpretation: Participants may respond differently to negatively worded items; Mistake: Participants may simply forget to reverse their score without truly intending to respond differently to a negative statement; and Miscode: Researchers may forget to reverse the data when scoring, thus reporting incorrect data.
Which features(s) of this software were the most useful in doing these tasks?
Calculating the resultant SUS score for a given system is done in two steps, namely [13]:
Which feature(s) of the software did you like the least while performing these tasks?
Subtracting 1 from positive questions, namely Q1, Q3, Q5, Q7 and Q9. And subtracting the responses to the negative questions from 5. This places all values on a scale from 0 to 4, with 4 being the most positive response; and
As participant’s time and knowledge are scarce resources, the number chosen was limited to seven, based on the roles they’ve played in enterprises and knowledge they possess which would make them a fair representative of the typical users of such a system. Participants of this study ranged widely in age from between 20 to 60 years of age, with 71% of the participants fitting inside the 20 year 25 year range (n = 5) and 28% of the participants being over 30 years of age (n = 2). Additionally 85% participants were male (n = 6), with 14% being female (n = 1). D. Research Instruments The SUS [11] is one of the surveys that can be used to assess a variety of products and services [12]. Among the many advantages that make the SUS attractive, are [12]: Size: The SUS consists of only ten statements – making it relatively quick to complete; Price: The SUS is non-proprietary (free), making it a cost effective scoring tool;
Sum the scaled responses, and multiply this number 2.5 to convert the number to a value out of 100 instead a number out of 40. The above procedure was followed for this study, with each questionnaire yielding a final score out of 100. Final scores for each system were summed and the final number divided by the number of participants (n = 7) in order to yield a final, weighted score for each system. The use of an alternating SUS presents potential disadvantages. Researchers have, thus, suggested the use of a positively worded SUS questionnaire in order to counteract potential disadvantages of alternation while preserving the advantages of the SUS [13]. A “positive” SUS is identical to the traditional, alternating SUS, with the exception of having even items worded positively rather than negatively (Table II). According to Tullis and Albert [10], the SUS questionnaire yields only a single, final score for the usability of a system. A
study conducted by Sauro and Lewis [32] demonstrates, however, that SUS questions may be partitioned to yield more accurate data. Sauro and Lewis partition questions into two groups, namely usability and learnability. Questions 1, 2, 3, 5, 6, 7, 8 and 9 align strongly with one another and are called “Usability” questions. Questions 4 and 10 align strongly with one another and may be known as “Learnability” questions [33]. As “Learnability” encapsulates those questions which describe with the amount of effort used to complete a task, it may be used as a measure for efficiency [10]. As such, the questions encapsulated in the “Usability” section must describe efficiency and satisfaction. User satisfaction is additionally measured through feedback received from participants after the study. The data gathered in this way was used in conjunction with the modified, positive SUS questionnaire to capture user data. The questions found in the positive SUS questionnaire are as follows (Table II): TABLE II.
Question Number
SUS QUESTIONNAIRE USING POSITIVE ALTERNATING QUESTIONS [13]
Positive SUS Questions
Q1
I think that I would like to use this system frequently.
Q2
I found the system to be too simple.
Q3
I thought the system was easy to use.
Q4
I think that I could use the system without the support of a technical person.
Q5
I found the various functions in this system were well integrated.
Q6
I thought there was too much inconsistency in this system.
Q7
I would imagine that most people would learn to use this system very quickly.
Q8
I found the system very intuitive.
Q9
I felt very confident using the system.
Q10
I could use the system without having to learn anything new.
Sauro and Lewis [13] suggest that researchers interested in designing new questionnaires should avoid the inclusion of negative items. Users of the “positive” SUS, however, need not change the version provided by Sauro and Lewis [13] provided they use proper coding methods. The positive SUS questionnaire used in this study uses a 5-point Likert scale ascending from 1, or “Strongly Disagree”, to 5, or “Strongly Agree” It has been shown that the internal reliability of both the traditional, negative and the positive SUS questionnaires are high [13, 21] and nearly identical [13]. Research revealed that the traditional SUS questionnaire (with 51 cases) had a Cronbach’s alpha value of .91. Positively worded SUS questions (with 110 cases) had a Cronbach’s alpha value of .92.
E. Results Two methods were used in order to record performance data during the study, namely: During the test, software was used in order to record participant actions as well as the amount of time spent on each system; and After the test, the positive SUS questionnaire was administered to record participant. The above approach yielded data needed to answer the research questions posed in (Section V-B). Participant data gathered during the test was able to ascertain that participants were able to complete the tasks assigned for each system (metric 1, Section V-B). It was notable, however, that all participants (n = 7) needed significantly more assistance in using System A than using System B. Though participants were eventually able to complete tasks, most found the controls employed by System A to be counter intuitive, and complex. Similarly, video data reviewed after participants had completed the evaluation found that participants took longer to recover from perceived mistakes with System A than with System B. The variety of features offered by System A make the system more flexible but also adds complexity [32]. From the above results, we may infer that the complexity inherent in System A thus makes it more difficult to use as well as more difficult to learn. Video data collected during the study was also used to evaluate metric 3. Data revealed that, similarly to metrics 1 and 2, users took longer to become proficient with System A rather than System B. One participant noted that System B’s similarity to Excel meant that they were able to adopt the system quicker and learn to use it faster than System A. Participant feedback in the form of the positive SUS scores as well as the two additional questions were additionally used to ascertain answer to metrics 4 and 5. (Fig. 4) and (Fig. 5) represent coded responses based on a 5-point Likert scale for System A and System B.
Fig. 4. Total responses to positive SUS questionnaire for System A.
System A’s Click and Drag functionality and many features (Table I) clearly support the benefits of visualisation proposed by Ware [8]. The system failed, however, to rank highly in terms of usability and overall user satisfaction, as indicated by the low mean scores of Q1, Q4 and Q9 (Fig. 4). System A often required increased concentration from participants. The researchers also noted that participants often required additional assistance in navigating around unfamiliar functions. Several problems with the usability of System A were reported by participants. After the study, one participant noted that: “I liked the dragging of items from the column to the table least. It was not immediately apparent what the result of my action would be.” While another noted that: “The whole left bar and how to use those buttons was not intuitive. ” Interestingly, while System A relies heavily on Graphical User Interfaces for users to create and assess KPIs, it received less criticism with regard to aesthetic appeal than System B. 85% participants (n = 6), however, found System A complex to use. Similarly 71% (n = 5) of participants did not feel confident using the system. System B requires the user to manually enter data into a spreadsheet based program, much like Excel to create and assess KPIs. The simplified approach to capturing and assessing KPIs garnered positive responses, with 100% (n = 7) feeling that the system was easy to learn, as indicated by Q4 and Q10 (Fig. 5). Furthermore, 71% (n = 5) of participants felt confident after using the system.
One participant also mentioned that: “I had no real problems. Tooltips on menu items would have been useful. Also a wizard as a Tutorial would have been useful.” Though both systems are used to complete the same task, they do so in completely different ways. The prolific use of drawing and design tools evident in System A enable users to design more aesthetically pleasing BS perspectives (Fig. 2), but are more complex than simple systems such as System B (Fig. 3). It is interesting, however, that despite its relative lack of visualisation tools System B continued to score more favourably than System A in almost every category while receiving fewer complaints. The above is also reflected in the final scores of each system. System A earned a final weighted score of 50% while System B earned a final weighted score of 57%. This weighted score may be used as a fair indication of how satisfied users were with each system (question 4) and which system they most preferred (question 5 - System B). Though participant feedback varies greatly with the two systems, the final SUS scores differ by only 7 points. It may be argued that as the two systems operate very differently, the difference in overall scores reflects those features which users find most useful. Aesthetic appeal and factors of user satisfaction do play an important role in evaluating and assessing a BS application. However, factors of efficiency and effectiveness are considered with greater importance when selecting a system. VI.
CONCLUSION
The BS framework provides enterprise’s with a flexible foundation upon which to build and implement their strategy. Tools implemented by software vendors allow enterprise’s to automate the BS and make it an integral part of an enterprise [1]. Creating an effective BS for an enterprise and selecting the software tool to support the BS requires the enterprise to carefully evaluate their mission, vision and strategic goals and choose tools and measures which suit the enterprise. This study provided an empirical study of the usability of BS tools. The empirical study also aimed to answer the question of which criteria most highly influence a user’s perception of the usability of a BS application?
Fig. 5. Total responses to positive SUS questionnaire for System B.
In contrast to System A, System B was criticised, however, for its aesthetic appeal, with one participant noting that using a different interface design may have made using the application more pleasing.
Results indicated that effectiveness and efficiency metrics such as the ease with which users are able to complete a task hold the greatest influence when selecting a BS application. In other words, results indicate that the most highly valued criterion when evaluating a BS application is the ease with which a user is able to complete their tasks using a tool. This study has concluded that BS applications are also measured independently of features such the use of visualisation techniques or aesthetic features of user interfaces. The findings of this study validate the researchers’ original theory, that BS applications are evaluated primarily on effectiveness and efficiency criteria. Findings also contradict, however, researchers’ earlier assertion of the importance of visualisation artefacts such graphs and images. Results suggest that while Ware’s [8] benefits are valid and play a role in the evaluation
of a BS application, they play a subsidiary role to usability metrics such as ease of use. This study has provided an empirical evaluation of the criteria which affects the selection of BS tools. Future evaluations of BS criteria should consider the inclusion of criteria affecting the selection of strategic goals and measures in an enterprise BS. Additional research may also consider the effect EA plays on the selection of BS software. The inclusion of a “live” case study may further illuminate the differences between enterprise choices of BS applications. REFERENCES [1]
[2]
[3]
[4] [5]
[6]
[7]
[8] [9] [10]
[11] [12]
[13]
[14]
[15]
[16] [17]
[18]
A. Neely and B. Marr. "Automating the Balanced Scorecard - Selection Criteria to Identify appropriate Software Applications." Measuring Business Excellence, vol. 7, pp. 29-36, 2003. I. Velitchkov. “Integration of IT Strategy and Enterprise Architecture Models.” in Proc. of the 9th International Conference on Computer Systems and Technologies, 2008, pp. 1-6. R. Kaplan. “Strategic Performance Measurement and Management in Nonprofit Organizations.” Nonprofit Management and Leadership, vol. 11, 2001, pp. 353 – 370. R. Kaplan and D. Norton. “The Balanced Scorecard: Measures That Drive Performance.” Harvard Business Review, 1992, pp. 71-79. A. Neely. “The performance management revolution: why now and what next?” International Journal of Operations and Production Management, vol. 19, pp. 205-228, 1999. M. Martello, J.G. Watson and M.J. Fischer. “Implementing a Balanced Scorecard in a Not-For-Profit Organization.” Journal of Business & Economics Research, vol. 6, pp. 67-80, 2008. R.S. Kaplan and D.P. Norton. “Transforming the Balanced Scorecard from Performance Management to Strategic Management: Part 1.” Accounting Horizons, pp. 87-104, 2001. C. Ware. Information Visualization: Perception for Design. San Francisco, CA: Morgan Kaufmann Publishers, 2002, pp. 3-4. R. Person. Balanced Scorecards and Operational Dashboards with Microsoft Excel. Indianapolis, IN: Wiley Publishing, Inc, 2009. T. Tullis and B.Albert. Measuring The User Experience: Collecting, Analyzing and Presenting Usability Metrics. Burlington, MA: Morgan Kaufmann Publishers, 2008. J. Brooke. “SUS - A Quick and Dirty Usability Scale.” Internet: http://hell.meiert.org/core/pdf/sus.pdf, 2011 [May 25, 2013]. A. Bangor, P. Kortum and J. Miller. “Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale.” Journal of Usability Studies, vol. 4, pp. 114-123, 2009. J. Sauro and J. Lewis. “When Designing Usability Questionnaires, Does It Hurt to be Positive?” in Proc. of the ACM CHI Conference on Human Factors in Computing Systems, 2011, pp. 1-9. International Standards Organization (ISO). “Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability.” Internet: http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm? csnumber=16883, [Apr. 7, 2010]. J. Sauro. “10 things to know about completion rates.” Internet: http://www.measuringusability.com/blog/completion-rates.php, Sept. 6, 2011 [Feb. 3, 2011]. R.F. Smith. Business process management and the balanced scorecard, Hoboken, NJ: John Wiley & Sons, Inc, 2007. B. Eastes, B. Scholtz and A. Calitz. “A Balanced Scorecard for Sustainability Reporting.” in Proc. of the 6th International Business Conference, 2012, pp. 1 - 23. C.H. Speshock. Empowering Green Initiatives with IT, A Strategy and Implementation Guide Hoboken, NJ: John Wiley & Sons, Inc, 2010, pp. 69-74.
[19] R.S. Kaplan. “Conceptual Foundations of the Balanced Scorecard.” in Handbook of Management Accounting Research, vol. 3, C.S. Chapman, A.G. Hopwood and M.D. Shields, Eds. Elsevier, 2008, pp. 1253-1269. [20] R. Sherman. “Microsoft Excel: The king of BI.” Internet: http://searchcrm.techtarget.com/news/1081869/Microsoft-Excel-Theking-of-BI, Apr. 20, 2005 [Apr 3, 2013]. [21] T. Tullis and J. Stetson. “A Comparison of Questionnaires for Assessing Website Usability.” presented at the UPA 2004 Conference, Minneapolis, Minnesota, 2004. [22] X. Faulkner. Usability Engineering, New York, NY: Palgrave Macmillan Ltd., 2000. [23] B. Kirchenham and S.L. Pfleeger. “Software quality: The elusive target.” IEEE Software, vol. 13, pp. 12-21, 1996. [24] J. Preece, Y. Rogers and H. Sharp. Interaction design: Beyond human computer interaction. Chichester, WSX: John Wiley & Sons Ltd, 2011. [25] A. Seffah, M. Donyaee, R.B. Kline and H.K. Padda. “Usability measurement and metrics: A consolidated model.” Software Qual Journal, vol. 14, pp. 159-178, 2006. [26] N. Kumari. “Balanced Scorecard for Superior Organizational Performance.” European Journal of Business and Management, vol. 3, pp. 1-15, 2011. [27] J. Chen. “An Empirical Analysis on Performance of M&A of Chinese Internet Companies Based on Balanced Scorecard.” Advances in Applied Economics and Finance (AAEF), vol. 4, pp. 5, 2013. [28] C. Teddlie and F. Yu. “Mixed Methods Sampling: A Typology With Examples.” Journal of Mixed Methods Research, vol. 1, pp. 80-83, 2007. [29] 2GC Active Management, “Performance Management Software - List of current packages.” Internet: http://2gc.eu/resource_centre/software#, 2013, [Jul. 24, 2013]. [30] R. Winter and R. Fischer. “Essential Layers, Artifacts and Dependencies of Enterprise Architecture.” in Proc. of the 10th IEEE Intertational Enterprise Distributed Object Computing Conference, 2006, pp. 1-12. [31] ISO/IEC/IEEE. ISO/IEC/IEEE 42010:2011, IEEE Std 1471:2000. 2000. [32] B. Nuseibeh and S. Easterbrook. “Requirements Engineering: A Roadmap.” in Proc. Conference on The Future of Software Engineering, 2000, pp. 37-49. [33] J.R. Lewis and J. Sauro. “The Factor Structure of the System Usability Scale.” in Proc. 1st International Conference on Human Centered Design, 2009, pp. 94-103.
