Evaluation of Mobile Ordering Post-Implementation Alfa Ryano Yohannis1, Alexander Waworuntu2 Faculty of Computer Science and Communication Science KALBIS Institute Jakarta, Indonesia 1
[email protected],
[email protected] Abstract—Many ERP system projects are not evaluated properly after their implementation, including the implementation of promising mobile technologies to improve business performance of companies. Mobile ordering system (MOS) is a system that harnesses the mobility of mobile devices that facilitates salespeople to create sales quotations at any time and any place, enabling them to create orders immediately at customers’ locations without the need to return to the home office. This research tries to evaluate MOS post-implementation by comparing sales performance before and after MOS implementation. Since the results are at the high-level abstraction that is prone to inaccuracy, fine-grained evaluation is conducted by comparing the performance of salespeople equipped with MOS and the performance of traditional salespeople. The evaluation indicates that mobile ordering systems have no significant contribution to improve sales performance. Ironically, salespeople tend to merely migrate from traditional, paper-based selling to just selling using mobile devices without enhancing their sales performance. Thus, the more realistic way to measure MOS contribution is by measuring to what extend MOS improves business process efficiency. It is found that MOS can reduce significantly the average time required for processing a sales quotation document to become a sales order document. Keywords—mobile ordering implementation; ERP; sales;
I.
system;
evaluation;
post-
INTRODUCTION
Mobile technology advances rapidly in the recent decade and affect many aspects of person's life, such as social networks [1], education [2], health [3], and including business [4], transforming the way of business delivered to customers [5]. Mobile applications now have been part of Enterprise Resource Planning (ERP) suites [6] providing stakeholders access to their business information at any time and from anywhere. Several works have been conducted to study the unification of mobile technologies in business applications [7][8][9], but it is rarely found works that evaluate mobile business applications as part of ERP systems, particularly after their implementation. This work aims to answer questions does mobile ordering system bring improvement on sales performance to a company that implement it? If yes, to what extend? If no, why? Moreover, what is its contribution? In order to answer such questions, mobile ordering system post-implementation is evaluated by comparing sales performance before and after implementation. Since the comparison only exhibits only high-level abstraction result, it is still not clear how mobile ordering system affects the
This work is supported by PT Sterling Tulus Cemerlang
company's business performance. Therefore, to obtain finegrained results, the performance of salespeople equipped with mobile ordering system is contrasted to the performance of traditional salespeople. This paper is structured as follows. First, in order to have illustration about what mobile ordering is, a brief description about mobile ordering system and the business model of the company implemented mobile ordering are presented. Then, the underlying sales business process, particularly the change of sales business process, prior and post mobile ordering implementation, is described. Following, the architecture of mobile ordering is explained. This work then puts much discussion about measuring mobile ordering post implementation and what lessons can be obtained from the evaluation. After that, conclusion is derived and future works are presented. II.
MOBILE ORDERING SYSTEM AND ITS ARCHITECTURE
Mobile ordering system (MOS) is a system that harnesses the mobility of mobile devices to equip a salesperson with a tool that facilitates him to create sales quotations at any time and from anywhere. Salespeople are enabled to create quotations at customers’ locations and the quotations will be transferred to an ERP system hosted on a server via HTTP(S) protocol. The architecture of MOS is displayed in Fig. 1. Server Master Data Customer 1
Salesman Mobile A Device A
Master Data
Customer 2
Customer 3
Sales Quotations
Salesman Mobile B Device B
Mobile Ordering Server Application Master Data Internet/ HTTP(S) Connection Sales Quotations
Database Engine
ERP Server Application
Sales Quotations
Figure 1. The architecture of mobile ordering system (MOS).
In order for MOS being applicable, a client application has to be installed on salesperson’s mobile device. Next, the salesperson is required to login to the application. He cannot login if he and his device are not registered on the server. Thus, an administrator needs to register him and his device first. After login for the first time, the salesperson must download all the necessary master data, e.g. customers, items, prices, etc., for taking orders. The client application is equipped with a synchronization feature in order to update its master data later.
At customer locations, the salesperson creates sales quotations, picking the right customer identity and items ordered by the customers. Moreover, he can upload the sales quotations at that very moment in order to be handled immediately by sales admins. If there is no internet connection, all created sales quotations are saved locally inside the mobile device ready for uploading at another time. The sales quotations are send to the server in the form of XML(s). After uploading the sales quotations through HTTP (S) connection, sales quotations are then received by the mobile ordering server application, transformed and saved to the database by using a format that is recognized by the ERP application as sales quotation objects. Finally, sales admins browse the uploaded sales quotation documents and decide whether to proceed them as sales order documents or not. III.
BUSINESS PROCESS CHANGE
MOS discussed in this work has been employed by a distribution company that sells products to small, mid, and large retailers. The company has many salespeople that operate by visiting various customers each day, marketing new products and taking orders based on customers’ requests. Since most of its customers are mid-to-low companies that run their business in traditional fashion, the condition demands the salespeople to visit customers actively from shop to shop offering goods and creating paper-based sales order documents. After collecting sales orders, the salespeople will return to their physical company and hand over the sales order documents to sales admins who will input the sales orders into an ERP system ready for further processes. Since the implementation of MOS, the company business process has changed, particularly in the initial phase of sales ordering process. Previously, salespeople needed to deliver all the paper-based sales order documents physically to their company location to be processed further by sales admins. Currently, salespeople can take orders at customer location and immediately send the orders as sales quotations to the ERP system. Finally, sales admins will be regularly notified when the sales quotations arrived at the ERP system and then use the sales quotations as the basis for creating new sales orders. There are two reasons why this business process change caused by implementing MOS is considered will increase sales rate. First, salespeople do not have to deal with physical sales order documents, orders are digitally managed on mobile devices, which will cut the cost of administrative materials. Second, created orders can be sent to the home office at that very moment, which then can be immediately followed by sales admins without having to wait the salesperson arrived at home office. This will reduce the time of the selling process, cutting down the time consumed from customer requests to sales order creation at the ERP system. Dash MOS really increase the sales level of the company? This question is discussed in the next sections.
IV.
MEASURING MOBILE ORDERING POSTIMPLEMENTATION
The evaluation of information technology (IT) system postimplementation has been closely reviewed by Al-Yaseen et al [10]. They stated that the success of an IT system implementation could be measured by comparing ‘prior operational use’ to ‘post-implementation’ of the IT system, resulting temporal view of the IT system implementation. If a measured dimension representing a success in its significant increase is inclined after the implementation, then it is likely the IT system implementation has met the company expectation. This temporal view of evaluation of IT implementation is adopted as the approach to evaluate MOS. Selling through salespeople supported by sales technology has been examined by Hunter and Perreault. They suggest there are factors that influence the performance of salespeople supported by sales technology: the sales ability of the salespeople themselves (experience, planning, and adaptive behaviors), technology support provided by their company, and customer approval as the external factor. The sales technology contributes on increasing salespeople’s information effectiveness, which improves their planning and adaptive behaviors that will lead to the improvement of performance with customers. By employing sales technology, salespeople will improve their internal role performance [11]. Both performance with customer and internal role of performance of salespeople are expected will increase sales. Since the increase of sales is the end goal of this MOS implementation, establishing an increase in sales performance as a state that represents success of MOS implementation is plausible. Therefore, sales performance is determined as the measured dimension, prior and after MOS implementation. There are three types of measurement that are conducted to measure the sales performance. First, the efficiency of MOS is measured by comparing total values of sales quotation and total values of sales order generated from MOS. Since in business process sales quotation is followed by sales order, the efficiency of conversion from sales quotation to sales order can be measured. Second, sales performance is measured at the company level, obtaining three instances of sales performance: prior implementation sales (PIS), after implementation sales (AIS), and sales generated from MOS (MS). AIS consists of MS and sales from other selling methods. Third, sales performance is measured at the salesperson level, providing fine-grained explanation does MOS really improve salespeople’s selling performance. The performance of salesperson equipped with MOS is compared to the performance of traditional salespeople, prior and after MOS implementation. All data used in these three measurements are obtained from production database by using queries that are specifically designed for each measurement. Since sales order documents have fields referring to its preceding sales quotations and vice versa, by using join operation sales order documents with their sales quotation documents can be traced, identified, and yielded, enabling the first measurement possible to be performed. The sales order documents also have fields that contain information related to items sold in those documents
V.
Millions
(including prices, quantities, and total values), their salespeople, the date and time of when those documents were created, and information whether the documents were generated from mobile ordering or via other methods. The availability of these informations enables the creation of queries, using join, filter, grouping, and sub querying operations, to obtain data for the second and the third measurements. The results obtained from the measurements are discussed in the next section.
Millions
T otal Values per Week (USD)
1.24
1.20
0.80 0.60 0.40
In this section, the result of MOS post-implementation evaluation is presented and discussed. The data provided in this work have been transformed in order to conceal their actual values for privacy matter, but still equivalent for understanding the real condition.
0.20 Mobile Ordering SQ
Mobile Ordering SO
Figure 3. Mobile ordering sales quotation (USD) vs mobile ordering sales order (USD) 1 year after implementation.
B. Sales Performance at the Company Level Generally, the comparison of sales performance before and after MOS implementation can be performed by measuring the total values of sales prior and after implementation. Hence, the data provided only available since 19 weeks before the implementation, measurement is conducted 19 weeks prior and 19 weeks after then implementation in order to have a fair result. Fig. 4 exhibits temporal, week-by-week view of total values of sales orders per week and total values of sales orders generated from MOS. The figure shows MOS only has a somewhat contribution since the first time its implementation. Nevertheless, there is a positive trend that MOS will contribute more in the future. Millions
In order to have move subtle comparison between sales quotation and sales order, both of them are compared with their total values in a year. Fig.3 exhibits the efficiency is very high, with 1.28 million USD of sales quotations is converted to 1.24 million USD of sales orders resulting an efficiency of 97%. Only 3% sales quotations that are not converted into sales orders. This condition may be interpreted that the business process between sales quotation generation and sales order creation runs as expected, leave only a small of sales quotations that don’t proceed into sales orders.
1.28
1.00
RESULT AND DISCUSSION
A. Sales Quotation vs Sales Order Since MOS transforms the sales business process of the company implementing it as stated in section three, this section explain the efficiency of business process, which proceeds sales quotations into sales orders. Fig. 2 exhibits a temporal, week-by-week view of the total values of sales quotations and sales orders over 1-year MOS implementation. The graph shows no extreme differences between the total values of sales quotations and sales orders week by week whilst indicating increasing sales trend over time.
1.40
1.00 0.80 0.60 0.40
0.10 0.09 0.08 0.07 0.06 0.05 0.04 0.03 0.02 0.01 -
0.20 -
-19 -17 -15 -13 -11 -9 -7 -5 -3 -1 2
4
6
8 10 12 14 16 18
19 weeks prior and after implementation Total SO
1 4 7 10 13 16 19 22 25 28 31 35 38 41 44 47 50 Weeks after implementation Mobile Ordering SQ
Mobile Ordering SO
Figure 2. Mobile ordering sales quotation (USD) vs mobile ordering sales order (USD) 1 year after implementation.
Mobile Ordering SO
Figure 4. Week-by-week comparison between total sales values (USD) and mobile ordering sales value (USD) from 19 weeks prior to 19 weeks after implementation.
In order to have a better general comparison of sales performance, total values of sales orders are summarized into two groups, 19 weeks prior implementation and 19 weeks after implementation as displayed in Fig. 5. The company scored 7.65 million USD before implementing MOS and reached 8.01 million USD after MOS implementation, indicating an inclination of 0.36 million USD, equals to 4.5% rise, to their sales performance.
Millions
Suppose the total sales values of conventional sales methods before and after MOS implementation are assumed constant or only have slight difference and MOS’ total sales value contribution is excluded from the after-19-week sales, the total difference of sales value between before-19-week sales and after-19-week sales will only be 0.07 million USD. In other words, one might say that the total sales value increment is 0.36 million USD, which its 0.29 million USD, nearly 80% of the total sales increment, is contributed by MOS after the implementation, leaving the remaining percentage as the contribution of other methods of sales (Fig. 6). From this way of inference, the implementation of MOS can be said successful as MOS contributes dominantly on the increment of sales. 9.00 8.00 7.00 6.00 5.00 4.00 3.00 2.00 1.00 -
8.01
7.65
0.29
0 Prior 19 weeks Total SO
After 19 weeks
Mobile Ordering SO
Figure 5. Comparison of total sales values (USD) 19 weeks prior and 19 weeks after MOS implementation.
Others 20.76%
insured, as the underlying factors that cause total sales values of sales methods—whether MOS or the conventional ones— constant, increasing, or decreasing can’t be exactly comprehended at the company-level measurement. The use of high-level abstraction measurement in the evaluation of IT implementation is prone to be misinterpreted since the underlying factors that produce the results are difficult to be understood. Therefore, a lower-level measurement need to be conducted which is discussed in the next section. C. Sales Performance at the Salesperson Level A more detailed examination to what extend MOS increasing sales performance is conducted by measuring the sales performance of salespeople. Since the actors that use MOS in daily basis during selling activities are the salespeople, measuring their sales performance is perceived suitable to explain the influence of MOS on sales performance. Sales performance is quantified by measuring the average number of generated sales order documents per week and the average of total sales values per week. Fig. 7 depicts the sales performance of MOS-Equipped salespeople based on the average number of generated sales order documents, grouped into 19-week prior implementation sales (PIS), 19-week after implementation sales (AIS), and MOS-based sales (MS)—also measured for 19 weeks after implementation. The figure shows a slight declination on the average number of created sales order documents per week, declining 1.75 docs/week, from 31.02 docs/week to 29.27 docs/week, after MOS implementation. Furthermore, MOSequipped salespeople contribute averagely 17.54 docs/week (59.92%) of the 29.27 docs/week average of generated sales orders per week made by them since MOS implementation. Average Number of Sales Order per Week
Affirming the small sales contribution of MOS displayed in Fig. 4, MOS only contributes 0.29 million USD or 3.6% of the total sales value 19 weeks after implementation. This small contribution of MOS indicates that traditional methods of selling still have a dominant role to gain revenue, which positioning MOS as a compliment.
35.00
31.02
25.00 17.54
20.00 15.00 10.00 5.00 Total Sales PIS
Mobile Ordering SO 79.24%
Mobile Ordering SO
Others
Figure 6. The contribution of mobile ordering sales 19 weeks after MOS implementation, if total sales values prior and after implementation are assumed having no significant difference.
However, the assumption of constant or slight difference total sales values of conventional sales methods can’t be
29.27
30.00
AIS
MS
Figure 7. Comparison of prior implementation sales (PIS), after implementation sales (AIS), and MOS sales on their average number of sales order documents per week for MOS-equipped salesmen.
In accordance to Fig. 7, Fig. 8 also exhibits the sales performance of MOS-Equipped salespeople, but the measurement is on the average of total sales values per week, grouped into 19-week prior implementation sales (PIS), 19week after implementation sales (AIS), and MOS-based sales (MS). The figure displays a decrease in the average total sales value, dropping 0.80 thousand USD/week, from 4.92 thousand USD/week to 4.12 thousand USD/week, after MOS implementation. Moreover, MOS-equipped salespeople attain
T housands
Average T otal Saes Value per Week (USD)
6.00
4.92
5.00
4.12
4.00 2.49
3.00 2.00 1.00 Total Sales PIS
AIS
MS
Figure 8. Comparison of prior implementation sales (PIS), after implementation sales (AIS), and MOS sales on their average (USD) of sales total values per week for MOS-equipped salesmen.
Average Number of Sales Order per Week (USD)
Moreover, Fig. 10 shows the sales performance of traditional salespeople based on the average of total sales values per week, grouped into 19-week prior implementation sales (PIS) and 19-week after implementation sales (AIS). The figure depicts slight declination on the average total sales value, decreasing 0.62 thousand USD/week, from 6.34 thousand USD/week to 5.72 thousand USD/week after MOS implementation. 30.00 25.00
24.35
26.31
20.00 15.00 10.00 5.00 Total Sales PIS
AIS
Figure 9. Comparison of prior implementation sales (PIS) and after implementation sales (AIS) on average number of sales order documents per week for traditional salesmen.
Thousands
In order to have a comprehensive understanding about the sales performance of the whole salespeople, the sales performance of traditional salespeople is also measured. Fig. 9 displays the sales performance of traditional salespeople based on the average number of generated sales order documents, grouped into 19-week prior implementation sales (PIS) and 19week after implementation sales (AIS). The figure exhibits slight increase on the average number of created sales order documents per week, rising 1.96 docs/week, from 24.35 docs/week to 26.31 docs/week after MOS implementation.
Both measurements on the numbers of documents (Fig. 7) and measurement on the sales values (Fig. 8) present decreasing patterns which indicate that generally the performance of MOS-equipped salespeople is not improved after the implementation of MOS. This result is quite surprising since the contribution of MOS exceeds 50% on both measurements, numbers of documents and sales values, after MOS implementation. One explanation proposed is that paperbased selling to just selling using mobile devices without improving their sales performance. Average Total Saes Value per Week (USD)
averagely 2.49 thousand USD/week (60.44%) of the 4.12 thousand USD/week average of total sales value per week made by them since MOS implementation.
7.00
6.34 5.72
6.00 5.00 4.00 3.00 2.00 1.00 Total Sales
PIS AIS Figure 10. Comparison of prior implementation sales (PIS) and after implementation sales (AIS) on their average (USD) of sales total values per week for traditional salesmen.
The measurement on sales performance of traditional salespeople (Fig. 9 and Fig. 10) provides a context of what is happening on the salespeople generally. The average of total sales values per week of traditional salespeople is also declining (Fig 9), together with the declining performance of MOS-equipped salespeople (Fig. 7 and Fig. 8), opening the possibility to deduce that the sales performance of salespeople is generally declining, even though the average number of sales order documents generated by traditional salespeople is increasing, as the increment is relatively small, only 8.05% (1.96 docs/week) of prior MOS implementation sales (Fig. 8). An explanatory assumption can be given to answer why sales performance at the company-level measurement is generally increased (Fig. 5) whereas sales of the salespeople is dropping is that there are other sales methods that increase sales performance rather than salespeople do. For example, customers request sales orders directly to office without salesperson mediation. To examine this assumption, total sales value resulted from other methods besides salespeople are compared with 19weeks prior and 19-weeks post MOS implementation. From the production data provided, it is found total sales value resulted from other methods increases by 186 thousand USD (5.5%), from 3.376 million USD to 3.562 million USD, prior to after MOS implementation. The number of sales order documents is also increasing by 178 documents (5.3%), from 3,358 to 3,536 documents, indicating that the increase of sales performance at the company level is not caused by the increasing performance of salespeople, but by the inclination of sales performance via other methods of sales. Thus, it can be concluded that MOS
does not meet its expectation on sales performance improvement. Therefore, a more pragmatic, realistic measurement is required rather than using sales performance as the dimension for judging the success of MOS implementation. Based on the work of Delone, task productivity can be used as one of dimensions to measure the success of information systems, which is the number of user output per unit of time [12]. However, instead of using task productivity, the average time consumed for a sales quotation converted to a sales order is measured and then contrasted to a standard value. This measurement can be performed since both sales quotation and sales order documents have timestamps that inform when those documents are created and the sales order document has a reference pointing to its base sales quotation. The company that implements MOS has a key performance standard, which enforce that a sales quotation must be converted to sales order no later than a day or 24 hours. Thus, from the measurement, it is found that 0.602 days or 14.44 hours is the average time consumed for a sales quotation converted to a sales order, which is less than 24 hours. Thus, it can be concluded MOS contributes on task efficiency by reducing the time consumed for processing one document type to other document types. VI.
CONCLUSIONS AND FUTURE WORKS
In conclusion, the evaluation conducted by this work indicates that mobile ordering systems still have no significant contribution to improve sales performance. Ironically, salespeople tend to merely migrate from traditional, paperbased selling to just selling using mobile devices. As a business-supporting system, the implementation of MOS does not bring direct improvement in sales profit. This finding can be taken into account for companies to consider wisely before implementing business-supporting technologies as one of cutting-edge strategies. For the future, there are three lessons that are noteworthy for researchers, consultants, or companies in evaluating an IT system, particularly for a business-supporting system, after its implementation. First, avoid choosing high-level abstraction of measurement for the reason that it can lead to misinterpretation as the underlying factors that produce the results are ill defined; there many factors that might affect the results. Second, finegrained measurement is more worthy to be conducted since it measures direct and immediate effects of IT implementation, such as measuring the business process efficiency (section V. A). Finally, the finding of this work can also act as an example that investing an IT system does not always bring direct, immediate business advantages to the company. It might have contributed but limited to internal process improvement or intangible benefits that are not directly connected to company profits. Thus, the next works will be related to measuring to what extends MOS contributes to internal process improvement or intangible benefits. ACKNOWLEDGMENT Thank you to PT Sterling Tulus Cemerlang for the opportunity to conduct this work.
REFERENCES [1]
L. Humphreys. “Mobile social networks and urban public space,” New Media & Society 12, no. 5, 2010, pp. 763-778. [2] R. S. Shariffudin, C. H. Julia-Guan, T. Dayang, N. Mislan, and M. F. Lee. “Mobile learning environments for diverse learners in higher education,” International Journal of Future Computer and Communication 1, No. 1, 2012, pp. 32-35. [3] D.H. Lee, A. Rabbi, J. Choi, and R. Fazel-Rezai. “Development of a mobile phone based e-health monitoring application,” Development 3, No. 3 , 2012. [4] H. Mohelská, “Mobile technologies and their use in a company,” in Applied economics, business and development: proceedings of the world multiconference, WSEAS/IASME international conference, 2010, pp. 141-146. [5] D. Lopez, J. A. M. Davila, and C. de Pablos. “Business model transformation in the mobile Industry: co-creating value with customers,” Transformations in Business and Ecomomics, Vol. 11, No. 2 (26), 2012, pp.134-148. [6] R. D. P. Pavin, and A. Z. Klein. “Mobile ERP systems adoption: case studies in Brazil,” CONF-IRM 2013 Proceedings, Paper 33, 2013. [7] Helin, et al. “Context-aware business application service co-ordination in mobile computing environments,” in AAMAS05 workshop on Ambient Intelligence-Agents for Ubiquitous Computing, Utrecht, 2005. [8] P. Alahuhta, J. Ahola, and H. Hakala. “Mobilizing business applications,” Technology review 167, No. 2005, 2005. [9] J. Morony. “Viability of developing cross-platform mobile business applications using a HTML5 Mobile Framework,” PhD dissertation., Flinders University, 2013. [10] H. Al-Yaseen, T. Eldabi, R. J. Paul, and R. El-Haddadeh. “Postimplementation evaluation of IT systems: A close review of practice,” in Evaluating information systems, Z. Irani and P. Love, 2008, pp. 134. [11] G. K. Hunter and W.D. Perreault Jr. “Sales technology orientation, information effectiveness, and sales performance.” Journal of Personal Selling and Sales Management 26.2, 2006, pp. 95-113. [12] W. H. Delone. “The DeLone and McLean model of information systems success: a ten-year update.” Journal of management information systems, 19(4), 2003, pp. 9-30.
