A framework of combining casebased reasoning ... - Wiley Online Library

British Journal of Educational Technology doi:10.1111/bjet.12043

Vol 45 No 4 2014

595–605

A framework of combining case-based reasoning with a work breakdown structure for estimating the cost of online course production projects Wu He Wu He is an assistant professor of Information Technology at Old Dominion University, USA. His research interests include data mining, case-based reasoning, learning technologies and human information behavior. Address for correspondence: Dr Wu He, Department of Information Technology and Decision Sciences, Old Dominion University, Norfolk, VA 23529, USA. Email: [email protected]

Abstract Currently, a work breakdown structure (WBS) approach is used as the most common cost estimation approach for online course production projects. To improve the practice of cost estimation, this paper proposes a novel framework to estimate the cost for online course production projects using a case-based reasoning (CBR) technique and a WBS. A prototype was developed to illustrate the effectiveness of the framework. Preliminary results suggest that the combined approach of using CBR and WBS is preferred and enables project managers to generate reasonable estimates for online course production projects more efficiently.

Practitioner Notes What is already known about this topic • Currently, a work breakdown structure (WBS) approach is the most common cost estimation approach for online course production projects. • A WBS approach depends on the past experience of practitioners and requires the creation of an original estimate, which is time-consuming. What this paper adds • A novel framework of combining case-based reasoning with a WBS is proposed. • Grounded in the proposed framework, a prototype is implemented to help estimate the cost of developing online courses. Implications for practice and/or policy • The proposed framework can help practitioners to generate cost estimates more quickly and effectively. • The proposed framework is a step forward in improving the existing practice of cost estimation for online course production projects.

Introduction There are rising demands for accountability at all levels of higher education including the rapidly expanding online distance education sector (Bramble & Panda, 2008). A report by Allen and © 2013 British Educational Research Association

596

British Journal of Educational Technology

Vol 45 No 4 2014

Seaman (2010) shows that, in 2009, the number of American higher education students taking online courses was more than 5.6 million. To maintain the growing demand for online distance education, academic leaders need to decide how much money they should invest in technology, online course development, faculty professional development and other necessities of online distance education (Heineman, 2011). To meet external demands for accountability, more and more academic leaders are adopting a data-based approach in decision making (Coburn & Turner, 2011; Heineman, 2011; Little, 2012; Means, Padilla & Gallagher, 2010). Potential benefits of this data-based approach in education include increasing teacher and staff accountability (Means et al, 2010), monitoring and improving academic programs and student performance (Little, 2012; Means et al, 2010), enabling instructors to make more effective instructional decisions (Heineman, 2011), etc. On the other hand, the use of a data-based approach in education also has some limitations. For example, people may not have easy access to the data needed; the quality of some data may be low; staff may lack adequate skills or the knowledge to analyze, interpret and use the data (Coburn & Turner, 2011; Little, 2012; Marsh, Pane & Hamilton, 2006). To provide support for academic leaders to make the decision to invest in online course development, practitioners of online distance education need to provide high quality estimates for online course development projects. However, it is often difficult to estimate the costs for online course development projects because such projects involve many variables and involve a relatively complicated process. For example, an online course development project may use a variety of technologies such as Flash-based animation, videos, audios and simulations for developing different online course modules. Since, for their implementation, different technologies involve different complexities and interactivities, it is often hard to accurately estimate the time and costs needed for their implementation. In addition, developing an online course often requires user input and multiple rounds of revisions and refinement, so developers usually have a hard time estimating how much time they will need to spend for revision and refinement. As for the online course development process, many organizations follow a life cycle process that includes steps such as planning, instructional design, interface design, module and content development, integration, testing, revision and refinement, quality review and assurance, and maintenance (Abdous & He, 2008). Because of the complexity involved in online course development, techniques for estimating the costs of online course development projects still remain largely unexplored in practice. To improve the existing practice and to offer techniques for estimating the development cost of online courses, this paper proposes a novel framework for the cost estimation of online course production. Grounded in the proposed framework, a prototype is implemented to estimate costs by using case-based reasoning (CBR) technique and a work breakdown structure (WBS). The rest of the paper is organized as follows: the second section describes the background of cost estimation for online course production projects. The third section provides a brief literature review for two cost estimation techniques: CBR techniques and a WBS. The fourth section presents a cost estimation framework for online course development. The fifth section introduces a prototype developed based on the proposed framework. Suggestions for future research can be found in the sixth section. Background Estimating project cost is important for any organization that completes projects for clients (Kultur, Turhan & Bener, 2009). Cost estimates should be as accurate as possible in order to avoid either overestimation or underestimation (Gordon, He & Abdous, 2009). So far, researchers and practitioners have developed a number of approaches including Regression (eg, algorithmic models), Analogy (eg, CBR), Expert Judgment, Work Breakdown, Function Point, Simulation, Machine Learning, Neural Network and a Combination of Estimates for cost estimation in the © 2013 British Educational Research Association

Estimate the cost of online course production projects

597

software engineering field (Auer & Biffi, 2004; Duan & Xu, 2012; Duran, Rodriguez & Consalter, 2009; Jørgensen & Shepperd, 2007; Kultur et al, 2009; Srinivasan & Fisher, 1995). However, few studies have been conducted to explore the cost estimation issue in the online course development process. A thorough literature review conducted by the author shows that only the WBS approach has been discussed as a method for estimating costs and time in course development (B’ettcher, 2000; Clark, 2004; Gordon et al, 2009). There has been no discussion whatsoever of other methods or techniques for estimating the cost of online courses in the e-learning or distance education literature. The author feels that there is a need to bring other cost estimation techniques developed for and in use by various fields (IT, construction, marketing, etc) to the fields of e-learning and distance education. Online course production is a relatively complicated and costly activity. For example, Verizon Communications reports that it needs at least 40 hours and $15 000 to develop 1 hour of an e-learning course (George & Mcgee, 2003). To estimate the cost or efforts for online course development, practitioners mainly use a WBS approach. They divide an online course into multiple units and modules. Then they estimate the cost and time for each unit and module based on their past experience, personal intuition and judgment. However, the WBS approach depends on the past experience of practitioners and requires an estimate from scratch, which is time-consuming. Cost estimation is usually done in the planning phase. Estimating the cost needed for online course development is difficult and has always involved a degree of uncertainty. The cost associated with online course development is related to such factors as selected technologies, instructional design, project management, quality assurance, interaction models, course materials and equipment, type of training, experience and the skill levels of the staff (Chao, Saj & Tessier, 2006; Gordon et al, 2009; Rumble, 1998, 2001). A recent systematic review of software development conducted by Jørgensen and Shepperd (2007) identified that there is no single best method for cost estimation. As many industries have specific business needs and complicated requirements, general-purpose cost estimation systems do not always work well. For industries with specific needs, requirements and tasks, specific approaches and systems are generally preferred. In practice, developing online courses is different from creating software in many aspects. Online course developers often use existing software to create course materials such as graphics, audios and videos. In addition to the technical aspects, online course developers need to pay attention to pedagogical aspects such as course structure, instructional design, content engagement (quizzes, tests, etc), assessment, accessibility, standards and grading, etc. (Gordon et al, 2009). Thus, more research regarding the techniques for estimating the costs of online course development is needed. As a starting point, the author proposes a framework of using the CBR technique and the WBS to estimate the costs needed for online course development. Literature review WBS A WBS is a tool used to define, group and organize various components and elements of a project (Pritchard, 1999). In project management, a WBS can help project managers divide a large and complex project into smaller and manageable modules or components. A WBS reduces the complexity of cost estimation and makes the cost estimation relatively easier for project managers (Gordon et al, 2009). Typically project managers, technical and subject matter experts will work together to construct a WBS for the projects. Currently, the WBS is the dominating approach for estimating the costs of online course production (B’ettcher, 2000; Clark, 2004; Gordon et al, 2009). © 2013 British Educational Research Association

598


Vol 45 No 4 2014

Figure 1: An example of cost estimates for interface design category of an online course

Clark (2004) provided a downloadable Excel Spreadsheet Cost Estimator on his website to help with estimating costs and time for course development. This tool uses a structured template that includes a number of elements such as duration of courses (hours), difficulty level to design course, multimedia level, experience level of designers, number of instructors per hour of training, equipment, etc. The user can simply enter the data for each element and then the tool will automatically calculate the itemized hours, labor costs and grant total costs. Gordon et al (2009) developed a WBS named Asynchronous Cost Model (ACM) to help e-learning staff estimate the cost needed for online course development. The ACM model is composed of a number of specific modules such as instructional design, user interface design, audio, video and so on (Gordon et al, 2009). Figure 1 shows an example of cost estimates for interface design category of an online course. On the other hand, a WBS has several limitations. First, it requires extensive expertise and substantial efforts and time to create a quality WBS that is accurate, complete and logical. A WBS often includes a number of work elements and needs a lot of discussion and work to figure out how specific and detailed the final work elements will be. Second, a WBS needs to be updated to reflect the changes in a timely manner. Third, it requires extensive experience and time to gather appropriate input data for each of the work elements in the WBS in terms of cost estimation (Haugan, 2002). CBR CBR is a widely used artificial intelligence methodology for problem solving and learning (Kolonder, 1993). Humans often rely on their past experience to solve problems. When we encounter a problem, we often try to recall a previous problem that is similar to the current problem. If a past similar problem is found, we will reuse or adapt past experience and lessons to solve the current problem (Aamodt & Plaza, 1994; He, Wang, Means & Xu, 2009). Grounded on this type of memory-based problem-solving idea, CBR systems have been created to solve problems in many areas. Technically, people collect a lot of past experience cases and then store these past experience cases in a case base. When people face a new problem situation, they can search the case base to see if they could find similar cases to the current problem. If similar cases are found, people can either reuse or adapt the solutions to solve the current problem (He, Wang, Means & Xu, 2009; He & Xu, 2011; Xu, Liang & Gao, 2008; Xu, Wang, Bi & Yu, 2012). Typically, a fully developed CBR system includes four basic cycles: retrieve, reuse, revise and retain (Aamodt & Plaza, 1994). Figure 2 illustrates an example of the CBR cycles (Aamodt & Plaza, 1994). © 2013 British Educational Research Association


599

Figure 2: A CBR cycle (adapted from Aamodt & Plaza, 1994)

CBR has been widely used to estimate the cost of software project and construction project (An, Kim & Kang, 2007; Chou, 2009; Kolonder, 1993; Koo, Hong, Hyun & Koo, 2010; Li, Xie & Goh, 2009; Shepperd & Schofield, 1997). In essence, CBR is an approach of “estimation by analogy.” In recent years, researchers found that estimation by analogy can actually generate quite accurate estimates (Mendes, 2005; Shepperd & Schofield, 1997). CBR is especially suitable for cost estimation in complex domains (Bisio & Malabocchia, 1995; Duan, Street & Xu, 2011; He, Erdelez & Wang, 2010; Moore, Erdelez & He, 2006; Prietula, Vicinanza & Mukhopadhyay, 1996; Shepperd, Schofield & Kitchenham, 1996). A limitation with the use of CBR in cost estimation is that CBR requires a database of previous cases as the basis. CBR may yield inaccurate results if there are no sufficient cases in the case base (Ji, Hong & Hyun, 2010). Furthermore, if a problem is completely new, we may not be able to find similar cases from the case base. In this case, we will have to develop our own solution to solve a totally new problem. A framework of cost estimation for online course production projects Although CBR is a valuable cost estimation technique (Delany & Cunningham, 2000), a search of the literature shows that CBR has not been used to estimate the costs in online course development. Because of the complexity of online course development, the author feels that CBR can be effective in assisting managers with online course cost estimation. As the result, the author has extended a previously developed WBS-based prototype and has created a more advanced framework that integrates both CBR and a WBS. In the framework, the advantages of CBR and a WBS are combined to address their respective limitations and cost estimate challenges as a single technique often fails to address the challenge sufficiently. Figure 3 depicts the proposed integrated framework, which demonstrates the ways in which different technologies can be integrated to support online course cost estimation. Technically, a project manager will use a WBS-based cost estimation tool to fill out forms sequentially in order to estimate the costs of the development of an online course. Then the generated online course cost estimate will be reviewed by staff members for feedback and changes. After this, the cost estimate of the course will be saved into the case base database for sharing and later retrieval. As the case base grows, an e-learning project manager can search the case base to find similar cases based on course characteristics and then can adapt their cost estimates for a new online course. As far as the adaptation is concerned, the e-learning project manager may need to discuss these online course design and development decisions with other staff members and content experts (eg, faculty members) in order to © 2013 British Educational Research Association

600


Vol 45 No 4 2014

Staff (online course designers, developers, testers, etc) Retain

Review and Feedback

WBS-based Cost Estimation Tool for Online Courses

Online Course Cost Estimates

Review and Refine

Adapt

Case Library Powered by CBR Retrieve Similar Case

Update Cases as Needed Use E-learning Project Manager

E-learning Project Manager Send New Online Course Cost Estimation Request from Clients

Figure 3: An online course production cost estimation framework

determine how to adapt the cost estimates appropriately. For example, the e-learning project manager may need to figure out the balance between the quality and quantity of media selected for an online course by discussing it with faculty members. After the adaptation is done, the cost estimates of the new online courses can be further reviewed and retained into the case base as needed. Overall, the proposed framework can be a good tool to guide the further development of an online course cost estimation system (He, Xu, Means & Wang, 2009). In the proposed framework, a “case” is an estimate of an online course (either new or finished courses). A cost estimate for a new online course can be obtained by searching the case base for one or more similar completed online courses, each containing attributes or characteristics similar to the new online course. A challenging part of implementing the proposed framework is to collect previous cases for the case base. The author recommends interested organizations to check with their related personnel and to collect previous cost estimates manually from them. If such information is not available, organizations need to require project managers to record and document cost estimates right away so that they could build a case base at a later time. A web-based online course production cost estimation prototype Guided by the above framework, an online course cost estimation prototype was developed by the author using the CBR and work breakdown techniques. The web-based prototype was implemented by the author using PHP, JavaScript and MSSQL database. The CBR module was created by using a CBR generator developed by the University of Missouri (Wang, Moore, Wedman & © 2013 British Educational Research Association


601

Figure 4: Screenshot of a web-based cost estimate prototype

Figure 5: CBR search interface

Shyu, 2003). The CBR generator uses a near-neighbor algorithm (He et al, 2010) to match a query-case with existing cases in the database. The prototype provides two ways to estimate the cost: • Users can follow the instructions provided on the interface to select relevant modules and fill out information needed for the items that are appropriate for their own courses (Figure 4). The required information includes the number of hours needed and/or the quantity of certain materials. • The prototype provides users with the capabilities to search the case base and retrieve similar courses based on characteristics such as subject, course level, credits and estimated development time (Figure 5). Once cases are found (Figure 6), an experienced project manager can view the detailed cost estimates of each case and then make adaptation for the new online courses. When faced with a new course, the project manager just needs to adjust the raw numbers of the cost estimate model of a similar course and then the system can automatically generate an estimate. This CBR retrieval and adaptation approach expedites the cost estimate process by reusing previously recorded similar experience. A formative evaluation of the prototype indicates that, compared with the common WBS approach, the approach of combining CBR and WBS has a strong potential to make cost estimation work much easier for practitioners by helping them find similar estimations more efficiently and by helping them to make adjustments more quickly. Users who have tested the prototype indicate that the search feature of the system saves much time in generating an estimate for a new © 2013 British Educational Research Association

602


Vol 45 No 4 2014

Figure 6: A list of cases in search results

course, compared with starting the estimate from scratch. A project manager also indicates that this prototype provides a basis for clients to understand and discuss the cost and efforts involved in an online course production project. Preliminary feedback also indicates that the cost estimation results really depend on a database of similar historical projects and on the accuracy of the historical data. When past projects with similar characteristics were available, the prototype was able to successfully guide developers to produce realistic estimates for online courses. Even for a completely new course without similar courses in the database, the prototype can still help project managers to store the estimates and to make further adjustment when more information or feedback is obtained later. However, the proposed approach does have several limitations: (1) the proposed approach mainly focuses on the quantity portion of estimation (such as the number of hours needed for developing various course materials). The quality portion mainly relies on the past experience of the e-learning project managers to estimate how many hours are needed in total for individual components. This requirement could be challenging and could lead to an incorrect estimate if an e-learning project manager lacks sufficient experience in quality assurance. Integrating a detailed quality measurement rubric with the prototype can increase the accuracy of cost estimation; (2) to create a high quality cost estimate, the proposed approach expects that an e-learning project manager will have sufficient experience in online course development to be able to make reasonable estimates about the hours needed and the quantity of various course materials; and (3) successfully implementing or adapting the proposed approach is resource-intensive because previous cost-related cases need to be collected and stored systematically in order to build an effective cost estimation system. e-Learning organizations that are interested in the proposed approach are strongly recommended to follow knowledge management strategies (Abdous & He, 2008; He, Means & Lin, 2006; Ubon & Kimble, 2002; Zhang, Wang, Cao, Wang & Zhao, 2012) to capture, track, collect, store and administer their online course development-related information as much as possible for cost estimation and other purposes such as training, continuous improvement and quality assurance. Conclusion and future work Currently, a WBS approach is used as the most common cost estimation approach for online course production projects. Although several new approaches have been developed to estimate development costs in the software field, these new approaches have not been applied to estimate the costs for online course production. In this paper, the author presents a novel online cost estimation framework for course production that integrates WBS and CBR techniques. The proposed framework provides a valuable means to estimate the costs needed for an online course production project by using information from similar past online course development projects. © 2013 British Educational Research Association


603

A main limitation with the proposed cost estimation framework is that it requires substantial resources to collect previous cases for the case base and requires e-learning project managers to have sufficient experience in online course development. A prototype developed based on such a framework has been presented to illustrate the potential value of the framework. The present prototype is a work in progress; future development primarily involves conducting an ongoing formative evaluation of the prototype to improve the usability and functions of the system, collecting cost-related cases from multiple sources, and using the feedback received and the actual online course development costs to update the estimated costs and to lower the risk of underestimation or overestimation. More advanced functions, such as integration with Web 2.0 tools (He & Xu, 2011), business intelligence tools (Tan, Shen, Xu, Zhou & Li, 2008; Xu, Liu, Wang & Wang, 2009; Zeng, Li & Duan, 2012; Zeng et al, 2012) and software adjustments that automatically update the course costs to reflect the effects of inflation caused by salary increases and license fee increase, would also be beneficial to the prototype and will be further investigated in future research. References Aamodt, A. & Plaza, E. (1994). Case-based reasoning: foundational issues, methodological variations, and system approaches. In: AICOM, Vol. 7, 1. Abdous, M. & He, W. (2008). Streamlining online course development process by using project management tools. Quarterly Review of Distance Education, 9, 2, 181–188. Allen, I. E. & Seaman, J. (2010). Class differences: online education in the united states, 2010. Needham, MA: The Sloan Consortium. An, S., Kim, G. & Kang, K. (2007). A case-based reasoning cost estimating model using experience by analytic hierarchy process. Building and Environment, 42, 7, 2573–2579. Auer, M. & Biffi, S. (2004). Increasing the accuracy and reliability of analogy-based cost estimation with extensive project feature dimension weighting. Proceedings of Intl. Symp. Empirical Software Engg., 147–155. B’ettcher, J. V. (2000). How much does it cost to put a course online? It all depends. In M. J. Finkelstein, C. Frances, F. I. Jewett & B. W. Scholz (Eds), Dollars, distance, and online education: the new economics of college teaching and learning (pp. 172–197). Ph’enix, AZ: American Council on Education/Oryx Press. Bisio, R. & Malabocchia, F. (1995). Cost estimation of software projects through case-base reasoning. Proceedings of First International Conference on Case-Based Reasoning Research and Development, 11–22. Bramble, W. J. & Panda, S. (2008). Preface. In W. J. Bramble & S. Panda (Eds), Economics of distance and online learning (pp. ix–xii). New York, NY: Routledge. Chao, T., Saj, T. & Tessier, F. (2006). Establishing a quality review for online courses. Educause Quarterly, 29, 3, 32–39. Chou, J. S. (2009). Web-based CBR system applied to early cost budgeting for pavement maintenance project. Expert Systems with Applications, 36, 2, Part 2, Pages 2947–2960. Clark, D. R. (2004). Estimating costs and time in instructional design. Retrieved August 1, 2012 from http://www.nwlink.com/~donclark/hrd/costs.html Coburn, C. E. & Turner, E. O. (2011). Research on data use: a framework and analysis. Measurement: Interdisciplinary Research & Perspective, 9, 4, 173–206. Delany, S. J. & Cunningham, P. (2000). The application of case-based reasoning to early software project cost estimation and risk assessment. Technical report, Department of Computer Science, Trinity College Dublin, TDS-CS-2000-10. Duan, L., Street, W. N. & Xu, E. (2011). Healthcare information systems: data mining methods in the creation of a clinical recommender system. Enterprise Information Systems, 5, 2, 169–181. Duan, L. & Xu, L. (2012). Business intelligence for enterprise systems: a survey. IEEE Transactions on Industrial Informatics, 8, 3, 679–687. Duran, O., Rodriguez, N. & Consalter, L. A. (2009). Neural networks for cost estimation of shell and tube heat exchangers. Expert Systems with Applications, 36, 4, 7435–7440. George, T. & Mcgee, M. K. (2003). Educational advantage. Information Week, March 10, pp. 57–58. Gordon, S., He, W. & Abdous, M. (2009). Using a web-based system to estimate the cost of online course production. Journal of Distance Learning Administration, September, 2009. Haugan, G. T. (2002). Effective work breakdown structures. Vienna, Virginia: Management Concepts. He, W., Erdelez, S. & Wang, F. K. (2010). Examining a case-based reasoning system using mental models as a framework. International Journal of Learning Technology (IJLT), 5, 1, 63–79. © 2013 British Educational Research Association

604


Vol 45 No 4 2014

He, W., Means, T. & Lin, G. (2006). Tracking and managing field experiences in teacher development programs. International Journal of Technology in Teaching and Learning, 2, 2, 137–151. He, W., Wang, F., Means, T. & Xu, L. (2009). Insight into interface design of web-based case-based reasoning retrieval systems. Expert Systems with Applications, 36, 7280–7287. He, W. & Xu, L. (2011). Integrating both wikis and XML with case bases to facilitate case base development and maintenance. Expert System with Applications, 38, 7, 8632–8638. He, W., Xu, L., Means, T. & Wang, P. (2009). Integrating web 2.0 and the CBR cycle: a system approach. System Research & Behavior Science, 26, 6, 717–728. Heineman, W. (2011). The role of data in decision making about online distance education: a case study of three community colleges. Unpublished Doctoral Dissertation. University of Massachusetts Boston. Ji, C. Y., Hong, T. & Hyun, C. (2010). CBR Revision model for improving cost prediction accuracy in multifamily housing projects. Journal of Management in Engineering, 26, 4, 229–236. Jørgensen, M. & Shepperd, M. (2007). A systematic review of software development cost estimation studies. IEEE Transaction on Software Engineering, 33, 1, 33–53. Kolonder, J. (1993). Case-based reasoning. Mountain View, CA: Morgan Kaufmann. Koo, C., Hong, T., Hyun, C. & Koo, K. (2010). A CBR-based hybrid model for predicting a construction duration and cost based on project characteristics in multi-family housing projects. Canadian Journal of Civil Engineering, 37, 739–752. Kultur, Y., Turhan, B. & Bener, A. B. (2009). Ensemble of neural networks with associative memory (ENNA) for estimating software development costs. Knowledge-Based Systems, 22, 6, 395–402. Li, Y. F., Xie, M. & Goh, T. N. (2009). A study of mutual information based feature selection for case based reasoning in software cost estimation. Expert Systems with Applications, 36, 3, Part 2, 5921–5931. Little, J. W. (2012). Understanding data use practices among teachers: the contribution of micro-process studies. In C. E. Coburn & A. C. Bueschel (Eds), The practice of data use, a special issue for American Journal of Education Vol. 118, number 2 (pp. 143–166). Chicago, IL: University of Chicago Press. Marsh, J., Pane, J. & Hamilton, L. (2006). Making sense of data-driven decision making in education. Occasional paper. Santa Monica, CA: RAND Corporation. Means, B., Padilla, C. & Gallagher, L. (2010). Use of education data at the local level: from accountability to instructional improvement. Washington, DC: US Department of Education, Office of Planning, Evaluation, and Policy Development. Mendes, E. (2005). Exploring case-based reasoning for web hypermedia project cost estimation. International Journal Web Engineering Technology, 2, 1, 117–143. Moore, J., Erdelez, S. & He, W. (2006). Retrieval from a case-based reasoning database. Academic Exchange Quarterly, 10, 4, 65–68. Prietula, M., Vicinanza, S. & Mukhopadhyay, T. (1996). Software effort estimation with a case based reasoned. Journal of Experimental and Theoretical Artificial Intelligence, 8, 3–4, 341–363. Pritchard, C. L. (1999). Nuts and bolts series 1: how to build a work breakdown structure. Arlington, VA: ESI International. Rumble, G. (1998). The costs and economics of open and distance learning. London: Kogan Page. Rumble, G. (2001). The costs and costing of networked learning. Journal of Asynchronous Learning Networks, 5, 2, 75–96. Shepperd, M. & Schofield, C. (1997). Estimating software project effort using analogies. IEEE Transaction Software Engineering, 23, 736–743. Shepperd, M., Schofield, C. & Kitchenham, B. (1996). Effort estimation using analogy. Proceedings of the 18th International Conference on Software Engineering, p. 170–178. Srinivasan, K. & Fisher, D. (1995). Machine learning approaches to estimating software development effort. IEEE Transactions on Software Engineering, 21, 2, 126–137. Tan, W., Shen, W., Xu, L., Zhou, B. & Li, L. (2008). A business process intelligence system for enterprise process performance management. IEEE Transactions on SMC Part C, 38, 6, 745–756. Ubon, A. N. & Kimble, C. (2002). Knowledge management in online distance education. Proceedings of the 3rd International Conference Networked Learning 2002, University of Sheffield, UK, March 2002, 465–473. Wang, F., Moore, J. L., Wedman, J. & Shyu, C. (2003). Developing a case-based reasoning knowledge repository to support a learning community – an example from the technology integration community. Educational Technology Research and Development, 51, 3, 45–62. Xu, L., Liang, N. & Gao, Q. (2008). An integrated approach for agricultural ecosystem management. IEEE Transactions on SMC Part C, 38, 4, 590–599. Xu, L., Liu, H., Wang, S. & Wang, K. (2009). Modeling and analysis techniques for cross-organizational workflow systems. Systems Research and Behavioral Science, 26, 3, 367–389. © 2013 British Educational Research Association


605

Xu, L., Wang, C., Bi, Z. & Yu, J. (2012). AutoAssem: an automated assembly planning system for complex products. IEEE Transactions on Industrial Informatics, 8, 3, 669–678. Zeng, L., Li, L. & Duan, L. (2012). Business intelligence in enterprise computing environment. Information Technology & Management, 13, 4, 297–310. Zeng, L., Li, L., Duan, L., Lu, K., Shi, Z., Wang, M. et al (2012). Distributed data mining: a survey. Information Technology & Management, 13, 4, 403–410. Zhang, L., Wang, H., Cao, X., Wang, X. & Zhao, K. (2012). Knowledge management component in managing human resources for enterprises. Information Technology & Management, 13, 4, 341–349.

© 2013 British Educational Research Association