Requirements' Elicitation Needs for eLearning Systems

Requirements’ Elicitation Needs for eLearning Systems Nancy M. Rizk, Mervat H. Gheith

Eman S. Nasr

Information Systems & Technology Department, Institute of Statistical Studies and Research (ISSR), Cairo University, Egypt [email protected], [email protected]

Independent Researcher, Cairo, Egypt [email protected]

Abstract—Electronic Learning, or more popularly known as eLearning, is generally defined to be the use of technology in the delivery of education or training. eLearning Systems (eLS) are now integral parts of educational organizations. eLS are diverse in nature and size. They are nowadays integral parts also of some commercial or governmental organizations as they are costeffective means of delivering training to employees. With the diversity of people using eLS, there is a need for continuous improvement, and software development teams need to better understand the stakeholders’ requirements for faster delivery, enhancement, or personalization of eLS. Requirements elicitation is an activity within requirements engineering that is concerned with discovering needs of stakeholders, either for software development from scratch or evolution. In this paper we identify the special properties of eLS that characterize them from other software systems to help with better understanding of such domain, discuss the special requirements elicitation challenges that the special properties introduce, and introduce the main current requirements elicitation approaches used for the domain. Our research so far revealed that there are very limited approaches that are especially tailored for such domain. Hence we propose in this paper the use of crowdsourcing, which means exploiting the power of the crowd in performing tasks, as a new approach for eliciting requirements of eLS, the paper presents a framework of the necessary elements needed under the umbrella of this new approach to fill in the identified current research gap in the domain. Keywords—Requirements engineering; elicitation; electronic learning; eLearning.

RE;

requirements

I. INTRODUCTION Electronic Learning, or more popularly known as eLearning, is generally defined to be the use of technology in the delivery of education or training [1]. There is actually no unified definition for eLearning. Another common definition is the online access to learning resources, anywhere and anytime [2], [3]. The use of technology in learning has led to mixing and confusing terminologies for distance education such as, distance learning, eLearning, online learning and blended learning [4]. In this paper we will consider all of these terms to be under eLearning. eLearning Systems (eLS) are now integral parts of educational organizations. They can be used as web-based systems for online education, or as an additional tool for physical on-campus study by providing a supportive eLearning

tool for students [5]. eLS are diverse in nature and size. They are nowadays integral parts also of some commercial or governmental organizations, as they are cost-effective means of delivering training to employees [6]. eLS in this paper are not necessarily meant to be Learning Management Systems (LMS) which help in the administrative process of education and training e.g. scheduling, and registration [7]. With the diversity of people using eLS, there is always a need for continuous improvement. Improvement in information systems means better and enhanced software development process, which started by requirements identification phase or the requirements engineering (RE) phase. Software development teams need to better understand the stakeholders’ requirements for faster delivery, enhancement, or personalization of eLS. Requirements elicitation is an activity within RE, that is concerned with discovering needs of a system’s stakeholders in order to reach the system’s goals [8], either for developing a software from scratch or for software evolution. RE is defined by the Department of Defense (DOD) to be the software development phase that “involves all life-cycle activities devoted to identification of user requirements, analysis of the requirements to derive additional requirements, documentation of the requirements as a specification, and validation of the documented requirements against user needs, as well as processes that support these activities” [9]. RE activities according to Sommerville [10] are: requirements elicitation, requirements analysis, requirements specification and requirements validation. We are only concerned with the requirements elicitation activity in this paper. According to Zowghi and Coulin [11] there is no one agreed on definition for requirements elicitation. Hickey and Davis [12] define it as “learning, uncovering, extracting, surfacing, or discovering the needs of customers, users, and other potential stakeholders”. Zowghi and Coulin [11] define it to be “concerned with learning and understanding the needs of users and project sponsors with the ultimate aim of communicating these needs to the system developers”. What is agreed on is that requirements elicitation is an early software development activity within the RE phase of any software development. This activity is very important for the success of the software development project, where detecting errors at the initial stages of development can save time and money [13]. According to Standish group report [13], users involvement is one of the top success factors in any software development

project and this begins to happen during the requirements elicitation activity. It has a social nature because most of its activities depend on people [14]. In many current software systems requirements elicitation is also concerned with the software system evolution, where user feedback is collected to help in producing better future versions. Enhancing software is imperative for all software producers, in order to achieve higher usage and sales rates. Since software focuses on the customers who are the final users and its success depends on their acceptance of it, users’ feedback is usually the source of evolution in software products from version to version [15], [16]. This is an investigative paper of the current state of the art of requirements elicitation for eLS, and present what we envision to solve current problems. In this paper we identify the special properties of eLS that characterize them from other software systems to help with better understanding of such domain, discuss the special requirements elicitation challenges that the special properties introduce, and introduce the main current requirements elicitation approaches used for the domain. However when it comes to RE for eLS, there is no special approach of RE for eLS, and because the current used approaches for requirements elicitation, can’t cope with the new technologies, and concepts used in eLS such as Web 2.0 technologies. These factors motivated the researchers to find a new approach of requirements elicitation for eLS. This approach can take the power of the social web, and the new Web 2.0 technologies to better understand the stakeholders’ requirements. The paper proposes the use of crowdsourcing as a new approach for eliciting requirements of eLS, and presents a framework of the necessary elements needed for this new approach. The rest of this paper is organized as follows. Section II gives an analysis of eLS and their types. Section III provides a discussion of the special characteristics of eLS. Section IV discusses eLS’ special needs in their software development. Section V presents the current requirements elicitation approaches available in the literature for eLS. Section VI presents the use of crowdsourcing as a new approach for eliciting requirements of eLS, by proposing a framework of the necessary elements needed for this new approach, and finally Section VII gives the conclusion and future work. II.

ELEARNING SYSTEMS’ TYPES

In this section we provide an overview of the different types of eLS. Highlighting the types is expected to help in investigating its special characteristics and its special requirements in software development processes. These characteristics will be discussed in the next section. The literature reveals that there are different types of eLS. They could be classified based on different criteria, e.g. the instructor’s role, delivery method, or technology and tools used. At the same time, this classification is overlapping with eLearning methods and tools. In this section we discuss all of them. It is a background part for the classifications of the different types of eLS to show how broad eLS are, so each one has different requirements and needs.

A. Instructor-led vs. Self-paced Instructor-led learning means that there is an instructor who controls the class with a definite start and end timing with an exact learning material. Instructor led learning is like the traditional way of learning; however it can be done using virtual classrooms and hence considered eLearning. Self-paced learning is also known as self-regulated learning; it is defined according to Magill [17] as “an offering in which a learner determines the pace and timing of content delivery”. B. Technology Used • Blended learning: It is a combination of offline (face-toface, traditional learning) and online learning in a way that one compliments the other. It provides individuals with the opportunity to enjoy the best of both worlds. For example, a student might attend classes in a real-world classroom setting, and then supplement the lesson plan by completing online multimedia coursework. This can give flexibility to physically attend class once a week and would be free to go at their own pace (without worrying about scheduling issues). • Distributed/Distance learning: The term is often used to describe the process of providing access to teaching geographically distant learners [18]. With the use of computers the term can be defined as “the delivery of education, the instructional delivery including an instructor who is physically located in a different place from the learner, as well as possibly providing the instruction at different times [18]. • Mobile learning: it is defined by “Any sort of learning that happens when the learner is not at a fixed, predetermined location” e.g. PDA, iPod, portable PlayStation, or Mobile phones [19]. • Social learning: is a new term that emerged after the rise of the social media and social networks. Corporate organizations have realized the power of social learning and encouraging their employees to interact more within this domain. Employees collaborate and network to discuss problems, queries, and experiences. Social collaboration platforms are also built within the LMS so that the learners do not have to discuss on public platforms. • Game-based learning: It is a competitive activity in which students are set educational goals intended to promote knowledge acquisition. The games may either be designed to promote learning or the development of cognitive skills, or else take the form of simulations allowing learners to practice their skills in a virtual environment [2]. • Personalized learning: the tailoring of pedagogy, curriculum and learning environments to meet the needs and learning styles of individual learners. Personalization is broader than just individualization or differentiation in that it affords the learner a degree of choice about what is learned, when it is learned and how it is learned [2]. • eLearning 2.0: it is based on the use of Web 2.0 concept which is read and write web with its applications e.g. social networks, blogs, wiki [20]. The social media and digital technologies are changing the way in which students learn

and interact as they provide a platform for social learning to take place [21]. C. Management Systems • Learning Management Systems (LMS): help in the administrative process of education and training. It is used by managers, administrators, instructors, and learners. It is used to schedule, register, bill, monitor and track learners through courses and other learning events, manage training programs and compile statistics and reports [7], e.g. Moodle, BlackBoard. • Learning Content Management Systems (LCMS) combine the administrative and management dimensions of a traditional LMS with the content creation and personalized assembly dimensions. LCMS are used for authoring, approving, publishing, and managing learning content, which is referred to as Learning Objects (LO) [22]. III. THE SPECIAL CHARACTERISTICS OF ELEARNING SYSTEMS eLS are special software systems. Studying their characteristics can help us better understand what is special about them, in order to best develop this type of systems. This section presents the special properties of eLS, which are mainly as follows: • Stakeholders’ number. We discussed previously how social learning and eLearning 2.0 increased the number of eLearning users. We also discussed how the evolution of the Web technologies led to a great increase in the number of people using eLS. Users are learners, instructors, and management. • Collaboration issue. eLS require continuous collaboration between students, learners and courses contents, because learning is a social process [23]. Figure 1 illustrates some forms of interactions between eLS’ participants. Improving the social interaction in eLS lead to more satisfaction in the learning process [24], which in turn leads to the success of the eLS under use. Collaboration can take many forms like open discussion, Storytelling, forums, project work, and brainstorming [24]. • Diversity (background, culture, regulations and geographical): we mentioned the large number of stakeholders for the eLS and the different forms and types of them, this leads to stakeholders’ diversity in different aspects [5], [25], [26]. Geographical diversity as learners and instructors may reside in different locations in the same country or different countries. The diversity in location leads to diversity in culture of people, their background, and the regulations that control each country. IV. THE SPECIAL REQUIREMENTS OF ELEARNING IN SOFTWARE DEVELOPMENT eLS have special nature, the paper discussed its special characteristics in the last section. Therefore there must be special requirements during RE phase in software development process. These requirements are discussed next. • Social aspect requirements: eLS as we mentioned before is a social-based systems. It depends on the social interactions

Fig. 1. eLearning participants’ interactions.

and participation between the participants [27]. Improving social interaction in eLS can improve user satisfaction [24]. Social interactions involve more collaborative activities. Social aspects needs in eLS are translated in RE process as social aspects requirements. • Learner-centered design: since eLS’ main users are the learner, so RE must focus on the learner requirements. As we mentioned in the previous part, learning depends on the social interactions. Accordingly, effective interaction is needed between user and the application to reduce the time needed. Thus, the eLearning developer must produce accurate learner-centered design that suits learners since learners have various learning styles, behaviors and expectations towards the system [24], [28]. • Internationalization of requirements: is a process of developing an adapted software application to various languages and regions without engineering changes [29]. Currently, some educational institutions provide, or others wish to provide, cross-countries learning material and eLearning tools for multicultural students. To support their learning activities, these institutions frequently use eLS which should grant specific internationalization features [29]. • Pedagogical requirements: according to Hammad et al. [30]. There are three elements for the success of eLS; Pedagogy, technology, and learners capability. Pedagogical requirements are important in eLS. It covers the pedagogical aspects of eLearning such as learning theories and models, assessment, learner behavior. • Non-Functional Requirements (NFRs) (sustainability, accessibility, flexibility, usability/UseCase Description (UCD): Another important requirements in any systems and so the eLS are NFRs [30]. V. REQUIREMENTS ELICITATION APPROACHES FOR ELEARNING SYSTEMS After analyzing the eLS nature, its special characteristics and requirements, there is a need for a new approach to follow when eliciting requirements for eLS. We searched for publications talking about special requirements elicitation approach for eLS, which can deal with its special characteristics and requirements, but we couldn’t find any. However, we found approaches taken from other disciplines, researchers used for requirement elicitation for eLS. We discuss some of these approaches in this section.

A. Activity Theory According to Fuentes-Fernández et al. [31] Activity Theory (AT) is an approach for the analysis of social groups that focuses on their contextualized acts. People are involved in a physical and socio-cultural context and their behavior cannot be understood outside of it. At the same time, they actively interact with the environment and change it. These communication acts with the environment are called activities, and their contexts activity systems. AT is used in requirements elicitation as it is a social activity, and it is also used in learning as well. AT satisfies the social and collaborative aspect of eLS. B. Design Thinking Design Thinking (DT), according to Soledad et al. [32], is a group of user-centered techniques and tools that supports an iterative process to produce solutions to face real challenges. An experience in refining requirements for an LMS is reported. Users were involved in the DT process. As a result, DT was considered adequate to be applied in the refinement of requirements in LMS. According to Souza and Silva [33]The DT process is divided into three phases. The first one is called immersion and it is responsible for gathering, analyzing and summarizing data. The second is called ideation and is responsible for defining the users’ profile that will cooperate in the creation of the solution based on innovative ideas. Finally, the third phase is called prototyping, and it is the ability to represent the ideas concretely to promote the validation of the proposed solution. VI. A FRAMEWORK OF THE PROPOSED REQUIREMENTS ELICITATION APPROACH FOR ELS The aforementioned approaches of requirements elicitation can satisfy the eLS needs during the software development process, but they lack an important aspect which is dealing with huge numbers of participants. Since the new concepts of Web 2.0 and the social networking lead to a great increase in the number of users of the systems, there is a need for a new approach to be used in requirements elicitation for eLS, which can satisfy the special characteristics. Crowdsourcing is a new term that was started in 2006 by Jeff Howe in his article “The Rise of Crowdsourcing” [34]. It is defined in Merriam Webster as “the practice of obtaining needed services, ideas, or content by soliciting contributions from a large group of people, and especially from the online community rather than from traditional employees or suppliers” [35]. Crowdsourcing is defined by Howe in a another blog post [36] as “crowdsourcing represents the act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call. This can take the form of peer-production (when the job is performed collaboratively), but is also often undertaken by sole individuals”. It has been advocated for the domain of information systems’ development, through the involvement of users in documentation, analysis, design, coding, testing and evaluating the software [37]. These tasks were normally conducted by the software company or the contracted people, but now it can be outsourced to anyone in the general public [37].

Hosseini et al. [38] provided initial work on investigation of crowd and crowdsourcing features in requirements elicitation and proved this investigation in their next paper using focus groups and experts survey [39]. Table 1 gives a comparison between crowdsourcing-based and the current requirements elicitation approaches. The comparison criteria are based on crowd and crowdsourcing features found in [38], also supported by Groen et al. [40], in addition to the before mentioned requirements elicitation approaches found in [11], [31], [33], [32]. It is worth saying that this is a broad view comparison not a specific one for each technique because it’s not the target of the paper, but we need it to highlight the importance of crowdsourcing. Below is an explanation of the comparison found in Table 1. • Number of users involved: most of traditional requirements elicitation techniques deal with small number of stakeholders. However crowdsourcing approach is mainly based on the use of large number of people so in systems such as eLS with large number of users, it’s better to use the crowdsourcing approach. • Unknown stakeholders: it happens at the time of requirements elicitation in software development that a number of stakeholders maybe unknown for the developing team. Late stakeholders’ discovery can lead to imprecise requirements elicitation process. Crowdsourcing-based approach using supportive tools can help in this issue of stakeholders’ discovery. Some new eLS can have unknown users, thus requirement elicitation using crowdsourcing approach is a way that can help. Cost of activity: current requirements elicitation approaches are time-consuming and require professional staff, which lead to high cost requirements elicitation process. On the other hands the use of crowdsourcing requires fewer numbers of professionals so it’s less costly than the traditional approaches. • Stakeholders’ diversity: in large systems there are diverse stakeholders. Diversity of stakeholders can take many forms; geographical diversity, cultural diversity, or background diversity, this may need more effort from the development team when using current requirements TABLE I. CROWDSOURCING AND ELICITATION APPROACHES COMPARISON Comparison criteria Number of users involved Unknown stakeholders Cost of activity Stakeholders diversity Need of motivation Feedback analysis Quality considerations

CURRENT

Current requirements elicitation approaches

REQUIREMENTS

Crowdsourcing- based approach

Small

Large (Crowd)

Can't work with unknown stakeholder High because it needs experts Great effort needed to deal with stakeholders diversity

There can be unknown stakeholders Low because uses online social tools Can deal with stakeholders diversity

Low need of motivation

High need of motivation

Partially analysis

Can rely on feedback analysis

use

feedback

High quality standards

Low quality standards

elicitation approaches. It requires the use of different techniques, or better communication skills. Crowdsourcingbased approach can solve this issue because every stakeholder feels free to communicate with his own way, using the available social tools. • Need of motivation: the use of crowdsourcing-based approach requires motivating the stakeholders because stakeholders are volunteering the involvement in requirements elicitation process. On the other hands current approaches require less motivation because the developing team interacts more lively with the stakeholders. • Feedback analysis: new systems have users’ feedback mechanism. The use of users’ feedback is very important and considered as one of the powerful tools of crowdsourcing-based approach. Conversely current approaches don’t use feedback analysis in its techniques. • Quality considerations: quality standards are well established in current approaches, however in crowdsourcing-based approach quality standards are not followed because of the involvement of crowd in requirements elicitation process. After recognizing the features of current and crowdsourcing-based approaches in requirements elicitation. We can conclude from the previous comparison that crowdsourcing based approach greatly matches eLS special characteristics, so we believe that the use of a new approach based on crowdsourcing can help in the requirements elicitation process for eLS. Figure 2 illustrates the crowdsourcing necessary elements, which are needed in the requirements elicitation approach to be used to satisfy what is special about eLS. The need for an interactive way to use the power of social networking, applying text mining tools and receiving users’ feedback are important elements for the needed requirements elicitation approach. Below is an explanation for each attribute in the framework. • Interactivity: Interactivity between the eLearning participants (Instructor, learner, course, and management) is a must. Interactivity leads to better course results for learners [41], and to know the participants’ opinions in the different modules of the eLS, e.g. course material, quizzes, assignments, scheduling, and eLearning process. • Social collaboration: eLS have many supportive tools that help in the continuity of learning process and satisfying its social nature. Such eLS supportive tools are discussion forums, social networking applications, or blogs and wikis.

• Users’ feedback: There must be lots of feedback from interactivity between eLearning participants in the different forms of social collaborations that support eLS. Feedbacks can be on the eLS itself, course and its material, or the instructor and management. What important in this paper is the feedback on eLS and its tools. • Text mining tools: text mining is away for analyzing unstructured text using data mining techniques. In this case we use text mining in order to analyze eLearning participants’ feedback, which exist in the different forms of interactions happen. Interactions forms can be found in discussion forums, social networking applications attached to the eLS, comments spaces underneath blogs posts, or other different posts. Using text mining techniques will differentiate the feedback on the eLS and its tools; which is the focus here; or any other feedback. Important feedback will go through text mining process in order to extract users’ requirements to help in requirements elicitation phase for eLS development or revolution. Users’ feedback can hide information about good, bad or recommended features, which simply mean addition, omission or modifications of requirements [42]. VII. CONCLUSION eLS are special kinds of systems, which have special characteristics that should be taken into consideration during the software development process. The number of stakeholders, the collaboration nature of eLS, and the diversity of eLS’ users are among these characteristics. When it comes to RE for eLS, there is no special approach of RE for eLS, and because the current used approaches for requirements elicitation can’t cope with the new technologies, and concepts used in eLS such as Web 2.0 technologies. These factors motivated us to find a new approach of requirements elicitation for eLS. This approach can take the power of the social web, and the new Web 2.0 technologies to better understand the stakeholders’ requirements. The new approach will be based on the crowdsourcing concept to suit what’s special about eLS. The paper presented an overview and a comparison between current requirements elicitation and crowdsourcing based approaches as we envision that it can be useful and offers solutions for current problems of the eLS domain. The paper discussed a framework for the necessary elements in the proposed requirements elicitation approach. As for our future work, we intend to analyze in details, and evaluate by applying different case studies, how crowdsourcing can help as a new approach of requirements elicitation for eLS. [1]

Fig. 2. Framework of the necessary elements in proposed requirements elicitation approach for eLS.

REFERENCES Australian Flexible Learning framework, "Practical Guide to E-learning for Industry", http://industry.flexiblelearning.net.au/guide. Accessed 7/4/2016

[2]

Epignosis LLC. (2014, Jan) E-learning: Concepts, Trends, and Applications. http://www.talentlms.com/elearning/elearning-101jan2014-v1.1.pdf. Accessed 7/4/2016

[3]

B. Holmes and J. Gardner, E-learning: Concepts and Practice.: Sage Publications, 2006.

[4]

Alexandru Pop. Distance learning Portal. [Online]. http://www.distancelearningportal.com/articles/269/blended-learning-elearning-and-online-learning-whats-important.html Accessed 7/4/2016

[5]

A. D. Alharthi, M. Spichkova, and M. Hamilton, "Requirements Engineering Aspects of ELearning Systems," in Proceedings of 24th Australasian Software Engineering Conference, Adelaide, SA, Australia, 2015.

[6]

M. Rose. Elearning Industry. [Online]. http://elearningindustry.com/blended-learning-ultimately-best-corporatetraining Accessed 7/4/2016

[7]

W. Horton and K. Horton, E-learning tools and techniques.: Wiley, 2003.

[8]

[9]

C. Mellon. Software Engineering Institute. [Online]. http://www.sei.cmu.edu/productlines/frame_report/req_eng.htm Accessed 7/4/2016 E. Hull, K. Jackson, and J. Dick, Requirements Engineering, 3rd ed.: Springer-Verlag London, 2011.

[10] Ian Sommerville, Software Engineering, 9th ed. USA: Pearson, 2011. [11] D. Zowghi and C. Coulin, "Requirements Elicitation: A Survey of Techniques,Approaches, and Tools," in Proceedings of Engineering and Managing Software Requirements. Berlin: Springer, 2005, pp. 19-46. [12] A. M. Hickey and A. M. Davis, "A Unified Model of Requirements Elicitation," Journal of Management Information Systems, vol. 20, no. 4, pp. 65-84, 2004. [13] Standish Group International, "CHAOS manifesto ," 2013. [14] J. Fernandes et al., "iThink: A Game-Based Approach Towards Improving Collaboration and Participation in Requirement Elicitation," in Proceedings 4th International Conference on Games and Virtual Worlds for Serious Applications , 2012, pp. 66-77. [15] L. V. Galvis Carreno and K. Winbladh, "Analysis of User Comments: An Approach for Software Requirements Evolution," in Proceedings of the 2013 International Conference on Software Engineering, NJ, USA, 2013, pp. 582-591. [16] W. Jiang, H. Ruan, and L. Zhang, "Analysis of Economic Impact of Online Reviews: An Approach for Market-Driven Requirements Evolution," Requirements Engineering Communications in Computer and Information Science, vol. 432, pp. 45-59, 2014. [17] D. S. Magill, "What Part of Self Paced Don't You Understand," in Proceedings of 24th Annual Conference of Distance Teaching and Learning, 2008. [18] J. L. Moore , C. Dickson-Deane, and K. Galyen, "E-Learning, Online Learning, and Distance Learning Environments: Are They The Same," Internet and Higher Education, vol. 14, pp. 129-135, 2011. [19] N. Winters , "What is Mobile Learning?," in Big Issues in Mobile Learning.: UNIVERSITY OF NOTTINGHAM, 2006, pp. 5-9. [20] T. O’Reilly. (2005) What is WEB 2.0 – design patterns and business models for the next generation of software. www.oreillynet.com/pub/a/oreilly/tim/news/2005/09/30/what-is-web20.html/. accessed 5/5/2016 [21] X. Wang et al., "INTEGRATION OF E-LEARNING 2.0 WITH WEB 2.0," Journal of Information Technology in Construction, vol. 17, pp. 387-396, September 2012. [22] M. Nichani. (2001, May) elearningpost. [Online]. http://www.elearningpost.com/articles/%20archives/lcms_lms_cms_rlos. accessed 5/5/2016 [23] Learning for sustainability. [Online]. http://learningforsustainability.net/social-learning/ accessed 7/4/2016 [24] N. Abdul Rahman and S. Sahibuddin, "Challenges in E-learning: an Requirements Engineering Prespective," in Proceedings of the 3rd International Conference on Computing and Informatics, Bandung, Indonesia, 2011. [25] K. Goldsworthy and L. Rankine, "Identifying the characteristics of elearning environments used to support large units," in Proceedings of Ascilite Auckland, Auckland, 2009.

[26] V. P. Docebo. [Online]. https://www.docebo.com/2015/04/23/8-typeselearning-students/. Accessed 10/5/2016 [27] E. L. Law, A. Chatterjee, D. Renzel, and R. Klamma, "The Social Requirements Engineering (SRE) Approach to Developing a LargeScale Personal Learning Environment Infrastructure," in 21st Century Learning for 21st Century Skills.: Springer Berlin Heidelberg, 2012, ch. Volume 7563 of the series Lecture Notes in Computer Science, pp. 194207. [28] N. Abdul Rahman and S. Sahibuddin, "E-Learning Technology for Problem-Based Learning Technique: An Introduction of Requirements Elicitation Model," , Kuching,Sarawak, 2010. [29] J. A. Cos and R. Toval, "Internationalization Requirements for Elearning Audit Purpose," in Proceedings of Global Engineering Education Conference, 2012 IEEE, Marrakech, 2012, pp. 1-6. [30] R. Hammad, M. Odeh, and Z. Khan, "Towards A Generic Requirements Model for Hybrid and Cloud-based e-Learning Systems," in Proceedings of Cloud Computing Technology and Science, IEEE 5th International Conference On Volume 2, Bristol, 2013, pp. 106-111. [31] R. Fuentes-Fernández, J. J. Gómez-Sanz, and J. Pavón, "Understanding The Human Context In Requirements Elicitation," Requirements Engineering, vol. 15, no. 3, pp. 267-283, August 2010. [32] M. P. da P. Soledade et al., "Experimenting with Design Thinking in Requirements Refinement for a Learning Management System," Anais do Simpósio Brasileiro de Sistemas de Informação, 2013. [33] C. L.de C. Souza and C. Silva, "An Experimental Study of the Use of Design Thinking as a Requirements Elicitation Approach for Mobile Learning Environments," CLEI ELECTRONIC JOURNAL, vol. 18, no. 1, April 2015. [34] J. Howe. (2006, June) Wired. [Online]. http://archive.wired.com/wired/archive/14.06/crowds.html?pg=4&topic= crowds&topic_set= accessed 30/6/2016 [35] Merriam Webster. [Online]. http://www.merriamwebster.com/dictionary/crowdsourcing accessed 30/6/2016 [36] J. Howe. (2006, June) Crowdsourcing: A Definition. http://crowdsourcing.typepad.com/cs/2006/06/crowdsourcing_a.html. accessed 15/10/2016 [37] M. Hosseini, K. Phalp, J. Taylor, and R. Ali, "The Four Pillars of Crowdsourcing: a Reference Model," in Proceedings of IEEE Eighth International Conference on Research Challenges in Information Science (RCIS), Marrakech, 2014, pp. 1-12. [38] M., Phalp, K. T., Taylor, J. and Ali, R. Hosseini, "Towards Crowdsourcing for Requirements Engineering," in Proceeding of The 20th International Working Conference on Requirements Engineering: Foundation for Software Quality , Essen,Germany, 2014. [39] M. Hosseini, K. Phalp, J. Taylor, and R. Ali, "Toward Crowdsourcing for Requirements Elicitation: Results from Expert Survey," in Proceedings of The 20th International Working Conference on Requirements Engineering: Foundation for Software Quality, Essen,Germany, April,2014. [40] E. C. Groen, J. Doerr, and S. Adam, "Towards Crowd-Based Requirements Engineering A Research Preview," in Requirements Engineering: Foundation for Software Quality, Samuel A. Fricker and Kurt Schneider, Eds.: Springer, 2015, pp. 247-253. [41] C. Palazuelos, D. García-Saiz, and M. Zorril, "Social Network Analysis and Data Mining: An Application to the E-Learning Context," in Computational Collective Intelligence. Technologies and Applications, Costin Bǎdicǎ, Ngoc Thanh Nguyen, and Marius Brezovan, Eds. Craivo, Romania: Springer Berlin Heidelberg, 2013, pp. 651-660. [42] N. M. Rizk, A. Ebada, and E. S. Nasr, "Investigating Mobile Applications' Requirements Evolution Through Sentiment Analysis of Users' Reviews," in Proceedings of 11th International Computer Engineering Conference (ICENCO), Cairo, 2015, pp. 123-130.