MIPRO 2013, May 20-24, 2013, Opatija, Croatia
An Approach to the Evaluation of Embedded Engineering Study Programs Ivan Kastelan*, Moshe Barak**, Vlado Sruk***, Margarita Anastassova****, Miodrag Temerinac* *
University of Novi Sad, Faculty of Technical Sciences, Novi Sad, Serbia ** Ben-Gurion University of the Negev, Be’er Sheva, Israel *** University of Zagreb, Faculty of Electrical Engineering and Computing, Zagreb, Croatia **** CEA, LIST, Sensory and Ambient Interfaces Laboratory, 91191 - Gif-sur-Yvette CEDEX, France
[email protected] results not only in better content knowledge but also in promoting students’ higher-order competences such as critical thinking, creative thinking, problem solving, metacognitive skills and communication and teamwork skills [3]. Yet, educators should consider that engaging students in problem-based learning (PBL) cannot take place in a vacuum, and learners must acquire some basic knowledge and skills in a specific area before they can handle complex engineering projects. With these ideas in mind, we sought to develop a constructivist methodology for designing assignments of increasing complexity for engineering students.
Abstract -This paper presents an approach to the evaluation of study programs in the field of embedded computer engineering. The study programs cover the following subject categories: 1) digital system design, 2) computer system design, 3) digital signal processing, 4) computer networking and 5) system integration. The results of the evaluation of hardware, software and instructional materials in embedded engineering learning are presented. Student assignments are evaluated using the proposed tasks taxonomy which consists of three levels: 1) exercises, 2) problems, 3) projects. Students and teachers were asked to analyze the difficulty, lab material, level of student understanding and the amount of individual and team work in the subjects. First research results suggest that there is a large overhead in the number of platforms used in the mentioned courses. It was shown that in learning embedded computer engineering, the students deal merely with doing basic exercises and solving simple problems, while additional work is required in shifting the learning process towards enhancing students' higher-order cognitive skills such as problem solving and creativity, and fostering teamwork. The research also showed that it would be helpful to develop a unified embedded engineering learning platform for multiple courses in the curriculum.
I.
The rest of the paper is organized as follows: section II discusses the three-level tasks taxonomy. Section III explains the evaluation method proposed by this paper. Section IV summarizes the findings from the application of the proposed evaluation method in the field. Finally, section V gives conclusions and future work. II.
TASKS TAXONOMY
The three-level tasks taxonomy illustrated in Fig. 1 was derived partially from Bloom’s well-known taxonomy of educational objectives in the cognitive domain [4], the revised version of this taxonomy [5], the Problem Solving Taxonomy (PST) in engineering proposed by Plants, Dean, Sears and Venable [6], and Barak’s [7] discussion of using taxonomies in engineering and technology education.
INTRODUCTION
This paper presents an approach to the evaluation of a study program in the field of embedded computer engineering. We discuss the rationale for developing three-level taxonomy for evaluating the teaching and learning of subjects in computer engineering, and present the findings from the application of this tool in the field. Engineering education today has been influenced increasingly by a range of fundamental principles derived from constructivist learning theories [1], [2]. Educators recognize more and more that: knowledge is constructed by learners, not passively transmitted by teachers; learning develops by assimilating new knowledge into the learners’ prior knowledge and cognitive structures; and learning is collaborative. Engineering education that adopts these notions must shift from traditional teacher-centered instruction to learner-centered pedagogies such as problem-based learning and project-based learning, which The research leading to these results has received funding from the European Union’s Seventh Framework Programme (FP7/2007-2013) under grant agreement no 317882.
Figure 1. Tasks taxonomy in engineering education
742
field of embedded engineering in two Universities. The students and teachers were asked to fill the questionnaire about their current experiences in learning embedded computer engineering.
The Tasks Taxonomy consists of the three levels described below. A. Level 1 - Exercises Exercises are well defined assignments in which the solving process and expected results are known in advance and the learners can check if they arrived at the correct solution.
The five subject areas the questionnaires aim to evaluate are:
This type of task is often used in the first steps of learning a new subject, for example, learning a new software or hardware, drill or practice. In terms of Bloom’s taxonomy of educational objectives (revised version), exercises could be classified within Level 1 – Remembering and Level 2 – Understanding.
Digital system design
Computer system design
Digital signal processing (audio, video)
Computer networking and interfacing
System integration.
Both questionnaires (students and teachers) consisted of six questions each. Each question asked the student or teacher to give a quantitative response about a particular statement.
B. Level 2 - Problems Problems are open-ended small-scale tasks in which the students might arrive at different solutions or use different solving methods.
Students were asked to answer the following questions about each subject:
The proposed solution must meet given specifications and constraints, for example, available resources.
1.
In learning this subject we deal with basic exercises, drills and practice.
In terms of Bloom’s taxonomy of educational objectives, problem solving by engineering students could be classified within Level 3 – Applying and Level 4 – Analyzing.
2.
In learning this subject we deal with solving open-ended small-scale problems or design tasks to meet given specifications and constraints.
3.
In learning this subject we are engaged in challenging projects in which we take part in defining the problem and the design objectives.
4.
In learning this subject we face difficulties.
5.
In learning this subject we work in teams.
6.
In learning this subject we develop creativity.
C. Level 3 - Projects Projects are challenging ill-defined tasks in which the students take part in determining both the objectives and the required resources for a system’s development. Project work is aimed at fostering participants’ technical knowledge, collaborative work, aspiration and imagination.
Teachers were asked to answer the following questions about each subject:
In terms of Bloom’s taxonomy of educational objectives, engineering projects could be classified within Level 5 – Evaluation and Level 6 – Creating. The learning outcomes that are expected from the students are well reflected in the task taxonomy described above. Firstly, we believe that it is important for the students to acquire basic theoretical knowledge and practical skills in learning a specific subject in embedded engineering, for example, digital systems and computer system design (see the next section). In the second stage, we expect that learners would be able solve small-scale problems in which they have to choose the problemsolving method or suggest their solution to a problem. Finally, the program intends to engage students to deal with broad projects in embedded engineering in which they develop not only deep understanding of the subject matter, but also higher-order thinking skills, for example, the ability to use complex non-algorithmic thinking to solve a problem, foster creativity and teamwork. III.
EVALUATION METHOD
1.
In learning this subject, the students deal with basic exercises, drills and practice.
2.
In learning this subject, the students deal with solving open-ended small-scale problems or design tasks to meet given specifications and constraints.
3.
In learning this subject, the students are engaged in challenging projects in which we take part in defining the problem and the design objectives.
4.
In learning this subject, the students face difficulties.
5.
In learning this subject, the students work in teams.
6.
In learning this subject, the students develop creativity.
Each question was responded to with one of the following five answers: (1) very much, (2) much, (3) no opinion, (4) little and (5) very little.
The proposed evaluation approach was applied in the field in the evaluation of current study programs in the
743
Figure 2. Students’ responses about the Digital logic course at the University of Zagreb
IV.
FINDINGS
The empirical evaluation was done at the University of Novi Sad and the University of Zagreb. At the University of Zagreb, the number of students who responded to the questionnaire was 80 for the Digital Logic and Computer Architecture 1 courses, 15 for the Computer Architecture 2 course, and eight for the Multimedia Architecture and Systems course (a total of 183 answers). At the University of Novi Sad, the number of students who responded to the questionnaire was 30 for the Digital System Design. Additionally, 13 teachers responded to the teachers’ questionnaire in four course categories at the University of Novi Sad.
Computer Architecture 1 (Univ. Zagreb),
Computer architecture 2 (Univ. Zagreb),
Multimedia architecture and systems (Univ. Zagreb),
Digital system design (Univ. Novi Sad).
TABLE I.
STUDENTS’ RESPONSES ABOUT COMPUTER ARCHITECTURE COURSES AND MULTIMEDIA COURSE
No
A. Students’ Responses This section presents the results of the students’ questionnaires in five courses from both Universities which were selected for this evaluation. Selected courses are:
Digital logic (Univ. Zagreb),
744
Students’ responses Question
Comp. Archit. 1
Comp. Archit. 2
Multimedia Arch. & Sys.
1
Exercises
much
much
much
2
Problems
much
little
much little
3
Projects
little
no opinion
4
Difficulties
much
much
much
5
Teamwork
very little
very little
very little
6
Creativity
much
little
no opinion
Figure 3. Students’ responses about the Digital system design course at the University of Novi Sad
summarized in table 2 which shows the answers which received the most votes for each question.
Fig. 2 and Fig. 3 present the distribution of answers for the courses in design of digital systems in University of Zagreb and University of Novi Sad respectively. The answers for the remaining courses are summarized in table 1 which shows the answers which received the most votes for each question.
TABLE II.
TEACHERS’ RESPONSES ABOUT COMPUTER SYSTEMS, DSP AND COMPUTER NETWORK COURSES
B. Teachers’ Responses This section presents the results of the teachers’ questionnaires in four courses from University of Novi Sad, which were selected for this evaluation. Selected courses are:
Digital system design,
Computer system design,
Digital signal processing (DSP),
Computer networks.
No
Fig. 4 illustrates the distribution of answers for the digital system design course at the University of Novi Sad. The answers for the remaining courses are
745
Students’ responses Question
Comp. Systems
DSP
Comp. Networks
1
Exercises
much
much
much
2
Problems
much
much
much
3
Projects
little
little
little
4
Difficulties
much
little
much
5
Teamwork
much
little
much
6
Creativity
much
little
much
Figure 4. Teachers’ responses about the Digital system design course at the University of Novi Sad
high-level assignments in the taxonomy; 44.2% marked that they "face difficulties"; only 3.3% answered that they "work in teams;" and 26.2% answered 'Very much' or 'Much' for the expression "we develop creativity."
C. Summary of the Findings The results from the University of Zagreb, Faculty of Electrical Engineering and Computing and University of Novi Sad, Faculty of Technical Sciences, are summarized below.
To summarize the teachers' answers, the percentage of teachers who marked 'Very much' or 'Much' for the six items for each of the four courses was calculated. The outcomes are presented in Figure 6.
To summarize the students' answers, the percentage of students who marked 'Very much' or 'Much' for the six items for each of the four courses was calculated. The outcomes are presented in Figure 5.
It may be seen in Fig. 6 that 100% of the teachers marked 'Very much' or 'Much' for the item "The students deal with basic exercises, drills and practice," which are basic assignments in the taxonomy (scale) we presented earlier in this document; 92.3% marked that the students "deal with solving open-ended small-scale problems or design tasks to meet given specifications and constraints," which are mid-level assignments in the taxonomy; however 0% think that the students "are engaged in challenging projects."
It may be seen in Fig. 5 that 71% of the students marked 'Very much' or 'Much' for the item "We deal with basic exercises, drills and practice," which are basic assignments in the taxonomy (scale) we presented earlier in this paper; 61.7% marked that they "deal with solving open-ended small-scale problems or design tasks to meet given specifications and constraints," which are mid-level assignments in the taxonomy; only 31.1% agreed that they "are engaged in challenging projects" which are
746
Figure 5. The percentage of students who marked ‘Very much’ or ‘Much’ to the six items for each of the five courses in computer engineering
Figure 6. The percentage of teachers who marked ‘Very much’ or ‘Much’ to the six items for each of the four courses in computer engineering
opportunities for engaging students challenging and more creative projects.
In addition, 61.5% marked that the students "face difficulties"; 30.7% answered that the students "work in teams;" and 76.9% answered 'Very much' or 'Much' for the expression "the students develop creativity." V.
with
more
REFERENCES [1]
CONCLUSIONS
[2]
This paper presents an approach to evaluation of embedded engineering study programs and the results of applying the proposed method in evaluation of the embedded systems engineering curriculum across two Universities.
[3] [4]
The results indicate that in learning computer engineering, the students deal merely with doing basic exercises and solving simple problems. Much work is required to shift the teaching and learning of embedded engineering and computer science towards enhancing students' higher-order cognitive skills such as problem solving and creativity, and fostering teamwork in the engineering class.
[5]
[6]
First research results also suggest that there is a large overhead in the number of platforms used in the mentioned courses. In the future, a unified platform will be developed for these and other courses in the curriculum. This platform will extend the previous work [8]-[9] which was used only in courses in digital design. The aim is to reduce the time lost in adapting the students to different platforms in different courses and increase the
[7]
[8]
[9]
747
J. Piaget, Judgement and Reasoning in the Child, London: Routledge & Kegan Paul, 1969. L. S. Vygotsky, Mind in Society: The development of Higher Psychological Processes. Cambridge. MA: Harvard University Press, 1978. J. W. Thomas, A review of Research on Project-based Learning, San Rafael, CA: Autodesk Foundation, 2000. B. S. Bloom, and D. R. Krathwohl, Taxonomy of Educational Objectives: The Classification of Educational Goals, Handbook 1: Cognitive Domain, New York, Longmans, 1956. L. W. Anderson, and D. R. Krathwohl, A Taxonomy for Learning, Teaching and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives: New York, Longman, 2001. H. L. Plants, R. K. Dean, J. T. Sears, and W. S. Venable, “A taxonomy of roblem-solving activities and its implications for teaching”, in J. L. Lubkin, (Ed.), The Teaching of Elementary Problem-Solving in Engineering and Related Fields. Washington DC, American Society for Engineering Education, 1980, pp. 2134. M. Barak, “Teaching engineering and technology: cognitive, knowledge and problem-solving taxonomies”, Journal of Engineering, Design and Technology, 2012, in press. D. Majstorovic, E. Neborovski, M. Katona: “A cross-curriculum embedded engineering learning platform”, Proceedings of the 33rd International Convention MIPRO, 2010, pp. 1039-1044 I. Kastelan, D. Majstorovic, M. Nikolic, J. Eremic, M. Katona: “Laboratory Exercises for Embedded Engineering Learning Platform”, Proceedings of the 35th International Convention MIPRO, 2012, pp. 1113-1117
