Seven Factors that Influence ICT Student Achievement Catherine Lang
Judy McKay
Sue Lewis
Swinburne University of Technology John Street, Hawthorn Victoria, Australia +613 9214 5884
Swinburne University of Technology John Street, Hawthorn Victoria, Australia +613 9214 5884
Swinburne University of Technology John Street, Hawthorn Victoria, Australia +613 9214
[email protected]
[email protected]
[email protected]
unattractive or unwelcoming to females[2]. Declining enrolments and interest in ICT courses amongst both young men and young women suggests that the ICT discipline lacks credibility as a suitable career path for most young women and an increasing number of young men [3].
ABSTRACT In the process of establishing an audit of student achievement by gender as part of a Women in IT project, seven factors were identified that affect student success. These seven factors had minimal effect when they occurred in isolation within a unit of study, but certain combinations of factors created a learning environment that was detrimental to all students, and in other instances a learning environment that was particularly unfavourable for female students. The impact of these findings has resulted in a set of recommendations to improve the teaching of IT in universities in general.
A major research study in the USA reviewed women in IT literature from the last 15 years and produced a summarised list of assumptions termed “Things we believe and expect to establish” [4 p.172] that required conclusive evidence before they could be established as factors that influence female participation in IT. The second on this list of twelve assumptions was: “Confidence, grades, and perception of grades all affect women’s participation.” According to these authors, before this assumption can be promoted to a list of known facts, any link between grades and women’s participation needs to be established. “The distribution of CSE [computer science education] grades also varies by gender according to one study. This finding must be replicated and linked to women’s retention and progression.” (our emphasis) [4 p.173]
Categories and Subject Descriptors K.3.2 [Computers and Information Science Education]: Computer Science Education: Curriculum: Information Systems Education.
General Terms: Theory
This current gender audit of selected units within the Faculty is one way of advancing the body of knowledge and contributing to theoretical understanding more generally. It is anticipated that it will also illuminate points eight and nine, “Same-sex peers help increase women’s entry and progression in CSE”, and “Self-confidence in computing ability affects the choice of computing major and career” [4 p.173]. Research in the social psychology field has shown that self-confidence influences course and career choice in general but no empirical evidence exists for computing as yet.
Keywords: Pedagogy, gender, assessment 1. INTRODUCTION There have been many intervention programs and initiatives applied at secondary and tertiary level in Australia and overseas to promote ICT to women over the last 30 years. Some of the more recent literature with a particular focus on attracting more women into the field has been labelled the “WOMEN in technology” approach because it was based on the assumption that advertising and information was all that was required to promote the discipline to girls and women [1]. This is also termed the deficit model [2] because it implied that there was a deficit in women’s knowledge about IT and that once this deficit was addressed, women would enrol in the discipline in greater numbers. Evidence shows that the deficit model has had limited success and that the problem of the discipline not being as attractive to women as it is to men, is more complex than a lack of awareness. More recently there has been emphasis on a “women in TECHNOLOGY” approach, which has viewed the nature of the discipline and changes in technology over time through the lens of gender, critically analysing the discipline for elements of masculinisation that contribute to making it
By contrast, a recently reported study conducted by a European University that focussed on gender issues and achievement within computer science and engineering found that there was no significant difference in achievement between male and female graduate computer engineers. In fact this study reported that female students generally completed studies slightly earlier than male students [5]. These findings and the previously reported assumptions gave impetus to investigate our own students and their experiences of ICT education to determine any gender differences. Understanding these results and gaining greater insights into our students’ experiences and performance became a priority. This paper reports part of a Women in IT (WIT) project funded by a three-year internal grant in an Australian University that began in 2005. The purpose of this part of the project was to provide an audit or baseline snapshot of the gender profile of the Faculty. In the process of establishing the gender profile a number of elements arose that seemed to affect the outcome of student achievement. This paper will report on these elements and their effect on student achievement. The paper is structured as follows. The background to this study will be detailed in the next section. A short description of
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ITiCSE’07, June 23-27, 2007, Dundee, Scotland, United Kingdom. Copyright 2007 ACM 978-1-59593-610-3/07/0006…$5.00.
221
the research design will be followed by a section to identify and describe the seven factors, which in certain combinations appear to have an impact on student performance. The final section will discuss the implications of these factors in terms of suitable interventions, followed by specific recommendations to improve the learning outcomes of ICT students.
4. THE SEVEN FACTORS In analysing the statistical data a pattern of factors emerged. Some of these factors appeared to affect the performance of female students, while others appeared to affect both male and female students. We have classified those factors affecting only female students as ‘gendered factors’. Those affecting all students were classified as ‘pedagogical’ if they were primarily associated with teaching and learning issues, or ‘sequencing’ if they were associated with course structural issues.
2. BACKGROUND The Women in IT project was closely based on a program at an American University that affected managed to improve female participation from 7% in 1995 to a healthy 42% in 2000 [6]. Since 2000 publications from the same institution indicate that the culture of the school has become more diverse and the critical mass of female students has had an influence in attracting a more diverse cohort of male students [7]. A major recommendation from this program was to gain an understanding of the student body before considering interventions because each university has its own particular culture. This paper is a major component of the understanding the teaching and learning culture in our Faculty and provides a baseline for our project that will inform future recommendations for interventions.
4.1 Gendered factors Critical mass [g1]: refers to the percentage of female students enrolled in the unit. This factor had an impact if the cohort was under a critical mass of 25%. Role model [g2]: The presence of female academic staff teaching the unit. These two factors appeared to determine whether a gendered culture existed, in this case perhaps a masculine or ‘clubhouse’ environment that is reported in the women in computing literature [6]. While there is no definitive study that verifies the critical mass figure of 25%, Valian (1999) quotes several experiments where "... being in a minority increases a woman's likelihood of being judged in terms of her difference from the male majority, rather than in terms of her actual performance"[8 p.140]. Cohoon and Aspray (2006) list as “Things we know” that: “Computing culture is masculine. Whether it has to be masculine, and whether the culture is a cause or a consequence of its gender composition, are different questions” [4 p. 171]. Our results indicate that the performance of female students appears to be detrimentally affected when female enrolments fall below 25% and there is no female academic teaching into the unit (Appendix 1). This can be further exacerbated by some of the following factors.
The Women in IT Gender Audit was the product of desk-based research and provided an analysis of units within the two major degree courses in the Faculty. The audit included select undergraduate units of study only, classifying them either as Information Systems (IS) or Computer Science (CS) according to the management unit that delivered the units within the Faculty and to reflect the nature of the content within each unit. IS units adopted a more business-oriented approach to content, even when technical topics were presented. The audit considered the gender mix of students within these selected units, provided a gender profile and teaching qualifications of academic staff and tutors teaching into each unit, and finally analysed grade distribution according to gender and degree program. An examination of assessment practices within some of the units followed with particular reference to the recognised preferences of some assessment types by gender as reported in the literature. Assessment practices were assessed according to individual or group assignments, written or practical assessment, degree of contextualised curriculum as well as proportion of results reliant on examination performance.
4.2 Pedagogical factors A further three factors also appear to affect the outcomes of students and are related to the quality of pedagogy. These were firstly the level of contextualisation in the curriculum, secondly the variety in assessment methods used and lastly the teaching qualifications of academics. Contextualised curriculum [p1]: A curriculum that uses real world examples that are non-gendered, i.e. not specifically related to a male or female pursuit or pastime.
3. RESEARCH APPROACH This investigation used resources available internally within the University. Firstly the degrees and subjects in the Faculty were classified into two categories: Information systems (IS), which incorporated all degree programs that had a business information systems focus, and Computer Science (CS), those with a strong programming or technical focus.
Assessment [p2]: A varied assessment portfolio with a mixture of open and closed assessment, summative and formative, and a mix of continuous and performance based activities. Formal education qualifications [p3]: Formal education qualifications of academic staff primarily responsible for delivery and planning of the unit including assessment.
Ten (2005) and four (2006) core units taught in the Faculty were selected based on their spread across both degree programs. The initial analysis was based on the enrolment cohort of 2005 which was the first year of the WIT research project. At the end of 2005 two new degree programs were implemented, one in each of the categories of IS and CS. The core subjects in some cases changed their identification code but covered similar content. Four of these new 2006 units were added to the audit.
4.3 Sequencing factors The final factors were related to the sequencing of the unit, specifically whether it was presented to first year students in their first semester of study, or whether the students were close to their final year of study. These factors are labelled: Unit sequence [s1]: whether the unit is in the first year of study. Student sequence [s2]: where the majority of students undertaking the unit are positioned in relation to progression in their degree course (e.g. in their first year or their final year).
222
Some of these definitions overlap. Both continuous and performance based assessment can be open, such as assignments that require the student to produce computer programming code, diagrams and schemas using fixed techniques and tools that primarily have only one correct answer but are open to some creative variation. In terms of good or bad pedagogy, much of the self-efficacy literature purports that students learn tasks better in small chunks and that they need a firm belief in their self-efficacy to mount and sustain the required effort to learn new and perhaps difficult tasks and concepts [9].
5. FINDINGS These seven factors had minimal effect on student achievement when they occurred in isolation. However, some combinations of factors appeared to affect successful outcomes for all students and other combinations appeared to affect successful outcomes for only female students. Factors labelled g1 critical mass of female students and g2, the presence of a female role model in academic staff, when combined with an academic staff member who had no formal education qualifications (p3) resulted in a less than optimal learning environment for all students and a greater than average fail rate for female students. This occurred in the units IS Technical, CS Programming 1 and CS Programming 2. The latter two subjects produced the highest female fail rates. The influence of a contextualised curriculum (p1) and variety in assessment modes (p2) enabled the majority of students to achieve satisfactory results, as shown in the three final year subject analysed, IS Project, IS Ethics and IS Management, each of which had extremely low failure rates overall. An abstract unit, usually programming, scheduled in the first semester of a degree course (s1) resulted in a less than optimal outcome for the majority of students and resulted in greater fail rates for female students (s2) as indicated in CS Programming 1 and CS Programming 2 where female failure rates were over 40%. These units were also delivered by academics that did not hold teaching qualifications. A programming unit delivered in the first year to first year students, IS Programming also had a high failure rate of 36%, yet the presence of a female academic who held formal educational qualifications appeared to affect the female failure rate, keeping it at only 28%, less than the overall failure rate (37%).
Each unit was analysed to indicate the predominant assessment practices, whether continuous and formative throughout the semester or performance based and summative. Where possible the exam was also analysed to determine the proportion of open and closed questioning. Research indicates that boys outperform girls on closed question types like multiple choice but girls outperformed boys on open essay questions and short free-written responses. Further the performance of boys was improved relative to girls when multiple-choice formats replaced written tests [10]. All subjects were assessed primarily by final exam and other tests but this varied significantly between 0 and 90% of exam-based assessment with an average of around 70%. Less fail grades were recorded in those subjects that had reduced weighting on an exam. Therefore we must acknowledge that all students perform better when assessed by a mixture of assignment, hurdle tests and other means, rather than exam only. The advantage of mixed assessment types in units is shown through the factor p2 and is most prominent in the third year units analysed. In the unit Information Systems 1 and Information Systems 1 (new) this was the only factor that changed between 2005 and 2006, and resulted in decrease in the failure rate from 20% to 11%.
5.1 Assessment practices
A body of research is evolving around the benefit of pairprogramming techniques in the highly abstract curriculum of the programming languages. “Pair programming, when used as a form of collaborative learning, has been shown to increase the number of women (and men) persisting in their previously stated intent to pursue degrees in computer science” [11 p.90]. The two lowest performing units where a third or more students received a fail grade were first-year computer science programming units. These units also had the smallest female cohorts. One tentative conclusion that could be drawn is that when female participation is low, so is their performance and this is supported by research that suggests a critical mass needs to be reached before females begin to be judged more on their ability than their gender [8]. However it is reasonable to assume that reasons are more complex and include issues also of pedagogy such as those noted here of educational qualifications of academics and assessment practices. Good pedagogical practices are required to create good learning outcomes for all students.
To gain a greater understanding of variations in assessment types the following classification system was used in this audit. •
Continuous and Formative: Group or individual tasks that involved student activity continually over a period of time. It also included assessment based on participation such as attendance, attentiveness and discussion in tutorials and hurdle requirements. Continuous assessment can also be formative. Some assignments and tests are formative, meaning that students can learn along the way – e.g. a hurdle test in which they have several opportunities to attempt and learn from their mistakes, or a mid-semester test where the solutions were discussed in tutorials
•
Performance and Summative: Tests and examinations that involve a performance-based event. These are considered summative where the correct or ideal solution is not provided for students.
•
Closed: Closed assessment requires only a predetermined correct answer such as multiple choice questions (usually marks are for right answers but in some cases marks deducted for wrong answers).
•
Open: is defined as that which encompasses an opportunity for the student to display their knowledge about a topic such as an essay, short written answer, or case study analysis.
The two highest performing subjects were third-year information systems subjects assessed mainly by assignment, 100% in one case and 45% in the other. They had varied female participation (30% and 17%). The method of delivery of these subjects delivered to final year students, using open, continuous modes of assessment had a positive effect on student outcomes as indicated by the low failure rates. The unit IS Project was assessed not by exam but totally by project assignment and had the highest pass rate. This unit used both continuous and formative assessment practices allowing students to gain feedback on the development of their final submission throughout the semester.
223
5.2 Educational qualifications of academics and tutors
6. CONCLUSION At first glance it appears that there is a link between low female participation and low overall performance of students but a closer look reveals that there are a number of other factors that influence student performance and create gender variations in grades awarded. As noted in the introduction we found that seven factors affected the performance of female students. One of the significant ones was cohort size, as mentioned, particularly in relation to our critical mass yardstick of 25%. When a low female critical mass was combined with no female teaching staff a masculinised or ‘clubhouse’ environment resulted. This masculine environment appeared to be detrimental to the learning outcomes of the female students. The aspects of pedagogy which affected all student performance were made of three factors: a contextualised or ‘real-world’ curriculum as opposed to an abstract one, varied assessment techniques and the teaching qualifications of staff. Other factors that affected performance were how the subject was sequenced in the program and the academic background of the students.
Due to privacy constraints the only issues on staffing that can be assessed are gender representation and qualifications. Of the thirteen academic convenors (some units had two listed) four were female and nine were male. Of the 43 tutors recorded only 21% were female which approximately reflects the balance in the student body. However, qualifications listed for staff were quite different according to gender: of the four female tutors three held postgraduate education qualification (75%), one held a PhD and two Masters in Science/IT. Of the nine males only one held a postgraduate teaching qualification (11%). Eight had science undergraduate degrees (88%) and one an Arts qualification (11%). Two held PhDs (22%) and three Masters in Science/IT (33%). Amongst this sample of academics formal education qualifications were held by more females than males. The other major difference was in the undergraduate qualifications held with nearly all of the males coming from the sciences and females from the arts. The issue may be whether or not talented programmers, the pool from which many of the male tutors are drawn from, are the best teachers of difficult subject matter to students in the first semester of their degree who have little familiarity with programming.
The two lowest performing units where students received a fail more than any other mark (around a third) were 2005 first-year programming subjects. These subjects had the lowest female participation rates (around 8%) and were almost exclusively taught by male staff with no formal education qualifications. In that masculinised environment females clearly performed worse than males. These subjects had high levels (80% and 70%) of performance-based assessment combined with abstract curricula taught by academic staff with no teaching qualifications. All these factors apparently combined to contribute to the low performance of students and especially that of females. Each of these factors are small things and may be dismissed as making mountains out of molehills “But mountains are molehills, piled one on top of another over time” [8].
5.3 Background of students and Sequencing Complementing the background to this gender audit is a 2005 quantitative survey report prepared for this same project. This report confirms that our female students are less experienced with IT than their male counterparts. The young women surveyed reported some experience with computing and creating programs prior to coming to university (30%). This was considerably less than the young men of the faculty (53.6%) and also less experience than the young men who chose NOT to study IT (of whom 36.5% reported previous programming experience). The overwhelming picture is of young women less prepared for some units within our courses than young men. It is on this base of disparity that our courses commence. We may wish to consider creating a level playing field at commencement not just for the sake of equity but also for the retention of our current students and future recruitment, females and internationals in particular.
6.1 RECOMMENDATIONS The audit provides indications of how interventions may be made in order to improve outcomes for students in ICT courses.
6.1.1 Assistance for first-year students Provide a transition subject for students new to computer programming, especially those taking programming units in their first semester and females in particular..
5.4 Pedagogy This gendered audit indicates that all students in the Faculty benefit from formative assessment practices that are continuous throughout a subject. Combined with a varied assessment portfolio, formative assessment allows students from diverse backgrounds to display their learning. The major gender differentiations are evident in subjects that have closed or summative assessment only, and where the critical mass of females is less than 25%. Female students appear to under-perform in such situations.
6.1.2 Change in Assessment Type Provide all academic staff with awareness that assessment can be gender biased. A mandatory Faculty wide workshop and the regular provision of this workshop to all new teaching staff is strongly recommended. Academics should be aware of and implement a broad portfolio of assessment methods to give the diverse student body the ability to display their knowledge. This is especially important for programming subjects, which could be improved with the addition of pair or group work to ensure optimal learning outcomes for computer programming curriculum. This awareness training and assessment variety should be incorporated in mandatory in-house training programs.
The findings from the subjects IS Project, IS Ethics and IS Management prove that all students perform better when assessed by open, continuous methods such as assignment rather than performance-based methods such as exam. However the nature of these units and the sequencing later in the degree program also affect this outcome. More importantly (if this proves true) less students fail when multiple assessment methods are used to provide students with the opportunity of displaying knowledge rather than summative and closed assessment practices.
6.1.3 Assistance for teaching staff Teaching qualifications programmes should be made available for all staff, especially those with no prior education qualifications who are teaching the more technical programming units.
224
4.
Cohoon, J.M. and W. Aspray, A critical review of the research on women's participation in postsecondary computing education, in Women and Information Technology: research on underrepresentation, J.M.C.a.W. Aspray, Editor. 2006, The MIT Press: Cambridge. p. 137 - 180. 5. Ilias, A. and M. Kordaki, Undergraduate studies in computer science and engineering: gender issues. inroads - the SIGCSE Bulletin, 2006. 38(2): p. 81-85. 6. Margolis, J. and A. Fisher, Unlocking the clubhouse: Women in computing. 2002, Cambridge, Massachusetts, USA: The MIT Press. 7. Blum, L. and C. Frieze. As the culture of computing evolves, similarity can be the difference. [paper] 2003 retrieved March 2003 [cited; 1 - 25]. Available from: www.cs.washington.edu. 8. Valian, V., Why so slow? : the advancement of women. 1999, Cambridge, Massachusetts: The MIT Press. 9. Bandura, A., Self-Efficacy: the exercise of control. 1997, New York: W.H. Freeman & Company. 10. Murphy and Whitelog, tba. 2006. 11. McDowell, C., et al., Pair programming improves student retention, confidence and program quality. Communications of the ACM`, 2006. 49(8): p. 90-95.
6.1.4 Greater awareness and education for tutors. Provide education about teaching pedagogies for tutors and promote the need for more female tutors to be present in all first year and programming units.
7. ACKNOWLEDGMENTS Our thanks go to Dr. Sara Niner, Research Assistant for this WIT project.
8. REFERENCES 1. 2. 3.
Craig, A., et al., Closing the gap: Women education and information technology courses in Australia. Journal of Systems Software, 1998. 40: p. 7-14. Spencer, S. Can you do addition? Questioning the domain of IT. in Women in IT AusWIT Conference. 2003. University of Tasmania. DEST, Department of Education, Science and Training Statistical publications. 2006, http://www.dest.gov.au/sectors/higher_education/publications_ resources/profiles/students_2005_selected_higher_education_s tatistics.htm.
Appendix 1: Gendered Achievement Factors Summary Factors
Subject Details
%
YEAR
Title
g1
g2
p1
p2
p3
s1
s2
Fail Rate
2005
IS 1
Yes
Yes
Limited
No
Yes
Yes
Yes
20
2006
IS 1(new)
Yes
Yes
Yes*
Yes*
Yes
Yes
Yes
11
2006
IS tech.l (new)
No
No
No Data
Yes
No
Yes
Yes
9
2005
SE 1
No
No
No
Yes
No
Yes
Yes
17
2005
CS Prog 1
No
Limited
No
Limited
No
Yes
Yes
27
2005
CS Prog 2
No
No
No
No
No
Yes
Yes
31
2005
IS Prog
No
Yes
No
Limited
Yes
Yes
Yes
37
2005
Dbase 1
No
No
Yes
Limited
No
Yes
Yes
26
2006
Dbase 1 (new)
No
No
No
No
No
Yes
Yes
24
2005
Human IT Study
No
Yes
Yes
Yes
Yes
Yes
Yes
29
2005
IS Prog 2
No
No
No
Yes
No
No
No
31
2005
IS Project
Yes
No
Yes**
No**
No
No
No
2
2005
IS Ethics
Yes
Yes
Yes
Yes
No
No
No
3
2006
IS (new)
No
Yes
Yes
Yes
Yes
No
No
10
Mgemnt
* change that may have affected lower fail rate in both genders and higher achievement of females in particular. **curriculum was highly contextualised. Assessment was open and formative with no examination.
225
