How to Use Diversity Surveys to Explore Disparities in ...

How to Use Diversity Surveys to Explore Disparities in STEM Participation and Retention Teresa Piliouras, Navarun Gupta, Bowen Long, Pui Lam Yu, Chuxuan Jin, Phillip Dunn, Haoran Zhu

Abstract— In this paper, we explore how surveys can be used to promote diversity and inclusion by providing a deeper understanding of the factors that influence student perceptions of school experiences and life choices. We present a methodology for conducting diversity surveys, and ways to deal with common challenges, such as low response rate, assessment of statistical significance, and results interpretation. We employ a case study to explicate the methodology, and to demonstrate the care that must be taken to ensure survey results are meaningful and scientifically valid.

course of their STEM studies and careers. Research shows Hispanics, African-Americans, American Indians, Alaskan Natives, women, and LGBT individuals are disproportionately negatively impacted by one or more of the following [1] - [17], [37]- [48]:

ispanics, African-Americans, American Indians, Alaskan Natives, women, lesbian, gay, bisexual and transgender (LGBT) individuals comprise approximately two-thirds of the United States population. This represents a vast reservoir of talent. However, within STEM disciplines, this is a largely untapped, underutilized reservoir [1] - [12]. This section explores why lack of diversity in STEM is cause for concern, and prevailing theories on why it occurs. According to the United States Office of the Chief Economist, “the STEM workforce has an outsized impact on a nation’s competitiveness, economic growth, and overall standard of living. STEM workers drive innovation (as measured by patents), and they have the flexible skills needed for the modern economy. … For workers, STEM jobs are linked to lower unemployment and higher wages, regardless of educational background or other factors. Strengthening the workforce training pipeline into STEM jobs can provide benefits to both businesses and workers [13].” Getting a STEM degree provides a path out poverty and opens many career options. In a competitive, global economy, a STEM education is one of the best insurance plans for landing and keeping a well-paying job. This, in turn, is associated with higher levels of personal health and well-being [14]. Many factors can derail the progress of students during the

A. Student Success Factors 1) Inadequate academic preparedness. Many educational specialists believe middle and high school students need more STEM coursework. Students who have a strong academic foundation are less likely to drop out of, and more likely to succeed in, STEM majors. 2) Limited use of support services. When students face setbacks, such as poor standardized test scores, resources exist to help them get back on track. Many students are unaware of them and flounder simply because they do not know what to do. Students need help developing strategies to recover from setbacks. They need to know where to go when they encounter difficulties. They also need encouragement to speak up and ask for help. 3) Limited awareness of STEM choices. Many students are unaware of the range of STEM majors from which to choose, and how much the selection of a major may impact one’s future pay and job prospects. 4) Lack of peer support. Peer support is a proven factor in student success. When groups are underrepresented in academic institutions, this limits the potential for peerbased social relationships, which in turn, affords students fewer opportunities for peer-to-peer academic support (e.g., study circles, problem-set groups). Lack of peer support also arises as a consequence of where one goes to school. High-achieving, low-income students are less likely to attend selective educational institutions, which provide students with better “professional network opportunities, through alumni, parents of classmates and eventually classmates themselves [40].” “The underrepresentation of high-achieving, low-income students at the nation’s selective institutions stems from two factors: low-income students are less likely to apply to

This work was submitted on March 28, 2018. Teresa Piliouras is CEO of Technical Consulting & Research, Inc., Weston, CT 06883 (email: [email protected]). Pui Lam Yu is Executive Vice-President, Engineering of Technical Consulting & Research, Inc., Weston, CT 06883 (email: [email protected]). Bowen Long, Statistical Data Mining Consultant, Technical Consulting & Research, Inc., Weston, CT 06883.

Chuxuan Sora Jin, Statistical Data Mining Consultant, Technical Consulting & Research, Inc., Weston, CT 06883. Phillip Dun, Advisory Board, Technical Consulting & Research, Inc., Weston, CT 06883 (email: [email protected]). Navarun Gupta is Associate Professor, Chair, Electrical Engineering, School of Engineering, at the University of Bridgeport, Bridgeport, CT, US (email: [email protected]) .

Index Terms—diversity surveys, STEM disparities, survey methodology

I.

THE PROBLEM AND CAUSES OF STEM DISPARITIES

H

selective schools, and low-income students who do apply receive inadequate consideration in the admissions and financial aid process [41].” 5) Limited role models. The lack of role models within academia and industry makes it harder for underrepresented groups to imagine a similar future in STEM for themselves. 6) Lack of teacher encouragement. Students value the good opinion of their teachers. Even simple encouragement can have a profound effect on students’ self-confidence and desire to continue their studies. When students struggle to understand difficult subject matter, a supportive teacher can help empower students to persevere in spite of learning difficulties and biases (their own and others’) that might otherwise hold them back. 7) Lack of teacher training and support. Professional development training and mentoring for teachers is necessary to build their pedagogical skills. B. Economic Factors 1) Economic hardship. Rising tuition makes it harder to afford a college degree, especially if one has limited means. Hispanics, African-Americans, American Indians, Alaskan Natives, and women are more likely to support themselves through college, without financial assistance, and to take on heavy student debt. They are among the first impacted by high tuition costs. 2) Systemic poverty. It is well known that poverty has pervasive impacts on childhood health and development. This hinders student success starting early in life. Without financial support from outside sources, poverty severely constrains school and job choices and opportunities, throughout one’s life. C. Cultural Bias Factors 1) Lack of teacher self-awareness. Educators need to be aware of their own implicit biases, and how this may impact students they teach. 2) Family obligations. Significant numbers of women drop out of school and the labor force to raise families [41] [47]. This is a problem for women who want to enter STEM fields but cannot afford to do so or cannot handle the additional commitment it requires. 3) Pervasive societal bias. Many Hispanics, AfricanAmericans, American Indians, Alaskan Natives, women, and LGBT individuals face bias in academia and in the job market. Cultivating a mindset that insulates against these effects may help one persevere, but the stress remains. Consider Ebony McGee’s research on what it is like to be a high-achieving minority in a mostly-white educational institution. Her surveys and interviews with minority students reveal they believe they will always be viewed through the lens of negative stereotypes, no matter how well they do or how hard they try [16] - [17]. Theories abound and much debate exists on the causes of underrepresentation of diverse groups within STEM disciplines; whether it is a problem; and how it should be addressed. Theories and opinions are useful in suggesting

hypotheses and avenues for scientific study. However, they may or may not have relevance to everyday decisions and circumstances that impact the trajectory of a particular student’s life. A well-designed and timely survey can provide a deeper understanding of the factors in play when students make crucial decisions about their education and career and provides an avenue for students to express their needs, feelings, and expectations. II. METHODOLOGY FOR CONDUCTING DIVERSITY SURVEYS Colleges and universities use diversity surveys to explore the dynamics of gender, race, and other multicultural issues manifesting within the classroom. Diversity surveys raise awareness of implicit biases and the impacts on students, and other factors that affect STEM performance and retention [42] - [44]. This section describes a methodology for creating a diversity survey that produces scientifically valid, meaningful results. This section also explores the difficulties that commonly arise when conducting a diversity survey, and strategies to deal with them. This section includes a case study, based on a student diversity survey conducted at an undergraduate engineering school in Connecticut between November 2016 and January 2017. The students and the school allowed publication of selected survey findings, on condition of anonymity, in the interest of helping the authors paint a more realistic picture of what is involved in conducting a diversity survey. For this reason, some details (such as number of survey participants, averages, distributions by gender identification, class, etc.) are omitted from the discussion and graphics. This should not distract from the paper’s goal of describing best practices associated with high-quality surveys. A. Initial Trade-offs and Survey Considerations Diversity surveys reflect numerous tradeoffs and considerations that must be considered at the onset. These include: • What budget is available to cover the survey costs? Costs may include: incentive offerings to survey participants; statistician’s time; data entry and coding; statistical survey software; and computer and networking equipment. • What are the critical resource constraints? Should the survey be paper-based or online? Will students use their own electronic devices to access the survey, or schoolissued devices? • When will the survey be administered? During classroom time? At what point in the semester should the survey be given? • How will ethical and privacy concerns be addressed in the survey design, data collection, storage, and reporting? Is IRB (internal review board) guidance and approval needed? • What survey design is most appropriate, given the research goals? • How will the reliability and validity of the survey results be determined?

•

What is the target population for the survey? Will sampling be needed? If so, what sampling techniques should be used to select potential respondents? • How will the methodology handle missing data? • What measures of statistical significance and hypothesis testing will be used? • How will results be reported? Who will be authorized to view the results? A cross-functional team, consisting of school administrators, students, faculty, and statistical analysts, should be engaged in the survey process. This helps ensure the questions posed above are thoroughly addressed. It also improves buy-in and the support of all concerned. The answers to the questions shape each phase of the diversity survey: i) survey design; ii) survey administration; iii) data collection; iv) data analysis; v) and reporting results. B. Survey Design Every survey should have a clearly articulated purpose and set of objectives. Diversity surveys often inspire many interesting research questions. The survey should focus on only a few. Survey participants typically resist taking lengthy questionnaires. This may lead to skipped questions, missing data, and reduced participation rates. The best surveys are concise and focused. The purpose of the case study survey was to help the university understand the needs of its students and how to encourage students – across diverse gender, race, ethnicity, and other cultural and social identities – to enter and stay in STEM majors. The three main objectives were: • Execute a survey with minimal resources; • Quantify extent to which students perceive bias and report experiences that suggest a need for greater sensitivity, awareness, and appreciation of diversity; • Identify and compare experiences and feelings of different student clusters (e.g., based on gender, race, ethnicity, religious affiliation, sexual orientation, national origin, political views, academic preparedness, financial status, work status, native language, disability status, etc.). A diversity survey captures data on student attitudes, beliefs, values, feelings, and college experiences. Qualitative surveys, such as this, are designed to answer questions like: “What is X, and how do different people, communities, and cultures think and feel about X, and why [19]?” Accordingly, the case study survey asked students about their perceptions of the university culture. Students were also asked about the nature of their interactions with other students—especially, their interactions with students who differ significantly from their own cultural and group identity. C. Preliminary Survey Testing and Evaluation Crafting survey questions is a multi-step, iterative process. Emotional content, question wording, ordering, reading level, answer format, and other factors can have a major impact on survey results. Two surveys, identical except for question order, may yield very different responses. For example, consider “a Pew Research Center poll conducted in October 2003 that

found that people were more likely to favor allowing gays and lesbians to enter into legal agreements that give them the same rights as married couples when this question was asked after one about whether they favored or opposed allowing gays and lesbians to marry (45% favored legal agreements when asked after the marriage question, but 37% favored legal agreements without the immediate preceding context of a question about gay marriage). [20]” Survey questions should be analyzed to detect signs of unintentional bias. It is time-consuming to verify the quality of questions before they are used in a “real” survey. However, this is an essential step. The process used to design the survey questions for the case study involved these steps: • Faculty and statistical analysts developed an initial set of survey questions. • The statistical team posted the survey questions online using an automated survey tool created by the authors. • An independent test group—representing multiple races, ethnicities, and native languages— was invited to take the survey. The invitees consisted of college student volunteers from different geographic regions across the United States. Volunteers were asked to give feedback on: i) question clarity; ii) interpretation(s) of question meanings; iii) reactions to survey format (e.g., use of drop-down menus, radial buttons, text placement, instructions, etc.) and perceived ease of use; iv) reactions to potentially sensitive questions; v) tolerance for the time needed to complete the survey. Statistical analysts used the feedback to prune and tweak the survey questions. The test group took several versions of the survey – until the faculty and analysts were satisfied that the survey language was clear and appropriate for the intended audience and that the survey was collecting the desired information without imposing too much cognitive load or difficulty on survey participants. Depending on the criticality and intended use of the survey, additional statistical tests, beyond those discussed here, may be warranted. For more details on how to construct and evaluate survey questions, the reader is referred to [19] - [20]. The final set of survey questions asked students about their: • Overall level of satisfaction with education; • Overall level of satisfaction with school environment; • Types of interactions with other students; • Level of academic preparedness as a freshman; • Access to and use of school support services; • Perceptions of faculty and guidance staff; • Academic performance; • Demographic characteristics (country of origin, native language(s), English proficiency, race, gender identity, political views, social class, religious views, disability status, etc.); • Access to financial aid; • Work for pay (number of hours per week, reasons for working, etc.).

D. Sampling Criteria and Participant Selection The next phase of the survey involved making decisions on the sampling method and the participant eligibility criteria. Convenience samples are often used in college surveys because they are easy. Although convenience surveys may reveal interesting insights, they may mislead, and are not suitable for scientific studies. The results generally cannot be extrapolated or generalized [21]. For these reasons, the case study survey did not use convenience sampling. The case study survey was not intended to test a hypothesis about the effectiveness of an intervention or treatment program. Thus, random selection and assignment to treatment and control groups was not appropriate [21]. The final decision was to invite every undergraduate student to take the survey. The engineering school was small enough to make this practical. Since the entire student population was eligible to participate in the survey, it was not necessary to select a random sample. E. Survey Administration Faculty supported the survey effort by giving students time in class to complete it. Students were told the purpose of the survey, and that their feedback was important to the university. They were advised that it is better to answer questions honestly, or not at all. Extra course credit and gift card prizes, selected by lottery, were offered as incentives to take the survey. An online survey tool captured the survey results. This offered several benefits: • It made it easy to send and manage the survey invitations. An email with a link to the diversity survey went to every student. Automatic email reminders went periodically to non-responders. This improved the survey response rate. • Data inputs were validated as students completed each question. This ensured data quality and standardized data formatting. • The tool had built-in statistical and graphing tools. This made it easier to analyze the data. • The online survey tool was cost-effective and timely. Other survey approaches (such as paper-based surveys) were not feasible with the available personnel and budget. • All students had access to a computer and the Internet. It was easy for them to take an online survey. The surveys were voluntary and the responses were anonymous. This influenced the survey design. It would have been easy to force responders to select an answer choice (including “I prefer not to answer”) using the online survey tool. This would have eliminated missing data and simplified the analysis. However, it would have violated the spirit of a voluntary survey. Survey participants were advised and allowed to skip any question they did not want to answer. F. Collection and Analysis of Survey Data An SQL database stored the survey responses and was used to manipulate and format response data in preparation for statistical analysis. Creating a strategy to analyze the survey data required careful consideration of the following: 1) What are the research questions? Faculty wanted to know: • How happy are students with their academic studies?

• • • • • •

•

What do students need to do well in school? Do students feel they are treated fairly? Do they feel others are treated fairly? If bias occurs, what form does it take? What is the reported frequency of bias? Do some students experience more bias than others? Does this impact their satisfaction with school? What types of student interactions occur that support a culture of diversity on campus? Do students feel they get the help they need from the university, guidance counselors, faculty, and other students? Do some student groups feel less supported than others? Are there distinct student clusters? What do they look like? How big are the clusters?

2) What are the characteristics of the survey data? These characteristics impact the selection of statistical analysis techniques [22] - [27]: • Possible data coding errors and invalid responses; • Variable distribution, frequency counts, and presence of significant outliers; • Missing data, including data missing because some students did not take the survey, and data missing because some students skipped survey questions. It is advisable to offer incentives (extra credit, prizes, appeals to civic duty, etc.) and to repeat appeals to non-responders to boost the response rate as much as possible. The voluntary nature of the diversity survey meant non-responders and skipped questions would be inevitable. Some of the questions in the case study survey asked about sensitive topics, such as sexual orientation and family financial status. As expected, and shown in Fig. 1, questions about sexual orientation, grade point average, and financial status were skipped most often, followed by questions about help received from guidance counselors, and political orientation. Questions about advice received from academic advisors, degree program, ethnicity, religion, overall satisfaction with school experience, and work outside of school had comparatively lower nonresponse rates. When a question has a high non-response rate, this may suggest a need to revise it, or drop it from the survey. • Data types (nominal, ordinal, interval and ratio). For example, nominal data must be analyzed using nonparametric statistical tests that do not rely on the assumption of normality. Examples of nominal variables include gender, ethnicity, and native language. • What is the overall survey response rate? What are the characteristics of responders versus non-responders? A low response rate significantly increases the potential for bias. This reduces the survey reliability and the strength of the results [28] - [33]. For the case study, the ideal response rate was 95%. The actual response rate was 40%. Low response rates are a common problem when conducting surveys.

Fig. 1. Questions about sexual orientation, GPA, and financial status were skipped most often, followed by questions about seeking help from guidance counselors, and political orientation

Chi-square analyses were used to assess the impact of the relatively low response rate and the potential for sampling bias. Responders and non-responders were compared across all variables of interest. This was possible because the university supplied demographic data and anonymous keys for all students. The anonymous key matched the survey responders with their demographic data. Non-responders were identified by process of elimination. Comparison of the responders and non-responders did not reveal statistically significant differences. If responders and non-responders are statistically indistinguishable, it is safe to conclude that the survey results are representative of the overall student population. This is true even if the response rate is low. “A survey’s response rate is a conventional proxy for the amount of response bias contained in that study. While there are more theoretical opportunities for bias when response rates are low rather than high, there is no necessary relationship between response rates and bias [24].” 3) What statistical and data mining techniques should be used to analyze survey data? These may include: • • •

Descriptive statistics (frequency counts, proportions, and percentages). Measures of dispersion (range), averages, and distribution characteristics. CHAID (Chi-square automatic interaction detection) clustering analysis. CHAID can reveal statistically significant subgroups associated with a dependent categorical variable (e.g., “level of satisfaction with class”). This is illustrated by the hypothetical example in Fig. 2. CHAID typically reveals more detailed variable interactions as n, the number of survey participants, increases. CHAID analyses conducted for the case study did not reveal statistically significant differences in student satisfaction or perceived bias with respect to gender identity, race, sexual orientation, or other combinations of

factors. Thus, the CHAID output was a single cluster similar to the top-level square in Fig. 2. • Chi-square and Fischer’s Exact tests. These were used to investigate statistically significant differences between nominal variables (e.g., is the number of women answering “Highly Satisfied” statistically different from the number of men answering “Highly Satisfied”?). • Goodman and Kruskal’s Lambda. This was used to determine if a relationship exists between nominal variables and if so, the strength or weakness of the relationship (e.g., are experiences of personal bias related to some extent with sexual orientation?). Data may be analyzed in many ways, and with many different statistical and data mining techniques, of which only a few are described here. As the analysis progresses, it often raises more questions. Deciding whether the survey objectives have been achieved, or more analysis is needed, is an on-going judgment call. Practical considerations (time, money, personnel) often determine when the analyses must end.

Fig. 2. Sample CHAID output shows ethnic group 1 has a statistically significant higher level of class satisfaction than ethnic groups 2, 3, and 4, (where 1 – very high; 2 – high; 3 – low; 4 – very low).

G. Visualization and Reporting of Survey Results Imaging studies reveal many people have a real aversion to math. When asked to solve a math problem, the pain centers of their brain light up [34]. Although statisticians may revel in mathematical details, general audiences find it easier to interpret survey results if they are presented in simple charts and colorful graphics, with labels that clearly state the main point(s). It is also customary to label graphs with the number of responses (n), and the means (𝑋𝑋), and standard deviations (σ) of key variables. As shown in Fig. 3, data supplied by the registrar’s office revealed a modest but clear difference between the grade point averages of males and females. T-tests were performed to confirm this difference is statistically significant. However, sometimes graphs can mislead. They can be scaled to exaggerate meaningless differences. In the absence of descriptive data and statistical tests, there is no way to judge if the differences implied in Fig. 3 are meaningful or real. Without

this knowledge, the graph’s reliability and informational value cannot be assessed. People naturally look for patterns in pictures and can easily make unwarranted inferences. This is where statistics and caveats are needed. To reinforce this point, consider Fig. 4, which shows a small, but distinct difference between females and males and their respective satisfaction with their college education. Statistical tests revealed the difference is not statistically significant. When students were asked on the case study survey, “What gender do you identify with? (Choose one: Male; Female; Trans male; Trans female, Gender queer/gender non-conforming, Other identity),” all students identified as either male or female. This result supports the use of only female and male gender categories in Fig. 3 and Fig. 4. Key findings from the college diversity survey include: • Females have higher grade point averages than males. • On average, students are satisfied with their education. • Financial aid was distributed across different gender and ethnicity groups in proportion to their representation at the school. • As shown in Fig. 5, when students were asked, “During this academic year, have you personally experienced an insult or threat based on social identity?” forty-six (46%) percent said yes. The response to this question was not correlated with college satisfaction or academic performance. These findings bear further investigation. More information is needed to understand the details and circumstances under which the insults and threats occurred, and what actions by the school, if any, might prevent future occurrences. • The top reasons given for choosing a STEM major were: 1) I have a strong personal interest in STEM subject matter (30%); 2) I think having a STEM degree will help me get a good paying job after graduation (16%); 3) My parents want me to study a STEM major (14%); 4) My guidance counselors advised me to study STEM (14%). This is illustrated in Fig. 6. • Twenty-six percent (26%) reported they held a job during the academic year. Of these, seventy-five percent (75%) work on campus. The major reasons cited for working, as shown in Fig. 7, are to cover school expenses (40%), to prepare for a future career (35%), and because they are participating in a needs-based work-study program (25%). • Diverse political views, religious affiliations, sexual orientations, ethnicities, socioeconomic groups, and nationalities coexist at the university. As shown in Fig. 8, many agreed with this statement: “Students here are respectful of one another when discussing controversial issues or perspectives.” Most students said they study and prepare for class and socialize or share a meal with a culturally and socially diverse mix of people during the academic year.

Fig. 3. On average, females have a somewhat higher GPA than males.

Fig. 4. This graph shows a slight difference between females and males and their respective satisfaction with their college education. Statistical analysis reveals this is not a statistically significant difference.

Fig. 5. A significant percentage (46%) of students reported experiencing personal instances of bias on campus within the last year.

Fig. 6. The single biggest reason students gave for choosing a STEM major is a strong personal interest in the subject matter.

Fig. 7. Most students who work say they do so to make money to cover the costs of school. More than a third of working students do so to help prepare for a future career.

Fig. 8. Most students agree they discuss controversial issues and perspectives with other students in a respectful manner.

At the end of the survey students were asked, “Is there anything you need to be more successful in your STEM studies?” Below are representative, lightly edited comments: • “Everything is okay”, “It is all good”, “I good for now” • “I need to improve my coding skills” • “I want more practical working experience” • “The only problem that the University has … is that is my advisor said something and the dean says another thing. I do not know which one I will believe the first or the second.” • “Chinese are attacked by other students for no reason. … College students should know something of China and if students still have no idea about the real China and try to insult Chinese for fun this is absolutely a shame on the school. You can order more than 100 different foods in KFC in China, why do American students still think we eat fried food every day? Any family in China can afford at least one car and remember a car in China is at least twice the price in US so why do people still think Chinese drivers can't drive? This is not a problem of ‘racism’ but a problem of ‘ignorance.’ Sorry about my bad words. But to me that's the no.1 thing and I hope the school can solve it.” Although the school may decide not to do anything, the student comments suggest actionable responses, such as: i) offer opportunities to learn about China; ii) support internships that offer practical work experience; iii) improve communication between faculty and advisors. At the least, the comments help the school to be more in tune with student thoughts and feelings. To recap, the case study survey achieved its purpose and objectives. The survey revealed a diverse, friendly, and supportive university culture. Significant numbers of students reported they had recently experienced and/or witnessed instances of bias relating to social identity (i.e., race, national origin, sexual orientation, values). The survey did not reveal a link between these experiences, and reported levels of satisfaction with the university, or their level of academic performance. Special needs and student sensitivities were identified and quantified. Students generally dislike taking surveys. They need frequent reminders and prodding to check their email for the survey invitations. The case study survey had a sixty percent (60%) nonresponse rate, which is typical for a survey of this kind. Students give different reasons for their resistance to participating in surveys: they find it boring, a waste of time, they would rather do something and/or anything else. Faculty support and incentives are essential to overcome this reluctance. An examination of records from the registrar’s office revealed that a small percentage of students invited to take the survey had left the school. Why did they leave school? Did they enroll in another school, did they have money problems, did they get sick, did they have failing grades, did they have a bad experience—or was it a combination of these or other factors? What kinds of interventions might have prevented the students from leaving? This remains a mystery. However, statistically, these students, who were part of the non-responder group, are

indistinguishable from their peers who stayed in college. If the purpose of the diversity survey is to understand retention issues, the survey must reach students who are contemplating leaving or actually leave the school before graduation. The school needs other outreach strategies to identify these students and initiate appropriate contact and follow-up, beyond those used to engage students in the case study survey. The timing, and how and who administers the surveys, have a significant impact on the quality and completeness of survey results. III. SUMMARY AND CONCLUSION This paper presents a methodology for conducting scientifically rigorous surveys. The paper uses a case study, based on a diversity survey conducted at an undergraduate engineering school, to illustrate the process. A key takeaway is that it is easy to be led astray by poorly constructed surveys and graphs. These tools are only as valuable as the quality of the scientific methodology and statistical analysis used to create them. An understanding of the limitations and strengths of the survey results helps guard against unwarranted inferences and misinterpretation of data. The survey team, faculty, administration, and students should hold post-mortem reviews to explore these issues and document their findings. The case study was based on a single survey which nonetheless offered valuable insights about the diversity of the school culture and student experiences with bias. The picture it painted was far from complete. To fill in the gaps, surveys should be conducted regularly at strategic “critical moments” (such as course start, course end, exam completion, key interactions with faculty, etc.). Specific times and events can be defined as triggers to issue survey invitations. Each critical moment becomes a data point and learning opportunity. Ideally, faculty and administrators should participate in surveys. Achieving a culture of inclusiveness within the college classroom depends in large part on the instructor’s prior assumptions and awareness of diversity issues, knowledge of students’ background, and consideration of potential sensitivities in the classroom [48]. Schools are increasingly using diversity surveys to help faculty recognize their own implicit and confirmation biases, and how they impact students. Diversity surveys provide a way for all students to voice concerns, needs, and feelings. Surveys provide a way to start a conversation on sensitive subjects and elicit diverse viewpoints, but first, students must be willing to answer the questions. The most interesting questions, from a diversity perspective, were the ones most likely to be skipped. The school plays an important role in getting students to overcome their natural reluctance to speak up, even anonymously, on controversial and uncomfortable topics. Students must feel safe and protected even if they express unpopular opinions. This is a common problem with course evaluations, which are a form of survey. Both faculty and students fear retribution if the reviews are poor, and the survey quality is often suspect (non-responders are typical, survey administration may be lax, etc.). Surveys, statistics, and data mining are powerful tools. Used properly, they offer potentially transformative insights on students’ pathway through school. “Completion by Design

(CBD) represents one such effort. Over the past six years, nine community colleges in three states engaged in a process of creating cross-campus teams to use data to diagnose loss points for their students and work across campuses to design solutions to those challenges. This includes providing more aggressive advising for students, as well as helping students create maps that reduce the likelihood of wasted time and credits [35].” “Georgia State’s exploration of the data … revealed that their students were getting tripped up on things like registering for courses, resulting in wasted credits and aid dollars that led to dropout for many. Another barrier that surfaced was relatively small but unplanned personal expenses that created a gap in students’ ability to pay tuition and thus stay enrolled. The university has tackled both of those obstacles, implementing live and virtual advising improvements that have substantially reduced registration errors, as well as a nationally-recognized retention grant program that has helped more than 7,000 students in just four years [36].” Other approaches are being used in the workplace. For example, Applied, a start-up company, offers a technology platform for gender-neutral interviewing. The company founder, Katharine Zaleski, states emphatically it is not focused on increasing diversity. “It’s about hiring the best person irrespective of their background. … We’ve been quite clear in the platform that we don’t allow for positive discrimination at all among reviewers [37].” When gender is masked, code written by women is accepted more often and rated more highly than when gender is known. Applied offers a gender-neutral view of candidates’ capabilities. Indirectly, this enlarges the pool of qualified candidates, because more women are judged highly. When company executives ask what female candidates want from prospective employers, Ms. Zaleski, replies: “The answer is, women want to feel that they can belong and thrive within an organization’s culture. They don’t want to pretend that they aren’t moms or that they are comfortable with bonding activities built for and by men. Yes, half the battle is getting more women in the room, but the other half is assuring women they won’t have to hide who they are when they show up [37].” Harvard is currently conducting a study “that looks at whether potential bias will go away when we use data analytics to offer more concrete and more objectively measurable criteria for such traits as analytical skills, emotional intelligence, people skills or client interaction. Generally, the arsenal of evidencebased insights that help address flaws in the promotion process is steadily increasing. Still, fixing the process alone won’t be enough.” The researchers stress the need for other efforts to make a substantive impact on the promotion path of women and minorities. These include: recognizing accomplishments, providing opportunities to work on plum job assignments, and giving regular feedback on performance [39]. The potential benefits of diversity — diverse perspectives and problemsolving strategies — go unrealized if they are not allowed expression in the workplace. Applied in a school setting, surveys can inform diversity and inclusion efforts, and drive more conscious, objective decision making. Surveys can improve awareness and understanding of problems students face in their studies and careers, by asking

them directly. As observed by Nicholas Epley, “The easiest way to get another's perspective is by simply asking them to describe what's actually going on in their minds in a context where they can report it both honestly and accurately. This solution may seem painfully obvious, but it is not so obvious to people who are in the midst of actually using it. True insight into the minds of others is not likely to come from honing your powers of intuition, but rather by learning to stop guessing about what's on the mind of another person and learning to listen instead. [49].” Surveys also help researchers explore different ways to promote student retention and measure their impact. This is needed to understand what works and what does not, and how to offer timely intervention that bears directly on student needs. To serve these ends, diversity surveys need to be incorporated into the school culture, and must be conducted regularly, in accordance with scientific, statistically valid methods. From a larger context and perspective, surveys provide a way to test the diversity hypothesis: Identity diverse groups perform better than homogeneous groups [51], [52]. Surveys can help explore circumstances under which diverse and homogenous groups are more likely to succeed or to fail. Depending on group dynamics, diversity can engender improved communication, coordination, cooperation, and problem solving, or it can hinder it. Surveys can help us reassess our notion of group identities and what it means to be diverse. In the case study survey, every participant could be labeled by a variety of non-mutually exclusive, categorical designations (e.g., gender-racecitizenship-sexual orientation-financial status). Are divergent political, religious, and social convictions more indicative of cognitive diversity than outward physical characteristics, such as gender or race? Are two people of different genders more alike if they speak the same language than two people of the same gender who do not? Should Asians from different countries and cultures and who speak different languages be classified in the same group identity? Are these distinctions meaningful in guiding admissions, hiring, and promotion decisions? Or do they reinforce stereotypes that constrain and mislead us in our predictions of a person’s ability to succeed? Surveys, statistics and data mining allows us to view people and the labels we associate with them from a fresh perspective and to reexamine their relevance. “Concerns with social justice and equality of opportunity have a place in our lives, a central one. … We must put our diverse shoulders to the wheel and find cures for disease, policies that reduce poverty, and technologies that produce clean energy sources [51].” REFERENCES [1]

[2] [3]

[4]

[5]

C. Volokh,“What percentage of the U.S. population is gay, lesbian or bisexual?,” Washington Post, July 15, 2014 https://wapo.st/1r2wp70?tid=ss_tw&utm_term=.b0e98cd955c2. D. F. Blau and L. Kahn, “The gender wage gap: extent, trends, and explanations," Journal of Economic Literature, 2017, 55(3): 789-865. National Science Board, “Science and engineering indicators 2016,” National Science Foundation, Arlington, VA: (NSB-2016-1), 2016, https://www.nsf.gov/statistics/2016/nsb20161/#/downloads/report. National Science Board, “Chapter 2, higher education in science and education appendix tables,” https://www.nsf.gov/statistics/2016/nsb20161/#/data/source. National Center for Science and Engineering Statistics (NCSES), “WebCASPAR, integrated science and engineering resources data

system,” National Science Foundation, https://ncsesdata.nsf.gov/webcaspar/ (accessed January 4, 2018). [6] US Census Bureau, “Overview of race and Hispanic origin: 2010 census briefs, https://www.census.gov/prod/cen2010/briefs/c2010br-04.pdf, downloaded 2017-04-03. [7] United States Department of Labor, “Women in the labor force,” https://www.dol.gov/wb/factsheets/Qf-laborforce-10.htm. [8] M. Suri, “Why Is science so straight?,” The New York Times, September 4, 2014, https://www.nytimes.com/2015/09/05/opinion/manil-suri-whyis-science-so-straight.html. [9] Cech, E. A. (2015, June). LGBT Professionals’ Workplace Experiences in STEM-Related Federal Agencies. Paper presented at 2015 ASEE Annual Conference and Exposition, Seattle, WA. [10] V. Hunt, D. Layton, S. Prince, Diversity Matters, McKinsey & Company, February 2, 2015, https://www.mckinsey.com/~/media/mckinsey/business%20functions/or ganization/our%20insights/why%20diversity%20matters/diversity%20m atters.ashx. [11] J. L. Chin, “Diversity Leadership: Influence of Ethnicity, Gender, and Minority Status,” Scientific Research, Open Journal of Leadership, Vol.2, No.1, 1-10, copyright 2013, https://file.scirp.org/pdf/OJL_2013032713311941.pdf. [12] S. E. Page, The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies, Princeton University Press, 2008. [13] Office of the Chief Economist, “STEM Jobs: 2017 Update,” U.S. Department of Commerce, Economics and Statistics Administration, ESA Issue Brief #02-17, http://www.esa.doc.gov/sites/default/files/stem-jobs2017-update.pdf. [14] OECD, OECD skills outlook 2013: first results from the survey of adult skills, OECD Publishing, 2013, http://www.oecdepublishing.org/multilingualsummaries/9789264204256 -sum/html/9789264204256-sum-en.html. [15] S. Saunders and D. Kardia, “Creating inclusive college classrooms,” University of Michigan Center for Research on Learning and Teaching, 1997, http://www.crlt.umich.edu/gsis/p3_1. [16] E. McGee, “High-achieving black students, biculturalism, and out-ofschool STEM learning experiences: exploring some unintended consequences,” Journal of Urban Mathematics Education, December 2013, Vol. 6, No. 2, pp. 20–41. [17] E. McGee, “Young, black, mathematically gifted, and stereotyped,” High School Journal, V96 n3, pp. 253-263, Feb-Mar 2013. [18] C. Staats, et al, “State of the science: implicit bias review 2015,” Kirwan Institute for the Study of Race and Ethnicity, 2015, http://kirwaninstitute.osu.edu/wp-content/uploads/2015/05/2015-kirwanimplicit-bias.pdf. [19] F. Fowler, Jr., Improving survey questions, volume 38, Sage Publications, 1995. [20] Pew Research Center, “Questionnaire design,” http://www.pewresearch.org/methodology/u-s-surveyresearch/questionnaire-design/, accessed December 31, 2017. [21] A. Fink, Arlene, How to sample in surveys, 2nd edition, Sage Publications, 2003. [22] A. Fink, How to analyze survey data, survey kit 8, Sage Publications, 1995. [23] M. Lewis-Beck, Data Analysis an introduction, Vol. 103, Sage University Paper Publication, 1995, pp. 26, 30-49. [24] A. Fink, The survey handbook, 2nd edition, Sage Publications, 2003, p. 61-78. [25] H. Reynolds, Analysis of nominal data, 2nd edition, volume 7, Sage Publications, 1984. [26] J. Gibbons and J. Dickinson., Nonparametric measures of association, volume 91, Sage Publications, 1993. [27] J. Gibbons, Nonparametric statistics an introduction, volume 90, Sage Publications, 1993. [28] W. Cull, K. O’Connor, S. Sharp S, S. Tang, “Response rates and response bias for 50 surveys of pediatricians,” Health Services Research 40.1 (2005): 213–226, April 9, 2017. [29] Sage Journals, The nonresponse challenge to surveys and statistics, volume 645, Issue 1, The ANNALS of the American Academy of Political and Social Science, January 2013, http://journals.sagepub.com/toc/anna/645/1 [30] P. Kadam and S. Bhalerao. “Sample size calculation.” International Journal of Ayurveda Research, 1.1 (2010): 55–57. PMC. Web. 1 Apr. 2017.

[31] R. Thorndike and T. Thorndike-Christ, Measurement and evaluation in psychology and evaluation, 8th edition, PHI Learning, 2010. [32] C. Enders, Applied missing data analysis, Guilford Press, 2010. [33] P. Allison, Missing data, volume 07-136, Sage Publications, 2002. [34] I. Lyons, S. Beilock, “When math hurts: math anxiety predicts pain network activation in anticipation of doing math,” 2012, PLoS ONE 7(10): e48076, doi: 10.1371/journal.pone.0048076. [35] D. Greenstein, “(un)Conventional wisdom: yes, we can,” Bill and Melinda Gates Foundation Postsecondary Success, October 27, 2017, https://postsecondary.gatesfoundation.org/directorsmessage/unconventional-wisdom-yes-can/. [36] D. Greenstein, “Wanted: Student-Centered Colleges,” Bill and Melinda Gates Foundation Postsecondary Success, October 2, 2017, https://postsecondary.gatesfoundation.org/directors-message/wantedstudent-centered-colleges/. [37] K. Zaleskijan, “Job interviews without gender,” New York Times, January 6, 2018, https://nyti.ms/2Ehww9n. [38] M. Reynolds, “Hiring tool uses behavioural science to stop recruitment bias,” The New Scientist, February 23, 2017, https://www.newscientist.com/article/2122323-hiring-tool-usesbehavioural-science-to-stop-recruitment-bias/. [39] I. Bohnet, “Tackling ‘the thin file’ that can prevent a promotion,” New York Times, October 3, 2017, https://nyti.ms/2yF3FJf. [40] J. Selingo, “Six myths about choosing a college major,” New York Times, November 3, 2017, https://nyti.ms/2zcBP8L, (accessed January 6, 2018). [41] J. Giancola and R. Kahlenberg, “True merit ensuring our brightest students have access to our best colleges and universities,” Jack Kent Cooke Foundation, January 26, 2016, https://web.archive.org/web/20170112020450/http://www.jkcf.org/asset s/1/7/JKCF_True_Merit_Report.pdf (accessed January 6, 2018). [42] National Science Board, “Science and engineering indicators 2016,” National Science Foundation, Arlington, VA: (NSB-2016-1), 2016, https://www.nsf.gov/statistics/2016/nsb20161/#/downloads/report. [43] National Science Board, “Chapter 2, higher education in science and education appendix tables,” https://www.nsf.gov/statistics/2016/nsb20161/#/data/source. [44] National Center for Science and Engineering Statistics (NCSES), “WebCASPAR, integrated science and engineering resources data system,” National Science Foundation, https://ncsesdata.nsf.gov/webcaspar/ (accessed January 4, 2018). [45] US Census Bureau, “Overview of race and Hispanic origin: 2010 census briefs, https://www.census.gov/prod/cen2010/briefs/c2010br-04.pdf, downloaded 2017-04-03. [46] United States Department of Labor, “Women in the labor force,” https://www.dol.gov/wb/factsheets/Qf-laborforce-10.htm. [47] United States Department of Labor, “Latest annual data women of working age,” https://www.dol.gov/wb/stats/latest_annual_data.htm#labor. [48] S. Saunders and D. Kardia, “Creating inclusive college classrooms,” University of Michigan Center for Research on Learning and Teaching, 1997, http://www.crlt.umich.edu/gsis/p3_1. [49] N. Epley, “Invisibilia: We all think we know the people we love. We're all deluded,” National Public Radio, Morning Edition, March 22, 2018, https://n.pr/2DvarDw. [50] N. Epley, Mindwise: why we misunderstand what others think, believe, feel, and want, Vintage; Reprint edition, January 6, 2015. [51] S. Page, The difference: how the power of diversity creates better groups, firms, schools, and societies, Princeton University Press, copyright 2007, p. 319-367. [52] A. Reynolds and D. Lewis, “Teams solve problems faster when they’re more cognitively diverse,” Harvard Business Review, March 30, 2017, https://hbr.org/2017/03/teams-solve-problems-faster-when-theyre-morecognitively-diverse.