Apr 13, 2017 - Gardner: motivation is âthe combination of effort plus desire to achieve the goal of learning the ..... London: Edward Arnold. Gardner, R. C. ...
1
How to design and validate a questionnaire on language learning motivation RESEARCH SEMINAR FACULTY OF LANGUAGES AND LINGUISTICS 13 APRIL 2017 PRESENTER: DR. LARISA NIKITINA
Why (or when) do you need to develop your own questionnaire? 1. The available questionnaires (or research instruments) are “outdated”. 2. The available instruments are not suitable (too long; not appropriate culturally; lack aspects you want to examine) 3. Other reasons ….. ?
2
Phases in developing a research instrument
3
Choose a suitable theory / theoretical framework
4
Motivation is a psychological domain. “Motivation” originates from “motus” meaning “a moving, motion”. Motivation is intent for an action or the why? of action . Ryan and Deci : “To be motivated means to be moved to do something” (2000: 54). Psychologists recognize motivational ‘orientations’ or “the underlying attitudes and goals that give rise to action – that is, it concerns the why of actions”.
Motivation = action + goal
Theoretical foundation: Socio-educational Model of L2 Motivation
5
Developed by Gardner and Lambert (1959) and refined by Gardner (1985). One of the most popular models in applied linguistics – and for a good reason. Gardner: motivation is “the combination of effort plus desire to achieve the goal of learning the language plus favourable attitude toward learning the language”. L2 motivation = effort + goals (orientations)** + attitudes **‘integrative orientation’ and ‘instrumental orientation’ (Gardner & Lambert, 1959).
Domain of L2 motivation: constructs & variables L2 motivation is the research domain.
6
We need to assess it, to measure it. -- > Operational definitions = define the study’s domain in terms of measurable aspects (or constructs / variables). The domain of L2 motivation was measured in terms of -
General motivation
-
Instrumental orientation
-
Integrative orientation
Operational definitions
7
☼ General motivation: effort and commitment that students are willing to expend to learn the target language (TL) in order to achieve their learning goals; ☼ Instrumental orientation language learners’ perceptions of the TL utility and their intention to use the TL for pragmatic purposes, such as future studies, travel, employment or obtaining financial benefits; ☼ Integrative orientation language learners’ intention to learn the TL in order to communicate with the TL speakers and their favourable interest in culture, worldviews and ways of life of the TL speakers. (each construct is expressed by statements / stem / items item pool
Prepare the item pool “Items” are the questions or statements in your research instrument. “Item pool” is all potential questions /statements for each construct (Colton & Covert, 2007). My initial item pool on L2 motivation totalled 59. Next – establish face validity -
8
Face validity (General motivation)
9
Face validity: Do the items appear -- “on the face of it” -- to measure what they intend to measure? Which items you think do not quite “belong” in the general motivation construct? 1. I am working hard at learning the Russian language. 2. I am glad that I decided to learn the Russian language 3. I put great efforts to understand everything my lecturer teaches us during the class. 4. If my lecturer wanted someone to do an extra assignment for the class, I would certainly volunteer. 5. My friends think that it’s cool to learn Russian 6. I always volunteer to answer the questions my teacher asks in the classroom. 7. After I complete Russian level 1, I will continue learning Russian at Level 2. 8. If Russian was not offered in UM I would try to go to the TL classes somewhere else.
Face validity (Instrumental orientation)
10
How about these items’ fitness “on the face of it” in the instrumental orientation? 1. Knowledge of Russian can be useful for my further studies, such as at the Master’s or PhD level. 2. Knowledge of Russian will be useful for my future career. 3. Learning Russian will give me an interesting experience 4. Knowledge of Russian will increase my job opportunities. 5. Proficiency in Russian would bring some financial benefits for me. 6. Knowledge of Russian will help me when travelling abroad. 7. Knowledge of Russian will make me smarter
Content validity: Q-sort methodology
11
Content validity establishes whether a research instrument adequately captures the study’s domain. It can be assessed by Q-sort methodology. 1. 2.
Prepare a deck of cards with the printed items / questions Ask someone (respondent, expert) to sort the deck of card into four options: - “most definitely include this item” - “include this item” - “possibly include this item” - “definitely do not include this item”
Tabulate and compare the results. Select the items for inclusion. (See more in Colton and Covert, 2007)
Q-sort methodology in my study
12
Experts (thesis supervisors) sorted out the items into 4 categories; The average value for each item (or stem) was computed. The items with the average value ≥ 75% were included. I did it this way --
13
14
15
16
Finalize the questionnaire
17
1. Prepare the title (which reflects the content of the instrument) 2. Provide a brief summary of the research purpose (what it is about; confidentiality; willingness to participate) 3. Separate the items into appropriate sections (but mix-up the items measuring different constructs) 4. Place the demographic section in the end 5. Write “Thank you!” ☺
in the end.
6. Make sure the questionnaire looks nice (design) AND professional --- !!! Conduct a pilot study with similar respondents, ask for their comments (are instructions clear? all wordings are easy to understand; they can write their comments in you questionnaire, so you can further improve it, if needed)
Administering a questionnaire
18
The overall conduct: (Consider how do you appear to the respondents? ) -
Friendliness
-
Proper attire
- Introduce yourself - State briefly why the research is important (e.g., improve the teaching) - Show Interest and keen involvement in the process - “Any questions?” - -ask me! -
Ensure confidentiality
-
Say the participation is voluntary
- Say “Thank you!” (see more in Dörnyei and Taguchi, 2009)
19
II Quantitative phase
Establishing Reliability and Construct Validity
Data (i.e., answers) were collected from 194 students learning one of 7 European languages in UM (French, German, Italian, Brazilian Portuguese, European Portuguese , Russian and Spanish). All the data were keyed-in to enable the use of software (SPSS and AMOS)
Reliability (remember: it must be assessed, always)
20 Reliability is the extent to which an instrument yields the same information with repeated use (=repeatable, consistent, constant). Several ways to assess it (see Colton and Covert, 2007). I assessed the internal consistency reliability; applies only to multi-item scales (i.e. those where all items measure the same construct).
Reliability: The findings
21
Reliability was assessed by Cronbach’s alpha (α) coefficients (SPSS software). Values between .60 and .70 are acceptable for the social sciences research (Hair et al. , 2006).
Exploratory factor analysis (EFA)
22
The main purpose of EFA was to establish the underlying structure of the students’ L2 motivation, and to assess the validity of the constructs. BUT FIRST Must check suitability of the data for the EFA (and each statistical procedure!). Performed: (1) the Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy (2) the Bartlett’s chi-square test of sphericity (Hair et al., 2006).
23
The KMO coefficient at .851 was above the meritorious .80 range for this index. The Bartlett’s test of sphericity was significant (χ2=979.167; p
