Available online at www.sciencedirect.com
ScienceDirect Procedia Engineering 159 (2016) 66 – 71
Humanitarian Technology: Science, Systems and Global Impact 2016, HumTech2016, 7-9 June 2016, Massachusetts, USA
On Selecting an Appropriate Customizable Electronic Self-Report Survey Research Technology Stan Mierzwa*, Samir Souidi, Craig Savel Information Technology, Population Council, New York, NY 10017, USA
Abstract
This paper will discuss relevant electronic survey technologies to consider when collecting self-report data in the contexts of humanitarian settings, social science research and global public health research. Many public health and epidemiology research efforts require the collection of survey data, particularly using self-report strategies. These strategies make it possible for a survey respondent to use technology such as a tablet or smartphone on her own/ independently, so that privacy is afforded to complete the questionnaires, particularly those that include sensitive or culturally taboo questions. We will outline two selfreport technology tools that were developed by Population Council Information Technology specialists and may be considered by scientists and researchers when planning studies. The tools have been used in actual clinical-trial protocols and researchbased settings in the developing world, such as clinic environments, field-based surveys, and refugee camps, as well as in developed environments. © 2016 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license © 2016 The Authors. Published by Elsevier B.V. (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of the Organizing Committee of HumTech2016 Peer-review under responsibility of the Organizing Committee of HumTech2016
Keywords: Self-Report Data Collection; Electronic Data Collection; Humanitarian Technology; Avatars; HTML5; SmartPhones; Tablets; Web-Based Survey; ACASI; Audio-Computer Assisted Self-Interview; CAPI; Computer-Assisted Personal Interview 1. Introduction In the animated children’s television program Handy Manny, the main character provides basic repair services to homeowners using a variety of tools. These tools include many of the basic devices found in any home repair or carpenter’s bag (pliers, screwdrivers, wrench, hammer, etc.). In essence Handy Manny will pick the right tool for the environment and job. A similar association can be made when technologists are approached to assist research scientists in undertaking self-report data collection. Some of the following questions are likely to be raised: Will the physical environment allow for privacy? Will the participants be able to read? Will the participants be able to recognize Hindu-Arabic numerals? Will the participants be able to understand the audio that is played while the questionnaire text is displayed? Will the participants remain engaged long enough to complete the survey? Will the participants prefer to do the survey away from the structured clinic, community centre, or refugee camp? Will there be uninterrupted Internet connectivity in the local environment? These are just some of the basic questions the authors will often raise when beginning to discuss or consider on how to address the technology needs for a research project involving self-report data collection. Face-to-face (FTF), interviewing is often considered the gold standard for survey research. This mode generally has higher response rates and allows researchers to probe respondents’ views in more depth, because interviewers can ask follow-up questions and press for more detail if responses are deemed incomplete [1]. On the other hand, this mode is considerably more expensive (due to interviewer training and travel expenses, among other things), especially for large, geographically distributed * Corresponding author. Tel.: 1-212-339-0500; fax: +1-212-755-6052. E-mail address:
[email protected]
1877-7058 © 2016 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of the Organizing Committee of HumTech2016
doi:10.1016/j.proeng.2016.08.065
67
Stan Mierzwa et al. / Procedia Engineering 159 (2016) 66 – 71
samples. FTF can also be more susceptible to social desirability biases and other interviewer effects. For instance, in some cultures, male respondents might be hesitant to reveal their beliefs and behaviours to a female interviewer [2][3]. These are some of the reasons we have continued to see expanded use of self-report technology such as ACASI (Audio-Computer Assisted SelfInterviewing), CASI (Computer-Assisted Self-Interviewing), CAPI (Computer-Assisted Personal-Interviewing) and web-based self-report surveys. Although we will report on several different technology options to consider in particular contexts, this is not a comprehensive list. We have chosen to focus on the ones that the Population Council has custom developed and helped to implement with research scientists, epidemiologists, clinicians and demographers. The shape and technology methods that involve self-report quantitative surveying can also include many different strategies. Some of these include Computer-Assisted Telephone Interviews (CATI), Interactive Voice Response (IVR), Paper and Pencil(PAP), Video Audio Computer-Assisted Self Interviews (VACASI), and Short Message Service (SMS) to name a few. 2. Methods (Tools) As organizations begin committing their resources and energy towards the United Nations Sustainable Development Goals, or SDGs, it is hard to imagine that technology will not play a part in the broad agenda. There are likely to be many different entry points for technology solutions and several SDGs could benefit from the specific tools we will outline. Of the SDGs main goals, the use of these self-reporting tools is most relevant for the following goals: Good Health; Gender Equality; Innovation and Infrastructure; Reduced Inequalities. In addition, several of the tools could be useful for self-report data collection in HIV, reproductive health and humanitarian research settings. Tools that apply to all of these issues are as follows: 2.1. PopCouncil ACASI/CAPI Audio-Computer Assisted Self-Interview/Computer-Assisted Personal Interview technology is integrated into the Population Council custom developed electronic survey solution. Computer assisted surveys are particularly valuable tools when there are concerns regarding privacy for recipients or of response, given the often sensitive nature of the questions. ACASI surveys include audio versions of the questions, which are recorded locally in the survey locations (country and village), and include both the question and response options. ACASI technology is useful in situations when the participant population is semi-literate and participants may not be able to read and understand the questions. ACASI/CAPI technology has been used primarily in public health research efforts, particularly in HIV prevention clinical trials and in social science research. The latest version of the PC ACASI/CAPI solution functions on lower-priced Android devices, which have become more available in the developing world with ample technology support in reach [4] [5]. An earlier Population Council Windows-based tablet version is available and has been implemented in over 19 distinct research studies including clinical and non-clinical efforts and in over 10 countries and 21 languages, but it is not referenced below as we have focused on the latest Android-based software in use. In addition to concerns over levels of literacy and privacy, a recent project in Ethiopia demonstrated that in some cases the audio component is an absolute necessity, as the researchers discovered that several languages spoken in the refugee camps were not documented, and therefore human surveyors lacked the language skills to communicate, potentially biasing sample results. In these environments the research focus was on violence to adolescents, particularly girls, in a refugee camp setting. Table 1 below outlines the PC ACASI/CAPI projects which used the Android version of the survey solution. Table 1. Android PC ACASI/CAPI
Project Name
Research Focus
Countries
Languages
Year(s)
Randomized Evaluation of HIV/FP Service Models (REaCH) Adolescent Girls Empowerment Program (AGEP) Social and Economic Assets for Vulnerable Adolescent Girls (SEAVAG) study in Kenya. COMPASS (Creating Opportunities through
HIV and Family Planning
Kenya; Zambia
2013; 2014
HIV; Genderbased violence; Adolescent Girls HIV; Genderbased violence; Adolescent Girls
Zambia
Swahili; English; Bemba; Nyanja Bemba; Nyanja
Number of data records >12,000
2013; 2014
>400,000
Kenya
Swahili
2013; 2014
>300,000
Adolescent girls in refugee camps – humanitarian setting
Democratic Republic of Congo (DRC) and Sudan-
Mashi; Swahili; Meban; Regarig;
2014-2015
>1,800
68
Stan Mierzwa et al. / Procedia Engineering 159 (2016) 66 – 71
Mentorship, Parental involvement, and Safe Spaces) Hi4TU
HIV and Reproductive Health
Ethiopia border
Ingessena Kulelek;and Funj/Berta
Uganda
Luganda; English
In-process (2016)
In-process
Fig. 1. (a) Color multiple-choice (b) Two-choice (c) Color and fruit/vegetable multiple-choice.
Sample ACASI screens can be seen in Figure 1. The PC ACASI/CAPI solution includes two main architected software components; a) Android front-end which includes the survey administration section, a practice survey session (answers not saved) and the actual survey b) The data management product, which can run off-line (without Internet connectivity) to centralize all data results or via the Microsoft Azure Cloud, giving immediate remote access to data for analysis. Using the Microsoft Azure Cloud was found to be feasible in a public health research project that was conducted in both Zambia and Kenya [11]. Audio files in .mp3 format are automatically played as a participant moves through a questionnaire while wearing a headset, as seen in Figure 2. Replaying the audio can be requested by touching/clicking an icon. Respondents generally touch/click color-coded boxes, relevant graphics, calendar days or numbers to indicate their responses. .
Fig. 2. (a) Person taking a PC ACASI/CAPI survey.
2.2. iQS Interactive Questionnaire System is an HTML5 web-based self-report questionnaire system that can be accessed on smartphones, tablets and other computer devices that have Internet connectivity [5]. The system allows for participants such as the adolescents
69
Stan Mierzwa et al. / Procedia Engineering 159 (2016) 66 – 71
in our sample to create their own customized Avatar that follows them through the self-report questionnaire. Avatars help keep adolescent youths engaged and remain connected until completing the questionnaire. Avatars are also helpful when self-report questionnaires are required in clinical trials when sensitive questions will be asked. For this type of survey, the population must be literate, since there is no audio component. However the system does support the ability to integrate complementing audio to play while the survey question appears alongside the avatar. HTML5-based iQS (Interactive Questionnaire System)
Project Name
Research Focus
Countries
Year(s)
HIV
Prevention Research
6 research clinic sites in the US
2013-2015
Number of data records >600
Fig. 3. (a) Avatar creation options (b) Sample question without custom avatar displayed.
3. Discussion (Future Investigation Potential) Based on a very basic Internet search, utilizing Google Search via Google Scholar content and searching for both “Articles” and “Citations,” it seems that there may be a lack of research into allowing a self-report survey technology or method to be customized for the individual participant. An example of a research question in this area might be whether, based on participant preferences (education level, sex, disability), could an appropriate self-report tool be automatically presented to the individual with the goal of obtaining the best and most accurate self-report data? In one study, image customizing was presented in a study on image-based surveys for patients with arthritis [8]. That study examined the use of publicly available images to help guide a participant through responding to a survey, as well as the importance of allowing the participant to customize images to reduce ambiguity caused by cultural and characteristic differences [8].
Search Term “Self-report survey choices” “Self-report survey technology” “Self-report survey systems” “Providing self-report options” “Self-report options” “Self-report technology” “Survey method choice” “Individualized self-report” “Individualized self-report survey” “Individualized survey” “Appropriate survey method” “Appropriate survey technology”
Articles Found 0 0 0 0 7 53 33 53 0 144 271 8
70
Stan Mierzwa et al. / Procedia Engineering 159 (2016) 66 – 71
“Personalized self-report survey” “Personalized survey technology” “Personalized self-report”
0 0 0
It would be undeniably useful for researchers to have an “out of the box” customizable survey technology that would facilitate self-report questionnaires. For the over thirty distinct self-report survey technology implementations in which the authors have been involved, the survey technology was designed at the outset to fit the needs of a specific survey population, but not allowed to be presented in a customized manner based on the participant needs. The next frontier would be that depending on the skill level, education level, age, etc. the appropriate technology survey tool would be administered. The authors are not aware of such a system, but wondered if in the future this would be a feasible option if greater research attention was given. So, for example, at initiation of the survey if it was found the participant could not read, an ACASI survey method would be provided, or if the participant was an adolescent and could read, the iQS Avatar survey method would be provided. Some research has been done demonstrating that survey mode design could have an impact on surveys. For example, a large-scale experiment embedded in the U.S. 2000 Decennial Census showed that the use of symbols (arrows) in combination with larger and darker fonts to direct people to the next appropriate question reduced errors of not skipping ahead when directed to do so by about one-third, from 19.7% to 13.5% [12][13]. At this time the authors have been involved in providing a standardized survey technology instrument that would be made available to all participants and not customized for the particular individuals need, but to the expected need of the participant population as a whole. A basic question that remains relevant with self-report survey is “Does the survey selfreport mode or technology used make a difference?” If so, is it feasible and should it be tested to provide a variety of options for participants based on quality information obtained as part of the administration of the self-report survey? 4. Limitations The authors did not perform a formal literature review after searching for specific search terms in Google Scholar, but we recognize this limitation and gauged overall content as a sort of “Word Cloud” for terms we relate to when it comes to self-report survey technology. The academic literature coverage of Google Scholar may vary by discipline compared to other databases [9][10]. Although self-report electronic surveys may be an efficient way to obtain data, the limitations of designing surveys with semi-literate populations should be noted, in they must use closed questions, which provide a limited set of response options, such as a predefined list or Yes/No responses. Although the resulting data may be is helpful in quantitative analysis, open questions would provide for an opportunity for greater in-depth responses. 5. Conclusion and Future Development As the needs for conducting self-report surveys remains and some may say continues to increase, technology will continue to play a part. Engineers and technologists can do their part by continuing to partner with research investigators to help find or build the best possible solutions that provide the best resulting data for analysis. In this paper we outlined several tools built at the Population Council and some suggested ways to consider what tool may be most suitable. The solutions offer the obvious benefits that come with automating administration of a self-report survey, mostly that a human (interviewer or survey administrator) does not need to be present while doing the electronic questionnaires. Over the past several years our technology team has been asked to either make the PC ACASI/CAPI survey system available via open-source, or make it possible for users or research teams to create their own surveys to be used without needing high level technical assistance. Going forward, we envision and indeed are in the process of creating a web-based survey building tool that would allow researchers to create their own PC ACASI/CAPI surveys without the need of higher level technical assistance. The web front-end tool would be similar to other commercial survey building tools, such as the web-based Survey Monkey, KoBoToolbox and Open-Data Kit. However, it would also include some of the customized features that have made PC ACASI/CAPI valuable – the ability to integrate audio custom files, the ability to translate surveys into any number of languages easily, and the possibility of customizing click responsive graphics to make them locally relevant. The authors agree that knowledge benefits would be made if more research into survey mode intelligence was done such that when participants enroll to partake in a self-report survey, a customized or appropriate survey or technology solution is automatically made available to them. This pseudo artificial intelligence mechanism for providing the survey would require additional piloting and research but we feel would provide a new element in the ongoing effort to innovate self-report surveying. Acknowledgements We would like to acknowledge Irene Friedland, at the Population Council, for her contributions to the thorough edit she has provided to this paper. This work was supported by the Adolescent Medicine Trials Network for HIV/AIDS Interventions (ATN) Grant NICHD 5 U01 HD 40533 and 5 U01 HD 40474, which are funded by the National Institute of Child Health and Human Development, and with co-funding from the National Institute on Drug Abuse (NIDA) and National Institute of Mental Health (NIMH). In addition, the work has been supported by Department for International Development and Makerere University-Johns Hopkins University Care Ltd.
Stan Mierzwa et al. / Procedia Engineering 159 (2016) 66 – 71
References [1] Biemer, P. P., Lyberg, L.E., (2003). Introduction to survey quality. Hoboken, NJ: Wiley. [2] Davis, R.E., Couper, M., Janz, N., Caldwell, C., Resnicow, K., (2009). Interviewer effects in public health surveys. Health Education Research. Published online – doi:10.1093/her/cyp046. [3] Bishop, B., Cooper, A., Hillygus, S., (2009). Innovative Survey Methodologies for the Study of Attitudes Toward Terrorism and Counterterrorism Strategies, Literature Review, Institute for Homeland Security Solutions, Duke University. [4] Mierzwa, S., Souidi, S., Austrian, K., et al., (2015). Transitioning customized ACASI Windows.NET solution to Android Java on lower-priced devices and technical lessons learned, The Electronic Journal of Information Systems in Developing Countries 66(2): 1-11. [5] Savel, C., Mierzwa, S., Gorbach, P., et al., (2014). Web-based, mobile-device friendly, self-report survey system incorporating avatars and gaming console techniques, Online Journal of Public Health Informatics, Vol. 6, No. 2. [6] Mierzwa, S., Souidi, S., Friedland, I., et al., (2013). Approaches that will yield greater success when implementing self-administered electronic data capture ICT systems in the developing world with an illiterate or semi-literate population, Population Council. [7] Bonn, S.E., Trolle Lagerros, Y., Balter, K., (2013). How Valid are Web-Based Self-Reports of Weight? Journal of Medical Internet Research, 15(4), e52. [8] Longqi Yang, Diana Freed, Alex Wu, Judy Wu, JP Pollak, Deborah Estrin, 2016, Computers and Society Cornell University. [9] Kousha, K., Thelwall, M., (2007) Google Scholar citations and Google Web/URL citations: A multi-discipline exploratory analysis, Journal of the American Society for Information Science and Technology, 57(6): 1055-65. [10] Falagas, M.E., Pitsouni, E.I., Malietzis, G.A., Pappas, G., (2007) Comparision of PubMed, Scopus, Web of Science, and Google Scholar: Strengths and weaknesses, The FASEB Journal 22(2): 338-342. [11] Souidi, S., Boccio, D., Mierzwa, S., Aguilar, J., (2015) The feasibility of using Microsoft Azure infrastructure for a monitoring and evaluation system in sub-Saharan Africa, IEEE Explore, 226-232. [12] Dillman, D. A., (2006). Why Choice of Survey Mode Makes a Difference, Public Health Reports, v121(1). [13] Redline, C., Dillman, D., Dajani, A., Scaggs, M.A., (2003) Improving Navigational Performance in U.S. Census 2000 by Altering the Visually Administered Languages of Branching Instructions, Journal of Official Statistics, Vol. 19. No. 4, pp. 403-419
71
