Handset-Based Data Collection Process and Participant Attitudes

International Journal of Handheld Computing Research, 3(4), 1-21, October-December 2012 1

Handset-Based Data Collection Process and Participant Attitudes Juuso Karikoski, Department of Communications and Networking, Aalto University, Finland

ABSTRACT Handset-based measurements are an emerging method for collecting behavioral data about smartphone users. Setting up these kinds of measurements is challenging because of the personal nature of the data collection device and a lack of standards related to behavioral data and the method as a whole. Privacy issues related to the participants of the data collection are of major importance when dealing with behavioral data. Introduced is the process of collecting handset-based data in the OtaSizzle project in the Aalto University community in Finland together with a literature review of other similar data collection efforts in academia and industry. A survey is also deployed to study the incentives for participation, privacy concern levels and innovativeness of the user group participating in the measurements. This article contributes to the body of knowledge regarding measurements conducted with smartphones and sheds light on participant attitudes about them. Keywords:

Behavioral Data, Handheld Computing, Handset-Based Measurements, Participant Attitudes, Privacy, Smartphones

INTRODUCTION The purpose of this article is to present the process of collecting handset-based data in the OtaSizzle project of the Aalto University in Finland together with a literature review of other handset-based data collection efforts in academia and industry. This research is continuation of previous handset-based measurements conducted in the Aalto University by Verkasalo (2009a). Handset-based measurements are implemented with software installed in the smartphones of the users who have opted-in to participate in the research (i.e., the particiDOI: 10.4018/jhcr.2012100101

pants). Because of the personal and “alwayson” nature of the device we are able to collect context sensitive data about real user behavior directly from the users. Setting up these kinds of measurements is a complex task including issues related to the selection of software used for data collection, behavioral data acquisition and participant privacy and anonymity to name a few. Furthermore, there are no standards for setting up such measurements and the solutions regarding privacy, for example, tend to be casespecific. Thus this article contributes to the body of knowledge regarding measurements conducted with smartphones and presents the OtaSizzle process as a case study. Also a survey study is implemented to research the incentives

Copyright © 2012, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited.

2 International Journal of Handheld Computing Research, 3(4), 1-21, October-December 2012

for participation, privacy concern levels and innovativeness of the user group participating in the measurements. OtaSizzle is a project of the Aalto University in Finland and focuses on social media and mobile services. One of the main targets of the project is to provide data for research purposes. Handset-based data have been collected in the project since September 2009 and the data collection is ongoing as the research will be in progress until the end of 2012. The data collection is implemented with a user panel in the university community. The smartphone has already been identified as a useful emerging tool for social scientists (Raento et al., 2009) and has been used by scholars in the field of sociology (Eagle et al., 2009) and in analyzing mobile service usage (Verkasalo, 2009a), for example. In this article the smartphone is defined as a programmable mobile phone where third party application software can be installed. The data collected in the OtaSizzle project have already been utilized by Karikoski and Nelimarkka (2011) in measuring social relations, for example, and there are numerous other research possibilities that this novel data collection technique enables. Similar articles as this one have also been published by other researchers. Chronis et al. (2009) present a data collection platform, called SocialCircuits together with system capabilities, example applications and real world deployment guidelines. Similar data collection activities by the Nokia Research Center (NRC) are also documented by Kiukkonen et al. (2010) and Aad and Niemi (2010). Furthermore, some data collection tools have been published in great detail such as the ContextLogger2 (Hasu, 2010). All the articles in the novel area of smartphone measurements help and provide guidelines for other researchers interested in following a similar data collection approach. More specifically, this article extends the literature regarding smartphone measurements by providing also participant attitudes in addition to the process description itself. This article is structured as follows: first the related work in the area of handset-based measurements is reviewed. Then the OtaSizzle project is presented from a general and techni-

cal point of view to see how the measurements fit to the overall picture of the project. Third the OtaSizzle handset-based data collection process is presented, including issues related to the data collection software selection process, data acquisition process, legal agreements and participant privacy. Fourth the panel used in data collection is presented together with the recruitment process. Fifth the questionnaire study and results are presented. Sixth the whole process and participant attitudes are discussed together with future research. Finally the conclusions are drawn.

RELATED WORK The emergence of the programmable smartphone has enabled data collection directly from the users’ handsets. The term used in this article is handset-based data collection, but there are also other terms used for this kind of data collection such as in-device measurements, people-centric sensing, mobile audience measurements or mobile analytics, for example. There are several tools that have been used for handset-based data collection in different academic and industry projects. This chapter is a literature review of handset-based data collection efforts which share similarities with the OtaSizzle project. One of the earliest handset-based data collection efforts was conducted in HIIT’s Context project. The ContextLogger of the ContextPhone software platform (Raento et al., 2005) was developed in order to provide researchers with a robust and reliable tool for collecting handset-based data. ContextLogger has been used in studying mobility patterns and collecting data on Bluetooth device proximity, for example. The ContextLogger is still being developed and the latest version of the software is documented in detail by Hasu (2010). MIT’s Reality Mining project (Eagle & Pentland, 2006) is one of the most cited handset-based data collection efforts and they used a version of the ContextLogger software (Raento et al., 2005) along with other self-developed software



to collect data. The reality mining data set has been used for various purposes including personal interactions modeling (Chronis et al., 2009) and social network structure inference (Eagle et al., 2009), for example. The MyExperience system developed by Froehlich et al. (2007) combines passive logging of device usage, user context and environmental sensor readings with active experience sampling triggered depending on the context. Their aim is to understand how people use and experience mobile technology. Similarly to this article, they have also provided design considerations both for the researchers using the tool to collect data, and the participants producing the data for the researchers. SocioXensor is a smartphone data collector documented by ter Hofte (2007). It captures objective data about application usage and (social) context of the user. The software is intended for social scientists as an addition to existing data collection methods such as surveys and interviews. ter Hofte (2007) also provides design goals and discusses the four phases of a typical SocioXensor study in a similar manner as Froehlich et al. (2007). Helsinki University of Technology’s COIN and MoMI projects (Verkasalo, 2009a) used the SmartPhone360 software developed by Nokia (2007) to collect handset-based data in 2005 and 2006 with an aim of studying mobile service usage in general in Finland. The data collected have been utilized in studying contextual patterns in mobile service usage (Verkasalo, 2009b) and dynamics of mobile service adoption (Verkasalo, 2008), for instance. Also frameworks have been developed based on the data, such as the one for measuring mobile service adoption (Verkasalo, 2010a). The Cenceme application (Miluzzo et al., 2008) is a people-centric sensing application developed to infer the presence of individuals and publish that information to social networks using mobile phones. Miluzzo et al. (2008) present and discuss the design, implementation, evaluation and user experiences of the application. This application has also been distributed through vendor specific application stores such as the Apple App Store (Miluzzo et al., 2010).

In the last couple of years several data collection initiatives have emerged. Nokia Research Center has been collecting handset-based data in Lausanne, Switzerland (Aad & Niemi, 2010; Kiukkonen et al., 2010). Similarly to this article they provide a comprehensive description of the data collection campaign highlighting the potential of the data set. As the research is at the outset they only provide early results of their analysis regarding social interaction and positioning, for example, and leave the future papers to demonstrate the insights produced by the data. Researchers at Rice University (Shepard et al., 2010) present the LiveLab methodology for collecting handset-based data. They discuss the challenges of conducting such measurements together with the solutions provided by LiveLab. They also present early results related to application usage, web site access and network conditions. Communication Explorer by Boase and Kobayashi (2011) is a smartphone application developed to collect data on mobile communication networks. The researchers study especially how the participants use their mobile phones to connect to their personal networks. The research is still in an early phase and so far only an exploratory study has been published (Boase & Kobayashi, 2011). Falaki et al. (2011) have developed a tool called SystemSens for monitoring contextual smartphone usage. The tool collects and logs smartphone usage parameters and its design is described in detail by Falaki et al. (2011). Research results utilizing the SystemSens have been published by Falaki et al. (2010a, 2010b) related to the traffic on smartphones and diversity in smartphone usage, respectively. The projects presented in this literature review all acknowledge the pivotal role of privacy in conducting these kinds of measurements and present their case-specific data collection systems. However, it seems that for these kinds of measurements to be successful in a larger scale in the future there need to be standards related to the implementation of the measurements. This is increasingly critical as the users’ awareness of different measurements happening



in the digital domain is increasing. For more information about mobile phone sensing and handset-based data collection, please see the literature review by Lane et al. (2010). Eagle (2011) also provides a good review of using mobile phones as sensors for social research.

OtaSizzle This chapter describes the OtaSizzle project from a general viewpoint and presents the technical overview of the OtaSizzle platform. For a more detailed description of the logical structure of the platform or the project in general, see Lehväslaiho (2009) and SizzleLab (2011).

General Overview OtaSizzle is an academic project of the Aalto University in Finland. It is coordinated by Helsinki Institute for Information Technology HIIT and other collaborators include Department of Computer Science and Engineering and Department of Communications and Networking of the Aalto University. The project concentrates on social media services and mobile systems and includes a mobile social interaction experiment platform on top of which (mobile) social media services can be developed (Mäntylä et al., 2009). Aalto University forms a tight knit community of three previously separate Universities, namely Helsinki University of Technology (TKK), Helsinki School of Economics (HSE) and University of Art and Design Helsinki (TAIK). The aim of the project is to provide services for the Aalto University student and staff community and engage in service design with them. The platform also allows experimentation and examination of different designs and provides data for research purposes. As said, the project aims at providing services for the student community. There have been a number of services developed so far and many are under experimentation. All the services use common data and share the

social network, which means that users who are friends in one service are that also in the other ones. The most successful service up to date has been Kassi (Suhonen et al., 2009) (http:// kassi.sizl.org) which is a social web service for exchanging goods and services. In April 2011 the OtaSizzle services had more than 2500 registered users in total.

Technical Overview Figure 1 illustrates the technical overview of the OtaSizzle platform and indicates how the different pieces work together. Previous versions and more information of the platform can be found from Lehväslaiho (2009) and SizzleLab (2011). The foundation of the OtaSizzle platform is a shared service execution environment called Aalto Social Interface or ASI. This environment provides tools for creating user identities and groups, building social networks and launching communication channels associated to identities and groups. Thus it is possible to gather and study a large amount of data on the users and the usage of the services. ASI is implemented as a RESTful platform for social media applications to facilitate outside developers to tap into ASI and create their own applications. This way anybody can become a developer by creating their own application that is available to the OtaSizzle user base and social network. There are two portals aimed at the OtaSizzle community. Sizl.org is more aimed at the users who are looking to use the services via desktop or mobile and will also include the CoreUI service component intended for setting advanced privacy controls. Sizzlelab. org on the other hand is mainly aimed at the developers, researchers and 3rd party service experimenters. The handset-based measurements (or in-device measurements in Figure 1) are the main component under study in this article. This component will be presented in more detail in the next chapter.



Figure 1. Technical overview of OtaSizzle

OtaSizzle HANDSET-BASED DATA COLLECTION Data Collection Software The data collection software selection process is depicted in Figure 2 and it follows simple steps from identification, screening and evaluation to selection. The implementation of the handset-based data collection process began with a comparison between different software available for data collection either publicly or through research partners. From the beginning it was clear that the software would not be developed in-house, but instead acquired from a third party. This led to having to deal with legal agreements related to the sharing of collected data, which will be discussed in more detail in the coming subchapters. In total four different software were first reviewed through public information and interviews in terms of various attributes listed in Table 1 and divided into two

categories depending on if the attributes were reviewed from a researcher’s perspective or participant’s, i.e., data producer’s perspective. From a researcher’s perspective it was reviewed how the collected data can be accessed (e.g., through an API or via periodic or requested data dumps), what data types are collected (in terms of, e.g., location, application usage, Bluetooth and WiFi entries etc.) and how easy it is to influence and extend the types of data being collected. Furthermore, it was reviewed if the data were pre-processed somehow or actual raw data. Data format, device support and application identification (e.g., through Symbian UIDs in case of Symbian devices) were also reviewed. Some of the software had a capability of implementing pop-up questionnaires in the devices as an additional data collection tool and this was also listed as an attribute in the software selection process. In addition to the attributes listed above maturity was crucial from a researcher’s perspective, i.e., if the software



Figure 2. Software selection process

Table 1. Attributes used in software selection Researcher Perspective

Participant Perspective

Data access

Transparency

Data types and extensibility

Data transfer

Data format

Anonymity

Data owner

Privacy

Data pre-processing

No. of installed clients

Application identification

Operation

Device support

Client profile

Pop-up questionnaires

Own web profile

Maturity / usage so far

Amount of data

Long-term interest w/ OtaSizzle

Recruitment process

had already been used extensively. Moreover, to enable longitudinal research it was reviewed what the possible long-term interest of the party developing the software with regard to the OtaSizzle project was. From a participant’s perspective it was reviewed whether the software was transparent to the user, i.e., if the client was hidden and running in the background in the device or whether it required some participant involvement to keep the data collection up and running. This also included reviewing if the data trans-

fer from the participants’ devices was automatic or had to be done manually. End user anonymity and privacy issues were naturally critical in the software selection process and reviewed extensively. These will be discussed in more detail in the coming chapters. Some of the software also required installing more than one client and thus the number of installed clients needed for data collection was reviewed. Because of possible international roaming cases it was reviewed if the software could be used in both on- and offline (to avoid expensive



data roaming costs incurred by periodic data uploads abroad). Furthermore, the actual client was reviewed to see if it enabled presentation of any kind of information of usage to the users. This was also reviewed in terms of possible web profiles where the users could observe their own device usage statistics. Because the data transfer in most cases happens over the air, the amount of data collected and transferred, e.g., in a week, was reviewed. Moreover, it was reviewed what steps were necessary to recruit people to the data collection panel (e.g., if IMEI codes were needed to install the client or if an invitation to participate and install the client was enough). After the review process the two most suitable software tools were used in a pilot test. The test was implemented in a small research group of five to ten persons and included actual usage of the software. In addition to installing and using the software the collected data were observed as well. After this test the final software was selected. In OtaSizzle the software used is called MobiTrack, which is a mobile audience measurement platform measuring real-life user behavior, usage of devices and mobile services and various technical parameters. The determining factors in selecting MobiTrack were the easiness of the recruitment process and the low involvement needed from the users in data collection. Moreover, the data types collected were the broadest. With MobiTrack we are collecting

data on application usage, application installations, processes, battery levels and charging, Bluetooth and WiFi entries, phone calls, SMSs, MMSs, URL entries, network sessions uploads and location. The collected data are summarized in Table 2. To preserve participant privacy SMS and MMS content is not recorded at all, only the email addresses of the participants are recorded and mobile phone numbers are recorded as cryptographic hash values. The platform also provides a possibility to collect user feedback through contextual pop-up surveys, but these have not been utilized yet, because the possible effects that these surveys have on the users are still under research. The monitoring software is available as a Symbian, Google Android, Windows Mobile and BlackBerry application at the moment. For more information about the MobiTrack framework and software for mobile audience measurements, see, for example Verkasalo (2010b).

Data Acquisition Figure 3 depicts the MobiTrack handset-based data acquisition process. First of all the participants to the measurements have to be recruited. This will be presented in more detail in the next chapter, but in principle the recruitment is mainly implemented with email invitations to the prospective participants. In the OtaSizzle project all the emails are sent to the Aalto University

Table 2. Data types collected with MobiTrack Type

Details

Phone call

Duration in/out, type (in/out/missed)

SMS, MMS

No. of messages sent/received, length

Application

Foreground usage, installations, background processes

Browsing

HTTP traffic

Bluetooth scan

Names, MAC addresses

WiFi scan

Names, MAC addresses

Location

MCC, MNC, LAC and cell ID

Network session

Bearer, IAP, uploads, downloads

Energy

Battery and charging status



Figure 3. Data acquisition process

students and staff. To prevent the participants from quitting the measurements and to acquire as many participants as possible, the invitation also includes some sort of compensation of participation for the users, such as vouchers and device lotteries. In the invitations the user is directed to fill in a short survey with basic demographics and contact details. After the user has agreed to join the study an SMS is sent to the user where a link to the download page of the software client is provided. After the user has installed the software they become active data producers. The amount of data produced by a user in a week is less than half megabytes, but nonetheless, some kind of a data plan is needed for the users. The data are pre-processed, stored locally in the device and sent compressed and encrypted daily to the MobiTrack servers. If the software detects international roaming, the data will be sent to the servers once in three days. The user can also disable the application during international roaming and send the data when returning to the home land. The data are aggregated from all the participants and the raw data are then exported to the OtaSizzle researchers by request. If the researchers also wish to implement pop-up surveys with the software, there is a questionnaire tool for them as well. The questionnaire results are exported to the researchers along with other usage data.

Legal Agreements A rather broad consensus on the basic standards for fair information practice and the protection of citizen privacy exist on an international level. The Organisation for Economic Co-operation and Development (OECD, 1980) guidelines on the protection of privacy and transborder flows of personal data have provided a basis for this. However, not all countries are part of the OECD and thus there are substantial differences in how privacy policies are implemented internationally. In the European Union, where this research is being conducted, a comprehensive legal rights approach to data protection and privacy is implemented with two main directives, Directive 95/46/EC on the protection of personal data and Directive 2002/58/EC on privacy and electronic communications. More recently, the European Commission (2010) launched a review of the current legal framework because of the new challenges for the protection of personal data that the rapid technological developments (in terms of, e.g., social networking sites and cloud computing) have incurred. This kind of an overarching privacy law is not available in the USA, however, and thus it is not always evident which law or regulation should be applied and which authority is concerned in a data collection effort (Kivi, 2009). Since we



are dealing with the directives of the European Union, the laws and regulations of the USA are not discussed further. The handset-based data collection software and platform are provided by MobiTrack and not developed at Aalto University. Thus there are legal agreements that have to be made to enable research collaboration between the parties. The principle of collaboration in the OtaSizzle project is data licensing and the purpose is to collect information for research to be conducted by the parties on the usage of mobile phones, mobile services and other communication technologies, especially social media services. No monetary transactions exist between the parties, only data are exchanged. In practice this means that MobiTrack is responsible for the technical implementation of the handset-based measurements and export the collected data to the Aalto University researchers as requested. Aalto University on the other hand is responsible for collecting data about the participants through web based surveys and setting up a description for the register containing personal data as required by the Finnish Personal Data Act (523/1999) (which is based on the main EU directives introduced previously). These data will then be exported to MobiTrack as requested. Aalto University is also responsible for recruiting the panel participants and their incentives and compensation. In addition to the agreements between Aalto University and MobiTrack, the data collection participants also have to opt-in, i.e., go through a registration process and approve the research agreements, terms of use and privacy policies of the parties collecting data (i.e., give their consent). If this process is successful, the participants are applicable for joining the panel.

Participant Privacy Although the software used in the data collection resembles spyware or adware, the software and the data collection procedures are legal and the data are collected only for research purposes. Because the users also opt-in to participate in the research and install the application them-

selves, they are aware of the data collection. Although this does not make the data collection fully transparent to the participants, it is a step towards it. The transparency is an integral part in the data collection as there have been concerns regarding the leakage and misuse of personal information in some mobile platforms (see, e.g., Smith, 2010; Enck et al., 2010). Processing of private data is restricted by law in general, but if the person gives consent, then almost any processing is allowed. The consent has to be specific and informed, however, and there has to be an acceptable purpose to process personal data. It is therefore essential what the participant knows and understands about the processing of personal data (Kosta & Dumortier, 2008). Because the users explicitly give consent, the research ethics have a very important role in this research. In OtaSizzle the parties collecting data are naturally required to keep all information received from the other party in whatever form as strictly confidential and shall not disclose it to third parties without a prior written permission of the disclosing party. Both parties shall also use the information disclosed by the other party only in accordance with the Finnish Personal Data Act (523/1999) and the consent given by the participants. The parties shall also not use a lower degree of care in safeguarding the other party’s information than it uses for its own information of like sensitivity and importance. The data collected directly from the users’ smartphones are highly sensitive to privacy issues because of the personal nature of the device. Therefore only anonymized, aggregate level data are being analyzed and unique identifiers such as mobile phone numbers and names are removed. This has been assured by separating the data handling roles between authorized researchers thus reducing the point of failure to a single researcher. This researcher is responsible for the register containing personal data and will only disclose anonymized data to other authorized researchers. As Figure 4 depicts, there is a dedicated person (person N) who combines and anonymizes the collected data using numerical identifiers. Then the anonymized data are



Figure 4. Data anonymization process

handed on to the person X who performs the actual aggregate level analysis. Person N also provides the aggregate level demographics of the user group to person X and thus the data are only quasi-identifiable. A quasi-identifiable data set contains identifiers such as age, zip code, and gender which can be matched with other data sets and can thus be used to de-anonymize the anonymous data set (Beach et al., 2010). Naturally only a very limited number of authorized researchers can perform these tasks and any personal data relating to the participants will be destroyed at the latest at the end of the project. After joining the panel the participant also has the right to check what kind of information is stored concerning him or her and to demand correction of erroneous, defective, unnecessary or outdated information concerning him or her. The participant can also quit the panel at any point of time by deleting the application and request Aalto University to cease processing his or her personal data. Langheinrich (2001) has presented six principles for guiding ubiquitous computing system design, which are based on a set of fair information practices that are in use today. These principles are used as a guideline for privacy design in the OtaSizzle handset-based data collection process as they are reflected in EU legislation also. Other privacy principles include the “Privacy by Design” principles used by Aad and Niemi (2010) in the NRC data col-

lection campaign, for example. Langheinrich’s principles include notice, choice and consent, anonymity and pseudonymity, proximity and locality, adequate security, and access and recourse. First of all notice or principle of openness is the most fundamental principle of any data collection system. In practice this means that data collection should not happen unnoticed of the subject that is being monitored. In the OtaSizzle handset-based data collection process this principle is fulfilled, since the users opt-in to participate in the measurements and install an application to their smartphones. The second principle, choice and consent, means that the data collectors need to receive an explicit consent from the data subjects to enable data collection. This principle is also fulfilled in OtaSizzle with opt-in and the agreements presented in the previous subchapter. The anonymity and pseudonymity principle is more difficult to fulfill than the two first ones because it is subject of debate what types of information can be linked back to a person. In principle if data cannot be traced back to an individual, then the data collection poses no threat to the privacy of the individual. As discussed above the anonymity of the persons is ensured with the separation of data handling roles in OtaSizzle. Although this process is adequate regarding the privacy concerns in a university setting and the participants have opted-in to participate in the research, there are always concerns re-



garding re-identification (Hay et al., 2008; Beach et al., 2010) and deductive disclosure (Eagle, 2009) of the participants, for example. Re-identification means that the quasi-identified data that is handed to person X in Figure 4 can be matched to a public data set and thus the anonymized data set can be re-identified. Deductive disclosure on the other hand is a problem related to the nature of behavioral data which enables deducing the identity of an individual with only a few anonymized observations. Currently there are no universal standards regarding the anonymity of these kinds of behavioral data and the solutions are case-specific. Recently, however, there has been discussion regarding the ownership of the data and it has been suggested that the users should own their data and be able to access and remove their data from databases at any point in time (Pentland, 2009). With the increasing computational power of the smartphones it could also be possible to make inferences in real-time locally in the device, thus making it unnecessary for private information to be taken off the device (Eagle & Pentland, 2006). It remains to be seen whether one of these models will become the standard in protecting behavioral data of users, but it seems clear that some sort of standardization is needed. The fourth principle, proximity and locality, refers to a situation where data would only be collected with the smartphone whenever the owner of the device is present. This principle has not received much attention in the OtaSizzle data collection process, because the smartphone is thought of as a personal device that is carried with the user all the time. This principle is thus more applicable to ubiquitous computing scenarios with sensor networks collecting data, for example. The fifth principle, adequate security, refers mainly to secure communications and storage methods. This is a rapidly developing technological field, however, and thus the levels for adequate security are changing all the time. In OtaSizzle the data transfers are encrypted and the data stored in local servers to ensure adequate security. Access and recourse from the final principle both belong into the realm of legal

practice. The access requirements require the data collectors to “only collect data for a welldefined purpose,” “only collect data relevant for the purpose,” and “only keep data as long as it is necessary for the purpose.” These requirements are all specified in the OtaSizzle agreements and thus we can conclude that the final principle is fulfilled in the OtaSizzle handset-based data collection process as well.

USER PANEL The OtaSizzle panel studies are used to collect longitudinal data of a certain group of users. This means that once the participants have been recruited and the handset-based data collection software is installed properly on their devices, the aim is to have the participants producing data up until the end of the project. The goal was to have a static panel with the same participants throughout the whole project but in practice the panel is dynamic with new participants replacing the old ones and old participants re-joining the panel. The users have to opt-in as mentioned above so that the researchers can examine the data they produce. As was introduced in the previous chapter there are some sort of compensation of participation for the users, such as vouchers and device lotteries. This introduces a possible bias, however, because it might have an effect on the usage patterns of the participants (see conditioning effects by Lohse et al., 2000). There were also some free devices and mobile data subscriptions distributed for the Aalto University freshmen who were then obliged to also install the data collection software on the devices. According to our experience gathered in the OtaSizzle project the most difficult thing in organizing these kinds of panel studies is first of all to recruit the participants and then to prevent them from uninstalling the research application and thus quitting the panel. This is a problem in spite of the rather strong incentives that are offered to the participants. This loss of panel participants over time, also called attrition bias (Lohse et al., 2000), is one of the



main challenges together with panel selection bias and conditioning effects in panel studies. Panel selection bias means that people in the panel are different from the population to be studied and conditioning effects mean that the implementation of the panel affects the behavior of participants. Because the sample is biased towards technologically oriented early adopters (as indicated by the survey results described later), i.e., the students and staff of the Aalto University, the generalization of the results is challenging. This kind of a sample is naturally networked to some extent, however, and that enables studying the social networks among the participants, as was done by Karikoski and Nelimarkka (2011), for example. Also the usage patterns of the participants reflect the situation that might take place this or next year among the majority of subscribers as discussed by Verkasalo (2009a).

Recruitment Process Figure 5 depicts the recruitment process used in setting up the handset-based measurement panel and the surveys. First the prospective participant has to become aware of the measurements conducted in the OtaSizzle project. This happens through targeted emails, wordof-mouth or publicity in the OtaSizzle portals, student events or courses, for example. Then the participant has to fill in the initial survey by entering a web page designed specifically for this purpose. After filling in the survey the user needs to agree with the terms of use related to the survey, otherwise the user won’t be able to participate in the research. In the survey the user also indicates whether they have a suitable device for the handset-based measurements or not. If they do not, they only participate in the survey part of the research. If they have a suitable device they will receive a link via SMS to go to the handset-based measurement software registration page. After filling in the registration details, agreeing with research agreement and privacy policy related to the software and installing the software, the user becomes an

active data producer. If the registration process terminates at any point in time, the users are still considered as participants in the survey part of the research. It is also possible for the users to skip the survey part of the research and navigate straight to the software registration page. After the initial setup there have been additional rounds for recruiting more participants, for example via the Aalto University student union mailing list. Also the previous participants that have quit the panel have been asked to join again and to our surprise there have also been users joining the panel outside the actual recruitment rounds. This indicates that it makes sense to have the software registration page up and running all the time and not just during the recruitment rounds. After the initial survey described in Figure 5 there have also been additional surveys targeted for the participants, such as the one described in more detail in the next chapter.

SURVEY ON PARTICIPANT ATTITUDES The survey was implemented during the fall of 2010. The survey was targeted to all the participants that had produced data in the measurements at some point since the beginning of the research in September 2009 (N=199). The motivation behind the survey was to study what incentives the users have for participation, what their privacy concern levels are and if they belong to an early adopter or innovator category (Rogers, 2003) of users regarding mobile phones and services. These factors were considered the most important attitudinal factors affecting participation in the measurements. Three hypotheses were formulated and the survey designed based on them. H1: The biggest incentives to join the study are the device and voucher lotteries. H2: The majority of the participants belong to “the Unconcerned” regarding general privacy concern levels.



Figure 5. Panel recruitment process

H3: The majority of the participants belong to the innovator or early adopter category of mobile phone / service users. First the importances of a selected set of incentives for participation were surveyed to study hypothesis one. The users also had a possibility to add their own incentives. The second

hypothesis was surveyed with a privacy index measuring general levels of privacy concern. A modified version of Westin’s privacy index from the Harris-Equifax Consumer Privacy Survey from 1991 (Kumaraguru & Cranor, 2005) was used to divide the respondents to three groups; the privacy Fundamentalists, the Pragmatic and the Unconcerned. The Privacy Fundamentalists



are respondents who are generally distrustful towards organizations that ask for their personal information. The Unconcerned in the other extreme are generally trustful of organizations collecting their personal information. The Pragmatic fall in between - they are respondents that weigh the benefits of collecting personal information against the degree of intrusiveness. In addition to the two hypotheses presented above it was also hypothesized that the respondents are early adopter or innovator type of users when considering mobile phones and services. The “electronic innovativeness” of the respondents was thus surveyed with a scale adapted from Goldsmith et al. (1995). Naturally the basic demographics of the respondents were surveyed and will be presented in the next subchapters along with the results. Because of the pivotal role of the privacy issues related to the measurements introduced previously, the focus of this survey was also on privacy. Naturally there are other factors that influence the decision to take part in the measurements, such as the fact that the Aalto University might be perceived as a more trustworthy organization implementing the measure-

ments when compared to a situation where an organization from outside the Aalto University would be implementing the measurements. These issues are, however, a research topic on their own and will not be discussed further in this study.

Demographics In total we received 70 usable responses, which total a 35% response rate from the participants. From these users 93% are Finnish and 7% nonFinnish, 13% are female and 87% male. 86% of the respondents are students and 14% staff of the Aalto University. The different schools of the Aalto University are represented as follows: HSE 16%, TAIK 6% and TKK 78% (the data collection started as a TKK effort and thus TKK is overrepresented). The average birth year of the respondents is 1984.

Results Figure 6 presents how the respondents valued the selected set of incentives. As hypothesized the device and voucher lotteries were valued the

Figure 6. Importance of participation incentives



most as very important incentives. However, if we look at both very important and important categories, there are also three other research related incentives that the participants value highly. This indicates that the respondents might be interested in research activities also in general. Other incentives that the respondents suggested included some sort of recognition of participants (e.g., with badges), and guaranteed discounts for participants instead of lotteries, for example. Figure 7 presents the innovativeness of the respondents measured with a scale adapted from Goldsmith et al. (1995). In the upper three questions disagreeing denotes innovativeness and in the lower three questions it is the other way around. Thus we can conclude from these statistics that the majority of the respondents belong to the innovator / early adopter category of mobile phone / service users as H3 suggested.

Figure 8 presents the general privacy concern levels of the respondents. If the respondents had three or four privacy concerned answers they were Privacy Fundamentalists (PF), if they had two privacy concerned answers they were Pragmatic (P) and if they had only one or no privacy concerned answered they were Unconcerned (U). The groups were represented in our study with 33%, 23% and 44%, respectively. Thus we can conclude that the majority of the respondents belong to the Unconcerned and are generally trustful of organizations collecting their personal information as suggested by H2. Compared to studies by Kumaraguru and Cranor (2005) using the same privacy index on consumer privacy in general (PF 25%, P 57%, U 18%) and health information privacy (PF 13%, P 45%, U 42%) our respondents seem to be more divided between the extremes, however.

Figure 7. Innovativeness of the participants



Figure 8. Privacy concern levels of the participants

The questionnaire study is not meant to be an extensive statistical analysis, but is rather used as a means to segment the participants and use that information in developing the data collection process itself. A similar approach was used by Kiukkonen et al. (2010) in segmenting their data collection participants.

DISCUSSION AND FUTURE RESEARCH The data collection implementation including data acquisition and privacy solutions applied in the OtaSizzle project presents an example case of how these kinds of measurements can be implemented and what the participants’ attitudes towards them are. This is by no means an ultimate solution for how the implementation should be conducted, but as long as there are no standards regarding these issues available, all the case studies contribute to the body of knowledge needed in standardization. When compared to other similar research conducted in the area of smartphone measurements and

reviewed in the related work chapter, this article extends the literature by providing also participant attitudes in addition to the process description itself. As a proof of concept, the handset-based data collected in the OtaSizzle project have already been utilized in studying social relations (Karikoski & Nelimarkka, 2011), substitution in smartphone communication services (Karikoski & Luukkainen, 2011) and diversity and context in smartphone usage sessions (Soikkeli et al., 2011). The questionnaire results indicate that, as hypothesized, most of the participants are innovator or early adopter type of users when considering mobile phones and services. A similar research result has also been reported by Kiukkonen et al. (2010) who discovered that in their handset-based data collection campaign high involvement participants were overrepresented. These users are more prone to replacing one’s mobile phone and use the services and functions of their mobile devices. The innovativeness of the participants together with the low number of participants makes the generalization of the results challenging, but



as discussed before the usage patterns of these participants may reflect the situation that will take place this or next year among the majority of subscribers. Furthermore, these early adopter users are a salient user group when analyzing services that are not yet adopted by the average user (such as the mobile Internet communication services studied by Karikoski and Luukkainen (2011), for example). The participants are also naturally networked to some extent (as demonstrated by Karikoski and Nelimarkka, 2011) which enables studying social networks among the participants which would be a lot more difficult with a random sample. Thus we do not see the challenges that the innovativeness of the users and the networked nature of the sample create to the external validity of the results as a problem, but rather as a possibility to study new innovative mobile services and social networks. The general privacy concern level among the participants is interesting, because they are divided among the extremes. Most respondents are trustful and unconcerned of privacy issues, but the third of respondents also report being generally distrustful towards organizations that ask for their personal information. This suggests that these users have more important factors affecting the decision to join the measurements than privacy. These factors are, however, a research topic on their own and are not discussed further in this article. As discussed before this research setting enables a number of future research items to study in the area of social network analysis and communication media usage, for example. Although the number of participants is considerably low, there are more traditional research methods that can be used as complements when studying the community in more detail. These methods include questionnaires and interviews, for instance. The aim is to increase the number of participants with additional recruitment rounds in the future as well. Because the data collection is implemented as a longitudinal panel there is also a possibility to study the dynamics of the social networks, although the panel attrition bias reduces the size of the network that can be used. A more detailed understanding of de-

vice usage in terms of application usage inside the community, for example, is also possible. This way we do not have to rely solely on the reported behavior of the users acquired through questionnaires, but rather we get the actual usage as measured with the data collection software and can deepen our understanding with questionnaire studies. Because of the nature of the OtaSizzle project, there are also data available on the services developed in the project. The data from these services can be combined with the handset-based data, because the user groups overlap. This enables comparing the social networks and communication habits inside the services and through the handsets as was done by Karikoski and Nelimarkka (2011). We have also developed a context detection algorithm to derive the context of usage from the data. Previously the context has been mapped to four different contexts, i.e., home, office, on the move and abroad and has relied mostly on cell identifiers and time (Jimenez, 2008). More recently, however, Soikkeli (2011) has developed the algorithm further and added more accurate location information by fingerprinting also the WiFi access points sensed by the devices. This context information has already been used in studying smartphone usage sessions (Soikkeli et al., 2011) and will be used in analyzing contextual usage patterns of smartphone communication services in the future. Regarding the data collection process itself, there are several areas in need of development and future research. First of all users could have a choice on what data are collected about them. At the moment the same data are collected from all of the participants with device-specific differences. An option could be offered to the users to delete their data themselves without a need for Aalto University to act as a middle man. It has to be noted, however, that the more control the users have on the data, the more it restricts the possibilities of research. Regarding the incentives and the value that the data collection creates to the users, the data could also be made available to the users through web profiles. Thus the users could benefit from participating by studying their own usage profiles online.



For researchers interested in following a similar data collection approach as the one presented in this article, we want to highlight some of the major lessons learnt during our handset-based data collection process. First of all the process is iterative, as the smartphones are equipped with more and more sensors. The process follows a similar structure as the “deploy-use-refine” model by Miluzzo et al. (2010). Furthermore, when you collect the first data sets you get more insight into how you could utilize them better and become aware of data that you might need but have not collected. Not all sensor data can be collected from the smartphones; however, as the battery capacity is limited. Thus it first needs to be planned carefully what data types are to be collected so that they complement each other as well as possible, but at the same time you need to be prepared to modify the collected data types when appropriate. Moreover, you need to be careful when burdening the participants, e.g., with experience sampling and provide them with strong enough incentives for participation. We believe that increasing transparency and trust are essential for the success of these kinds of measurements in the long-term, especially as there have been concerns regarding the leakage and misuse of personal information by some mobile platforms as discussed before.

As the survey suggests, the participants are innovator or early adopter type of users when considering mobile phones and services and are on average less concerned about privacy issues than the general consumer public, for example. Regardless of the current problems with the external validity of the results and data collection implementation challenges, these measurements have huge potential. This has been demonstrated not only with the studies utilizing the data collected in the OtaSizzle project, but also with other similar data collection efforts in academia and industry.

CONCLUSION

REFERENCES

This article presented the process of collecting handset-based data in the OtaSizzle project of the Aalto University in Finland and participant attitudes about them. The literature review on other similar data collection efforts in academia and industry together with the OtaSizzle process and survey results contribute to the body of knowledge regarding measurements conducted with smartphones. This body of knowledge needs to be increased since there are no standards available on how to implement these kinds of measurements and how to deal with the behavioral data that are produced by the participants, especially in terms of privacy.

Aad, I., & Niemi, V. (2010). NRC data collection and the privacy by design principles. In Proceedings of the International Workshop on Sensing for App Phones (PhoneSense2010).

ACKNOWLEDGMENT This work has been supported by the OtaSizzle research project that is funded by Aalto University’s MIDE program and Helsinki University of Technology TKK’s ‘Technology for Life’ campaign donations from private companies and communities. The work was carried out in the Econ@Tel COST605 context with support from the MoMIE project and the Future Internet Graduate School FIGS. The author wishes to thank MobiTrack Innovations Ltd. for providing the mobile audience measurement platform. The sponsoring from Nokia and Elisa to this work is also acknowledged.

Beach, A., Gartrell, M., & Han, R. (2010). qAnon: Rethinking anonymity for social networks. In Proceedings of the Second IEEE International Conference on Social Computing (Socialcom2010) (pp. 185-192). Boase, J., & Kobayashi, T. (2011). Mobile communication networks: An exploratory study using the Communication Explorer smartphone application. In Proceedings of the International Communication Association Conference (ICA2011).



Chronis, I., Madan, A., & Pentland, A. (2009). SocialCircuits: The art of using mobile phones for modeling personal interactions. In Proceedings of the 6th Workshop on Machine Learning for Multimodal Interaction (ICMI-MLMI ’09). Eagle, N. (2009). Engineering a common good: Fair use of aggregated, anonymized behavioral data. In Proceedings of the IEEE Conference on Engaging Data. Eagle, N. (2011). Mobile phones as sensors for social research. In Hesse-Biber, S. N. (Ed.), Emergent technologies in social research (pp. 492–521). New York, NY: Oxford University Press. Eagle, N., & Pentland, A. (2006). Reality mining: Sensing complex social systems. Personal and Ubiquitous Computing, 10(4), 255–268. doi:10.1007/ s00779-005-0046-3 Eagle, N., Pentland, A., & Lazer, D. (2009). Inferring social network structure using mobile phone data. Proceedings of the National Academy of Sciences of the United States of America, 106(36), 15274–15278. doi:10.1073/pnas.0900282106 Enck, W., Gilbert, P., Chun, B.-G., Cox, L. P., Jung, J., McDaniel, P., & Sheth, A. N. (2010). Taint-Droid: An information flow tracking system for realtime privacy monitoring on mobile phones. In Proceedings of the 9th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’10). European Commission. (2010, November 4). Consultation on the Commission’s comprehensive approach on personal data protection in the European Union (Draft Doc. COM(2010) 609/3). Brussels, Belgium: Author. Falaki, H., Lymberopoulos, D., Mahajan, R., Kandula, S., & Estrin, D. (2010a). A first look at traffic on smartphones. In Proceedings of the Internet Measurement Conference (IMC2010). Falaki, H., Mahajan, R., & Estrin, D. (2011). SystemSens: A tool for monitoring usage in smartphone research deployments. In Proceedings of the 6th ACM International Workshop on Mobility in the Evolving Internet Architecture (ACM MobiArch 2011). Falaki, H., Mahajan, R., Kandula, S., Lymberopoulos, D., Govindan, R., & Estrin, D. (2010b). Diversity in smartphone usage. In Proceedings of the 8th International Conference on Mobile Systems, Applications, and Services (MobiSys ‘10).

Froehlich, J., Chen, M. Y., Consolvo, S., Harrison, B., & Landay, J. A. (2007). MyExperience: A system for in situ tracing and capturing of user feedback on mobile phones. In Proceedings of the 5th International Conference on Mobile Systems, Applications and Services (Mobisys ‘07). Goldsmith, R. E., Freiden, J. B., & Eastman, J. K. (1995). The generality/specificity issue in consumer innovativeness research. Technovation, 15(10), 601–612. doi:10.1016/0166-4972(95)99328-D Hasu, T. (2010). ContextLogger2 - A tool for smartphone data gathering (Tech. Rep. No. 2010-1). Espoo, Finland: Helsinki Institute for Information Technology HIIT, Aalto University. Hay, M., Miklau, G., Jensen, D., Towsley, D., & Weis, P. (2008). Resisting structural re-identification in anonymized social networks. Very Large Data Base Endowment, 1(1). Jimenez, B. (2008). Modeling of mobile end-user context (Unpublished master’s thesis). Helsinki University of Technology, Espoo, Finland. Karikoski, J., & Luukkainen, S. (2011). Substitution in smartphone communication services. In Proceedings of the 3rd International Workshop on Business Models for Mobile Platforms (BMMP2011). Karikoski, J., & Nelimarkka, M. (2011). Measuring social relations with multiple datasets. International Journal of Social Computing and Cyber-Physical Systems, 1(1), 98–113. doi:10.1504/ IJSCCPS.2011.043606 Kiukkonen, N., Blom, J., Dousse, O., Gatica-Perez, D., & Laurila, J. (2010). Towards rich mobile phone datasets: Lausanne data collection campaign. In Proceedings of the 7th International Conference on Pervasive Services (ICPS2010). Kivi, A. (2009). Measuring mobile service usage: Methods and measurement points. International Journal of Mobile Communications, 7(4), 415–435. doi:10.1504/IJMC.2009.023692 Kosta, E., & Dumortier, J. (2008). Searching the man behind the tag: Privacy implications of RFID technology. International Journal of Intellectual Property Management, 2(3), 276–288. Kumaraguru, P., & Cranor, L. F. (2005). Privacy indexes: A survey of Westin’s studies (Tech. Rep. No. CMU-ISRI-5-138). Pittsburgh, PA: Institute for Software Research International, School of Computer Science, Carnegie Mellon University.



Lane, N., Miluzzo, E., Lu, H., Peebles, D., Choudhury, T., & Campbell, A. (2010). A survey of mobile phone sensing. IEEE Communications Magazine, 48(9), 140–150. doi:10.1109/MCOM.2010.5560598

Raento, M., Oulasvirta, A., & Eagle, N. (2009). Smartphones: An emerging tool for social scientists. Sociological Methods & Research, 37(3), 426–454. doi:10.1177/0049124108330005

Langheinrich, M. (2001). Privacy by design – principles of privacy-aware ubiquitous systems. In G. D. Abowd, B. Brumitt, & S. Shafer (Eds.), Proceedings of the International Conference on Ubiquitous Computing (LNCS 2201, pp. 273-291).

Raento, M., Oulasvirta, A., Petit, R., & Toivonen, H. (2005). ContextPhone: A prototyping platform for context-aware mobile applications. IEEE Pervasive Computing, 4(2), 51–59. doi:10.1109/MPRV.2005.29

Lehväslaiho, K. (2009). A living lab experimentation environment of mobile applications (Unpublished master’s thesis). Helsinki University of Technology, Espoo, Finland. Lohse, G. L., Bellman, S., & Johnson, E. J. (2000). Consumer buying behavior on the Internet: Findings from panel data. Journal of Interactive Marketing, 14(1), 15–29. doi:10.1002/(SICI)15206653(200024)14:13.0.CO;2-C Mäntylä, M. (2009). Building an experimentation platform for mobile social interaction. In Proceedings of Mobile Living Labs 09: Methods and Tools for Evaluation in the Wild (MobileHCI MLL ’09). SizzleLab. Miluzzo, E., Lane, N., Lu, H., & Campbell, A. T. (2010). Research in the app store era: Experiences from the CenceMe app deployment on the iPhone. In Proceedings of the First International Workshop Research in the Large: Using App Stores, Markets, and other wide Distribution Channels in UbiComp Research. Miluzzo, E., Lane, N. D., Fodor, K., Peterson, R. A., Lu, H., & Musolesi, M. …Campbell, A. T. (2008). Sensing meets mobile social networks: The design, implementation and evaluation of the CenceMe application. In Proceedings of the 6th ACM Conference on Embedded Networked Sensor Systems (SenSys ‘08). Nokia. (2007, October). Nokia Smartphone360 Study - Responding to what the user wants and needs. Retrieved April 27, 2011, from http://www. nokia.com/NOKIA_COM_1/Press/twwln/press_kit/ Nokia_Smartphone360_study_Press_Backgrounder_October_2007.pdf OECD. (1980, September 23). Recommendations of the council concerning guidelines governing the protection of privacy and transborder flows of personal data (Doc. C58). Paris, France: OECD. Pentland, A. (2009). Reality mining of mobile communications: Toward a new deal on data (The Global Information Technology Report 2008-2009). World Economic Forum.

Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York, NY: Free Press. Shepard, C., Rahmati, A., Tossell, C., Zhong, L., & Kortum, P. (2010). LiveLab: Measuring wireless networks and smartphone users in the field. ACM SIGMETRICS Performance Evaluation Review, 38(3), 15–20. doi:10.1145/1925019.1925023 SizzleLab. (2011). OtaSizzle overview. Retrieved April 27, 2011, from http://www.sizzlelab.org/ content/otasizzle-overview Smith, E. (2010). iPhone applications & privacy issues: An analysis of application transmission of iPhone Unique Device Identifiers (UDIDs) (Tech. Rep.). Lewisburg, PA: Department of Library & Information Technology, Bucknell University. Soikkeli, T. (2011). The effect of context on smartphone usage sessions (Unpublished master’s thesis). Aalto University, Espoo, Finland. Soikkeli, T., Karikoski, J., & Hämmäinen, H. (2011). Diversity and end user context in smartphone usage sessions. In Proceedings of the 5th International Conference on Next Generation Mobile Applications, Services and Technologies (NGMAST2011). Suhonen, E., Esko, N., Juho, M., Ville, S., Seppo, T., Antti, V., & Julia, W. (2009). Kassi: Everyday favors in social media - Matching resources by means of campussourcing. In Proceedings of the European Conference on Computer Supported Cooperative Work (ECSCW’09). ter Hofte, G. H. (2007). What’s that hot thing in my pocket? SocioXensor, a smartphone data collector. In Proceedings of the Third International Conference on e-Social Science (E-Social Science 2007). Verkasalo, H. (2009b). Contextual patterns in mobile service usage. Personal and Ubiquitous Computing, 13(5), 331–342. doi:10.1007/s00779-008-0197-0 Verkasalo, H. (2008). Dynamics of mobile service adoption. Journal of Business Research, 4(3), 40–63. doi:10.4018/jebr.2008070103



Verkasalo, H. (2009a). Handset-based analysis of mobile service usage (Unpublished doctoral dissertation). Helsinki University of Technology, Espoo, Finland.

Verkasalo, H. (2010b). Analysis of smartphone user behavior. In Proceedings of the 9th International Conference on Mobile Business and the Ninth Global Mobility Roundtable (ICMB-GMR 2010).

Verkasalo, H. (2010a). A measurement framework of mobile service adoption. International Journal of Handheld Computing Research, 1(2), 17–35. doi:10.4018/jhcr.2010040102

Juuso Karikoski works as a Researcher and Doctoral Candidate at the Department of Communications and Networking at Aalto University in Finland. He also holds a position at the Future Internet Graduate School FIGS. His research interests include handset-based measurements, mobile communication service usage and social network analysis. Especially he studies how different mobile communication services are used depending on purpose, context and type of social relation and how the related social networks differ. Karikoski has been actively publishing in academia and is expected to graduate as a Doctor of Science in Technology in 2013. Currently Karikoski holds a MSc (Technology) from the Helsinki University of Technology.


Handset-Based Data Collection Process and Participant Attitudes

Handset-Based Data Collection Process and Participant Attitudes

Suggest Documents

Attitudes towards Autonomous Data Collection and ... - CiteSeerX

Participant recruitment and data collection through Facebook: the role ...

Online Data Collection for Psychotherapy Process

Towards Process-Driven Mobile Data Collection Applications:

Background Objectives Design Data Collection & Analytic Process ...

Treatment Process and Participant ... - OMICS International

Electronic Process for Pressure Ulcer and Restraint Data Collection ...

Data Collection and Analysis

Data Collection and Sampling

Validation and Data Collection

Participant Cadre Data Entry Form

Innovative Approaches towards Data Collection Process for New

P4P Process Round Two Report Data collection period - Google Sites

Non-invasive software process data collection for expert ... - CiteSeerX

VPlan - Ontology for Collection of Process Verification Data

Oslo_090311_Data collection process with focus on data ... - DRGforum

Data Collection, Data Quality, Ontologies and ... - proDataMarket

Sampling data and data collection in qualitative

Participant 1 Participant 2 Participant 3 Participant 4 ... - PLOS

Data Collection and Sampling

Separating Mnemonic Process From Participant and ... - Michael Pratte

Separating Mnemonic Process From Participant and Item Effects in the ...

Using a scientific literacy cluster to determine participant attitudes in ...

Using a scientific literacy cluster to determine participant attitudes in ...