A Framework for Evaluating the Impact of High-Bandwidth Internet ...

A Framework for Evaluating the Impact of High-Bandwidth Internet Provision and Use on Digital Literacy and Social Life Outcomes in Australia S. Dane, M. Fahey, C. Mason, R. van der Zwan, J. Tucker, D. Bradford, and C. Griffith

Abstract In this paper we present a framework for evaluating the social impact of high-bandwidth Internet provision and use in Australia. High-bandwidth Internet will be provided through a national broadband network to be rolled out gradually and which began in 2011. The framework is based around four key aspects: (1) identifying provision of the national broadband network as an intervention, (2) specification of important outcomes, (3) understanding the behavioural link between intervention and outcomes and (4) conduct of high-quality population-based empirical research. The framework is sufficiently flexible that it can be adapted and applied to various regions of Australia and, depending on the appropriate focus, to the conduct of different types of studies. It is hoped that, by focusing attention on the human behavioural link between high-bandwidth Internet provision and individual outcomes in the general community, we will be able to identify ways of promoting social inclusion and other benefits through appropriate use of the national broadband network.

Abbreviations ACBI CSIRO GIS G-NAF HBI

Australian Centre for Broadband Innovation Commonwealth Scientific and Industrial Research Organisation Geographic information system Geocoded National Address File High-bandwidth Internet

S. Dane (*) • M. Fahey • C. Mason • D. Bradford • C. Griffith Commonwealth Scientific and Industrial Research Organisation (CSIRO), P.O. Box 883, Kenmore, QLD 4069, Australia e-mail: [email protected] R. van der Zwan • J. Tucker Southern Cross University, Coffs Harbour, NSW, Australia H. Linger et al. (eds.), Building Sustainable Information Systems: Proceedings of the 2012 International Conference on Information Systems Development, DOI 10.1007/978-1-4614-7540-8_8, © Springer Science+Business Media New York 2013

101

102

S. Dane et al.

NBN RISIR TPB

1

National broadband network Regional Initiative for Social Innovation and Research Theory of Planned Behaviour

Background

The Australian government announced in 2009 that a national broadband network (NBN) would be established to make high-bandwidth Internet (HBI) available to all Australians. A wholly government-owned company, NBN Co, was created to deliver the roll-out of the network and aims to provide the NBN to 93 % of the Australian population through fibre optic cable with the remainder to connect by fixed wireless or satellite (http://nbnco.com.au). The Commonwealth Scientific and Industrial Research Organisation (CSIRO) is Australia’s national science agency whose mission includes conducting research for the benefit of Australian society, industry and the environment. CSIRO has established the Australian Centre for Broadband Innovation (ACBI) as a collaborative research initiative to develop and demonstrate socially and economically useful applications and services that can use next-generation broadband networks such as the NBN. One of ACBI’s main objectives is to develop an understanding of communities’ needs and capabilities with respect to services and applications that are enabled by the NBN. To this end, CSIRO and ACBI have allocated approximately $US 3 million over three years to undertake this work. The initiative will draw on expertise from social and economic research areas within CSIRO such as mathematical and statistical sciences, human services evaluation, population health research, energy demand management and economic planning. CSIRO is also partnering with other research organisations such as the Regional Initiative for Social Innovation and Research (RISIR) Centre at Southern Cross University in Coffs Harbour, to ensure that relevant locally based research is undertaken. This paper proposes a general framework for evaluating the impact of the NBN on digital literacy and social and well-being outcomes. Section two makes explicit the key components of the framework and states the major objectives of the initiative. Section three details our approach to measurement of connection bandwidth, digital literacy and other life outcomes and outlines the study designs for surveying individuals in our first target populations and planned statistical analyses. The final section discusses aspects requiring further work and considers extensions to other studies and other regions of Australia.

2

General Framework to Evaluate the Impact of the NBN

In developing a general framework, we broadly followed the NONIE guidelines that detail key steps in designing and conducting an impact evaluation [16]. These steps are considered in turn below.

A Framework for Evaluating the Impact of High-Bandwidth Internet Provision…

2.1

103

Identifying the Intervention

The intervention of interest is the NBN. The NBN is to be provided in a staged roll-out that began in May 2011. It is expected to be complete for about one third of the country (approximately 3.5 million households) by 2015 (http://nbnco.com.au). Although universal coverage is planned, individuals may choose either not to connect to the Internet or to remain connected through low-bandwidth information transfer rates. Bandwidth is dependent on factors such as type of NBN connection (fibre, wireless, satellite), distance to Internet service provider (in the case of nonfibre connections) and client-side hardware. There are two ways in which the impact of the NBN could be evaluated as an observational intervention. The first is provision of the NBN per se. For example, knowing that the NBN has been provided to one community and not (yet) to another makes it possible to compare the two communities, on average, in terms of social and/or life outcomes. The second is connection bandwidth measured as a transfer rate (Mb/s) of information. In this case knowing an individual’s connection bandwidth makes it possible to estimate the rate at which outcomes such as “digital literacy” (see Sect. 2.2 below) change per unit increase in bandwidth. Greater variation among connection bandwidths is expected after the introduction of the NBN.

2.2

Specification of Outcomes Valued by Stakeholders

Our team initiated an evaluation of broadband outcomes with consultation among interested parties and stakeholders at the national and regional levels and by a literature review to determine what is currently known and to identify important gaps in knowledge. These investigations highlighted the capability for individuals and communities to make use of NBN-enabled services and applications and to transfer skills from one Internet-usage domain to another. We will refer to this cluster of capabilities as digital literacy. In order to compare changes in the adoption and diffusion of Internet usage among Australian residents before and after the introduction of the NBN, it was considered crucial that standardised measures examining a range of Internet behaviours be incorporated across all survey applications. Critical areas identified were Internet usage for information seeking, buying and using services, communication and social networking, and entertainment and leisure. In addition to measuring behaviour change for the above core set of Internet domains across all regions, interested parties have highlighted the need to incorporate customised survey items to focus on Internet behaviour in areas of interest to a particular community. For example, our research partners at RISIR in Coffs Harbour have identified three important components to digital literacy in their region. These include the use of the Internet for (1) financial services, (2) local government services and (3) social/business networking. Through the utilisation of a

104

S. Dane et al.

behavioural model (see Sect. 2.3), this tailored approach can identify not only the changes in Internet behaviour for specific areas of regional interest but also the underlying determinants of that change. Consultation has revealed also the importance of identifying the impact of digital literacy (i.e. the adoption and diffusion of Internet usage) on broader life outcomes including health, education, community engagement, standard of living and general life satisfaction. We expect the concept of digital literacy to have general relevance and applicability to these life domains, even if its underlying components vary by community.

2.3

Understanding the Behavioural Link Between Intervention and Outcome

We propose adopting a model for human behaviour to facilitate an understanding of the impact of the NBN on outcomes of interest and their determinants. Over the last two decades, a rich body of research has emerged on consumer adoption and diffusion of technology innovations, resulting in the publication of a number of reviews and meta-analyses (e.g. [4, 12, 15, 22, 24, 25]). ICT adoption researchers have advocated the use of “Integrated Models”, which generally use the Theory of Planned Behaviour’s [1] core constructs as a foundation (e.g. [6, 11, 24]). The Theory of Planned Behaviour (TPB) emphasises that people’s behaviour are driven by their attitudes, social influences (i.e. norms) and their sense of control in relation to the behaviour. The predictive validity of TPB is well established [3]. The model has been employed to investigate human behaviour in a wide range of domains, e.g. water conservation, smoking cessation, blood donation, adoption of cleaner vehicles, household recycling and Internet and communication technology adoption. Most of the research applying the TPB to draw a link between the Internet and related behaviour has been conducted in the workplace and has been crosssectional in nature. Little, therefore, is known about links between provision of the Internet and related behaviours and outcomes in residential settings [7, 12]. There is also a dearth of research studying these effects over time and the effects of interventions [25]. Figure 1 depicts a model for human behaviour that integrates the TPB with other theoretical constructs found to be effective predictors of the frequency and diversity of Internet usage, such as “perceived usefulness” and “perceived ease of use” from the Technology Acceptance Model [9] and “relative advantage” from Innovation Diffusion Theory [20]. In keeping with recent integrated behavioural models (e.g. [13, 18]), it takes into account both cognitive and emotional determinants of attitudes. This theoretical framework is used to identify which factors (attitudes, norms or perceived control in the second column from the left in Fig. 1) drive or inhibit behavioural intentions, along with the critical beliefs that underpin these factors (first column from the left in Fig. 1). Once identified, the relevant beliefs could, for


105

Fig. 1 An integrated model for the determinants of human behaviour

example, be measured over time to examine their relationship with behaviour change across subgroups of individuals and/or locations. Moreover, this approach facilitates the targeting of core beliefs in intervention strategies (e.g. education, training, marketing) to affect change.

2.4

Qualitative and Quantitative Empirical Research Studies to Answer Specific Research Objectives

To draw the link between the NBN and NBN-enabled applications and broader life outcomes, a critical step is to collect high-quality, reliable and valid data on individuals, households and communities. In new fields of enquiry, research is often needed to know exactly what to study, to develop survey instruments and to validate them. Following this preliminary work, substantive studies must be designed to answer the major research objectives and must be feasible given the available resources and time frame. As previously noted, an understanding of the impacts of broadband requires longitudinal and experimental studies in household and/or community settings [6, 12]. We plan to focus on population-based research by recruiting representative samples from two regional Australian cities: Coffs Harbour and Port Macquarie. Roll-out of the NBN has begun in Coffs Harbour, although connections will not be possible

106

S. Dane et al.

before 2013. The roll-out has not begun in Port Macquarie and it is not among the early release areas.

3

Measurement of Broadband Connectivity, Digital Literacy (Internet-Related Behaviour) and Life Outcomes

Empirical conclusions depend heavily on the measurement of the intervention and the outcomes described in section two. Therefore, these aspects are described in more detail here.

3.1

Measuring Broadband Connectivity

It is difficult to directly observe individual broadband connectivity because (1) there is no standard definition of “broadband Internet” in terms of bandwidth transfer rate (Mb/s), (2) individuals may not know their transfer rate, and (3) for privacy reasons bandwidth transfer rate will be difficult to obtain and/or verify from ISPs. We therefore plan to investigate two approaches to measuring broadband connectivity. The first is to use a web-based broadband speed tester to directly measure download and upload speeds. The second approach is to consider high-bandwidth connectivity as an unobserved category, e.g. an individual is (likely to be) connected to the NBN or is (likely) not connected. A set of simple, but relevant questionnaire items can be used as data from which to estimate the probability of belonging to each connection category. Questionnaire items need to relate to aspects of Internet use that are easily understood, e.g. Do you use the Internet at home? Do you have a wireless connection at home? Do you download films from the Internet at home? Do you play online games at home? A statistical model is used to estimate connection category probabilities [17]. If the second approach is validated against the first, it could prove useful in settings where it is not possible to directly measure the bandwidth available to an individual, e.g. owing to a participant opting for the paper version of a survey, when a network connection is not available, or if the web-based speed tester is down for technical reasons.

3.1.1

Validation Study for Measuring Broadband Connectivity

This study will be designed to validate our direct and indirect approaches to the measurement of broadband connectivity. For the direct approach, an ad hoc broadband speed tester will be engineered in our laboratories and tested for accuracy on hardware and software platforms that are likely to be encountered by users in the


107

general population. Both the direct and indirect survey approach will then be trialled on a sample of residents selected from an NBN release area that has already been switched on to HBI. In this study it will be important to include a large proportion of individuals who have chosen (or who are highly likely to have chosen) to make use of HBI, and to contrast them with a comparable sample from the same area who are not highly likely to have chosen to make use of HBI. Given that initial uptake of NBN services is low (http://nbnco.com.au), representative sampling of the population would lead to selection of very few HBI users and would be an ineffective strategy. As an alternative, NBN users could be randomly selected from users subscribing to an NBNenabled application available in an early NBN release area. A control group could then be randomly selected from the same area. This type of study is sometimes referred to as outcome-dependent sampling and has efficiency advantages (in cost and precision) over random sampling study designs when the outcome is rare [5]. For the broadband speed tester, statistical analysis will focus on the willingness and ability of users to correctly access HBI and run the selected application. For the indirect survey approach, analysis of data would focus on (1) the interpretation of the indirect observed connection categories and (2) the accuracy of the survey approach in identifying connection categories by comparison with the “gold standard” broadband speed tester.

3.2

Measuring Digital Literacy and Its Determinants

Digital literacy is by definition a latent phenomenon which is not directly observable. Our proposal is to observe this phenomenon indirectly, through a set of questionnaire items concerned with Internet-related behaviour (e.g. frequency and diversity of Internet usage in various domains). As discussed previously, we propose to use one set of measures that will be kept constant for different regions in Australia, allowing for regional comparisons in terms of the impact of the NBN. A second set of measures will be used to focus on identifying change in Internet-usage behaviour for specific areas of interest to a given community. For this customised approach we propose to employ the integrated behavioural model using the core constructs of the TPB and, thereby, examine the underlying determinants of the Internet-related behaviour (e.g. evaluative beliefs). These will be conceived similarly as unobserved phenomena that are to be indirectly estimated. Structural equation models will be used for estimating latent constructs [23].

3.2.1

Telephone Interviews and Pilot Study

Before constructing a TPB questionnaire, it is imperative to first conduct a qualitative elicitation study to indentify people’s salient behavioural, normative and control beliefs (see left column Fig. 1) regarding engaging in the behaviour under

108

S. Dane et al.

consideration [2]. This information is then used to effectively word the items to be used in a quantitative survey. Transcripts of interviews need to be analysed by at least two researchers using qualitative data management software (e.g. NVivo) to facilitate flexible and dynamic coding of text data. The goal is to explore the data for themes emerging from the text and to code, sort and query the data. An audit trail for each theme coded will be created with continual access to the original data for cross validation of themes. Once the questionnaire items are devised, a quantitative pilot study should be conducted to test if the items are easily understood and relevant to the target population. This requires responses from a small but diverse community sample.

3.2.2

Validation Study

Major findings of quantitative studies critically depend on well-validated measurements. Therefore, and following any survey item modifications based on the outcome of the pilot study, the next objective is to conduct a validation study to assess the measurement properties of the survey instrument. A factor analysis of the data will be done to assess the reliability and validity of the items and hypothesised latent constructs. These constructs relate to the integrated behavioural model (utilising the TPB core constructs) as depicted in Fig. 1. Participants will be selected to be representative of the general population (see Sect. 3.1.1). Sample size requirements will be based on estimating latent digital literacy constructs with sufficiently good precision. Sample size for factor analysis depends on the number of survey items, the number of factors and the communalities. Allowing for nonresponse of 10–15 %, then to detect true population factor structure to a high degree, the estimation of four factors based on six survey items per factor would require a sample size of 220 subjects assuming communalities ranged between 0.2 and 0.4 [19].

3.2.3

Cross-Sectional Benchmark Survey

Once the survey measurements have been validated as described above, the next goal is to conduct the survey proper. This survey would be administered prior to the introduction of the NBN in various regions and, therefore, serve as a benchmark of Internet-related behaviour to compare with follow-up studies. Ideally, a survey of this nature should be conducted with a representative sample of the population. In theory, representativeness can be ensured only by using some form of probability sampling, e.g. random selection. We plan to achieve representativeness by sampling participants as follows. First, residential addresses will be randomly selected using a geographic information system (GIS) and a Geocoded National Address File (G-NAF). Second, individuals will be randomly selected from households (e.g. by asking that the person usually resident in each household having the next birthday participate in the study).


109

To achieve a high response rate, we plan to use several strategies, including a media campaign to raise awareness of the study, provision of options for survey participants (e.g. online or hardcopy, delivered directly to households), incentives for participation and training of data collectors to better inform perspective participants. Sample size requirements will be planned using the data collected for the earlier validation study and based on estimating association between digital literacy and broader life outcomes (see below) with sufficiently good precision.

3.2.4

Measuring Broader Life Outcomes

It is also important to identify the potential impact of Internet-related behaviour on a range of life outcomes. The benchmark survey described above will, therefore, include various established outcome measures such as those relating to emotional well-being [26], community engagement and respectful treatment [8], environmental mastery— i.e. perceived control over one’s life [21], skills and opportunities—and general satisfaction with life [10].

3.3

Randomised Control Trial

Employing the integrated behavioural model in conjunction with life outcomes will also allow us to examine the effectiveness of different educational strategies aimed at improving digital literacy within various communities. The aim would be to compare general NBN training with education initiatives tailored to the needs of the local community. In such a trial, conducted following the roll-out of the NBN, participants would be assigned to groups that receive one of the three different types of treatment. The government has produced a number of generic materials to help users make the most of the high-speed network. That material relies on self-directed learning. Access to that material, which includes implicitly any other materials users may find, forms treatment 1. Treatment 2 will be comprised of a set of generic training modules delivered to users individually or as small groups. The content of the generic modules will be pitched to illustrate both the capacity of the NBN and strategies for using the NBN across a range of activities (information seeking, buying and using services, communication and social networking, entertainment and leisure). Treatment 3 will provide for small user groups more tailored content: The units in this treatment will take into account the needs of the user group, the community and environment and will utilise wherever possible peer-to-peer trainers. Content for those groups will be tailored, in the first instance, to older users with no previous experience of the Internet; to those living in social housing with special needs or those receiving social benefits (long-term unemployed and so on); to those with some experience of the Internet but who would not normally use the Internet for all service, health, business and banking; and to high-end users (four groups). Those four groups dictate the participants to be allocated to the other treatments.

110

S. Dane et al.

To that end, participants meeting the trial inclusion criteria will be allocated at random and from across the Coffs Harbour fibre service area maps to an initial treatment group. The impact of the training protocol will be measured as described above and follow-up assessments will be carried out at 3 months and then at 12 months after treatment.

4

Future Extensions and Discussion

We have described here some principles of impact evaluation and how these will be applied in our planned investigations of NBN impacts on the lives of people in several Australian regions. Our broader objectives include the promotion of NBN benefits and the identification (and reduction) of negative impacts. An example of the latter is a possible widening of the social exclusion gap and greater social inequality in some communities following provision of the NBN. Some of the foreseeable benefits of the NBN have been described in detail by Hayes [14]. We hope that by focusing attention on the behavioural links between high-bandwidth Internet provision and individual outcomes in the general community, we will be able to identify ways of promoting social inclusion through the use of the NBN. We intend to achieve these ends by conducting high-quality population-based research with the participation of regional institutes and input from local experts. Our evaluation framework can be adapted to suit various regions in Australia and address different areas of interest in terms of Internet-related behaviour. We intend to pursue this extension of the present work in collaboration with research partners in universities and other institutions, so as to expand expertise and application. We hope also to extend and generalise our evaluation framework to meet the needs of policymakers at the national level, as proposed in a recent paper based on experience with the introduction of HBI in the United Kingdom [27].

References 1. Ajzen I (1991) The theory of planned behaviour. Organ Behav Hum Dec 50:179–211 2. Ajzen I (2006) Constructing a TPB questionnaire: conceptual and methodological considerations. http://people.umass.edu/aizen/pdf/tpb.measurement.pdf. Accessed 12 Jul 2012 3. Armitage CJ, Conner M (2001) Efficacy of the theory of planned behaviour: a meta-analytic review. Br J Soc Psychol 40:471–499 4. Bagozzi RP (2007) The legacy of the technology acceptance model and a proposal for a paradigm shift. J Assoc Inf Syst 8:244–254 5. Chen HY (2003) A note on the prospective analysis of outcome-dependent samples. J Roy Statist Soc Ser B 65:575–584 6. Choudrie J, Dwivedi YK (2004) Towards a conceptual model of broadband diffusion. J Comput Inform Tech 12:323–338 7. Choudrie J, Dwivedi YK (2006) Investigating factors influencing adoption of broadband in the household. J Comput Inform Syst 46:25–34


111

8. Dane SD, Stenner K, Hobman EV (2012) Predicting subjective well-being: material versus psycho-social needs. Report EP 124179. Commonwealth Scientific and Industrial Research Organisation (CSIRO), Australia 9. Davis FD, Bagozzi RP, Warshaw PR (1989) User acceptance of computer technology: a comparison of two theoretical models. Manage Sci 35:982–1002 10. Diener E, Emmons R, Larsen J, Griffin S (1985) The satisfaction with life scale. J Pers Assess 49:71–75 11. Dwivedi YK, Choudrie J, Brinkman WP (2006) Developing a survey instrument to examine consumer adoption of broadband. Ind Manag Data Syst 106:700–718 12. Dwivedi YK, Williams MD, Lal B, Mustafee N (2010) An analysis of literature on consumer adoption and diffusion of information system/information technology/information and communication technology. Int J Electron Gov Res 6:58–73 13. Fishbein M, Cappella JN (2006) The role of theory in developing effective health communications. J Commun. doi:10.1111/j.1460-2466.2006.00280 14. Hayes R (2011) Valuing broadband benefits. Institute for a broadband-enabled society. University of Melbourne, Melbourne 15. Jeyaraj A, Rottman JW, Lacity MC (2006) A review of the predictors, linkages, and biases in IT innovation adoption research. J Inf Technol. doi:10.1057/palgrave.jit.2000056 16. Leeuw F, Vaessen J (2009) Impact evaluations and development: NONIE guidance on impact evaluation. World Bank, Washington, DC 17. McLachlan GJ, Peel D (2000) Finite mixture models. Wiley, New York 18. Montano DE, Kasprzyk D (2008) Theory of reasoned action, theory of planned behaviour, and the integrated behavioural model. In: Glanz K, Rimmer BK, Viswanath K (eds) Health behavior and health education. Theory, research, and practice, 4th edn. Jossey-Bass, San Franciso, CA 19. Mundfrom DJ, Shaw DG, Lu Ke T (2005) Minimum sample size recommendations for conducting factor analyses. Int J Test 5:159–168 20. Rogers EM (1995) Diffusions of innovations. Free Press, New York 21. Ryff CD (1989) Happiness is everything, or is it? Explorations on the meaning of psychological well-being. J Pers Soc Psychol 57:1069–1081 22. Silva L (2007) Post-positivist review of technology acceptance model. J Assoc Inf Syst 8:255–266 23. Skrondal A, Rabe-Hesketh S (2004) Generalized latent variable modeling. Chapman and Hall, Boca Raton, FL 24. Venkatesh V, Morris MG, Davis GB, Davis FD (2003) User acceptance of information technology: toward a unified approach. Mis Quart 27:425–477 25. Venkatesh V, Davis FD, Morris MG (2007) Dead or alive? The development, trajectory and future of technology adoption research. J Assoc Inf Syst 8:267–286 26. Watson D, Clark LA, Tellegen A (1988) Development and validation of brief measures of positive and negative affect: the PANAS scales. J Pers Soc Psychol 54:1063–1070 27. Williams T (2011) Connecting communities: the impact of broadband on communities in the UK and its implications for Australia. Huawei Technologies Co. Ltd, Sydney