White Paper

Tools for Improving Assessment through Real Time Data Collection October 2007 Dawn G. Gregg Assistant Professor, University of Colorado Denver President, Developing Minds Software

White Paper

The DDtrac Collective Intelligence Application Dawn G. Gregg* [email protected]

A collective intelligence application is one that harnesses the knowledge and work of its users to provide the data for the application and to improve its usefulness. The most hyped examples of collective intelligence applications have been labeled as “Web 2.0” applications. Web 2.0 is an amorphous term used to define a computing paradigm that uses the Web as the application platform and facilitates collaboration and information sharing between users [O’Reilly 2005]. Classic examples of Web 2.0 applications include: •

Wikis, collective Web sites that allow users to add, edit, and delete content;

•

Blogs, a Web journal that lets users post news or comments;

•

Social network services help build and verify online social communities that share common interests; and

•

Social bookmarking allows users to share bookmark for websites and organize them with tags.

Web 2.0 sites are database-driven and are considered to be “infoware,” in that, they are data intensive and the more data they contain the more valuable they become [McFedries 2007]. Much of what has been published on Web 2.0 to date has focused on using these new Web 2.0 tools in innovative ways. For example, in early 2005 Motorola created a corporate collaboration infrastructure which includes instant messages (12 million per day) and blogs (2,600 corporate wide), and wikis (3,200 corporate wide). After 18 months of use, Motorola had Motorola's collaboration infrastructure contained 17TB of searchable data [Gibson 2006]. Microsoft and IBM are linking Web 2.0 capabilities with enterprise software. These new tools will allow companies to manipulate and tag their data and to create internal social networks and virtual teams.

© 2007 Developing Minds Software

Collective intelligence is a fundamentally different way of viewing how applications can support human interaction and decision making. Most pre-Web 2.0 applications have focused in improving the productivity or decision making of the individual user. The emphasis has been on providing the tools and data necessary to fulfill a specific job function. Under the collective intelligence paradigm, the focus is on harnessing the intelligence of groups of people to enable greater productivity and better decisions than are possible by individuals working in isolation. This suggests that software developers need to have different ways of thinking about how their how software might be used and what features would enable better visualization and use of information among groups of people. The new breed of collective intelligence applications needs to center around user defined data that can be reused to support decision making, team building, or to improve understanding of the world around us. The users of these systems should play a central role in defining what data is important and how the data is used. The essential features of collective intelligence applications are similar to the design patterns for Web 2.0 applications originally suggested by Tim O’Reilly (2005) except that collective intelligence applications can be custom applications designed for small highly specialized domains instead of the larger Web audience served by most Web 2.0 applications (see Table 1).

1

Table 1: Collective Intelligence Application Requirements (adapted from O’Reilly 2005, Gregg 2007) 1. Task specific representations: Domain specific collective intelligence applications should support views of the task that are tailored to the particular domain. 2. Data is the key: Collective intelligence applications are data centric and should be designed to collect and share data among users. 3. Users Add Value: Users of collective intelligence applications know the most about the value of the information it contains. The application should provide mechanisms for them to add to modify or otherwise enhance the data to improve its usefulness. 4. Facilitate Data Aggregation: The ability to aggregate data adds value. Collective intelligence applications should be designed such that data aggregation occurs naturally through regular use. 5. Facilitate Data Access: The data in collective intelligence applications can have use beyond the boundaries of the application. Collective intelligence applications should offer web services interfaces and other mechanisms to facilitate the re-use of data. 6. Facilitate Access for All Devices: The PC is no longer the only access device for internet applications. Collective intelligence applications need to be designed to integrate services across handheld devices, PCs, and internet servers. 7. The Perpetual Beta: Collective intelligence applications are ongoing services provided to its users thus new features should be added on a regular basis based on the changing needs of the user community.

The processes involved in designing and implementing specialized collective intelligence applications are discussed below in the context of DDtrac, a web-based application that allows for the easy collection and summary of special education data. The DDtrac Application To illustrate how features of collective intelligence systems can be combined and used in special purpose applications, a system that combines distributed data capture, commenting, tagging, a wiki and a blog was created. DDtrac is an information system designed to


collect and summarize information that is used to improve decisions related to education and therapy options for special needs children. The heart of DDtrac is a web-based application that allows for the easy collection and summary of special education data. Figure 1 shows the DDtrac system architecture follows a basic hierarchical structure with functionality grouped into four major areas: data entry, creating goals and objectives, data analysis, and administration. The program is designed to allow data input from a Mac, a PC or a handheld device with a connection to the Internet.

2

Figure 1: DDtrac Architecture

applications that support the data collection and analysis needs of this sector.

The special education domain was selected for this study for three reasons. •

•

First, special education students often have many people involved in their education and therapy team. Students can receive services in school, in home therapy programs, in clinical outpatient environments and from consultants from other regions of the country. These practitioners rarely have time to communicate details related to student progress which can lead to an inefficient duplication of effort, gaps in treatment or team members working towards conflicting goals. Second, special education literature emphasizes the importance of using data collection and analysis procedures to monitor academic, social and behavior progress of students with intensive special education needs [e.g. Deno 2003, Gunter et. al. 2003, Lovaas 1987]. However, currently there are no


•

Third, there is the opportunity to harness the collective intelligence of these practitioners to identify patterns of behavior and best practices, for both individual students as well as groups of students with similar disabilities.

This suggests a need for software applications that can simplify the data collection and analysis activities of special education practitioners and harness their collective intelligence to improve their ability to make decisions about the children they serve. The DDtrac system serves two primary purposes. First it serves as a communication medium for therapists and teachers so that they know what to do when they sit down to work with the special needs child. Second it collects data and provides data analysis tools to enhance the ability to

3

assess the adequacy of student progress and determine whether and when instructional adjustments are necessary. The DDtrac system, shown in Figure 1, consists of four functional areas:

•

•

•

•

The data entry section allows four different types of data entry: instructional objective and target data, observed social data, behavior data and narrative comments. The goals and objectives section allows the creation and maintenance of longterm goals tailored to the needs of the individual student and shorter term objectives that define the activities involved in the day to day treatment and education of the child. The data analysis section includes reporting and charting. These features make it easy for special education teachers and therapists to examine student progress and modify student's objectives and targets to maximize a student's learning outcomes. The administration section contains additional functionality that was included to meet other identified needs (e.g. to manage access to data).

Distributed Data Entry. Special education instruction frequently occurs in locations throughout a school (e.g. the special education classroom, specialized therapy rooms, the regular education classroom, in the gym or on the playground) and can also occur at home and in the community. Similar to traditional Web 2.0 applications, DDtrac uses the Web as the application platform which allows data entry forms customized to the individual student to be accessed from anywhere there is an internet connection. Once the user has


logged in to DDtrac they can choose to take data in two different ways: either online using a wired PC or any wireless device OR offline, using a downloadable web form which can be uploaded later when a network is available. Figure 2 shows a series of data collection screens as would be displayed on a standard PC. Another way applications can support distributed data entry is by supporting a wide variety of data input devices. Using the web as a delivery platform is one way to support multiple platforms. However, to support handheld devices in addition to traditional PCs, DDtrac also uses style sheets targeted at different devices (e.g. mobile devices) and has data entry screens designed for devices as small as 320 pixels wide. In addition, the quantitative data entry is all accomplished using standard HTML form elements and were optimized for use with a stylus, the common data entry tool on many handheld devices. If the user does not have access to the Internet OR if the user has a poor/slow Internet connection they can also use offline data entry. In offline data entry mode all of the data entry forms for the selected students are downloaded into a single long page that has links to allow the user to quickly navigate to specific locations within the page. The data entered in the offline data entry page is automatically saved to cookies every 2 minutes and remains saved until the data is successfully uploaded. If the user accidentally closes the page they have to just reconnect to the Internet and download the offline data entry page (again) and the page will refill with the data they have already collected from the saved cookie.

4

Figure 2: DDtrac Instructional Data Entry1 Student-Centric Blog. Qualitative data is an important part of the information exchanged in many collective intelligence environments. For example, special education teachers use “back and forth books” to communicate daily notes to parents and parents write back to teachers about events at home that might impact learning in the special education classroom. In Applied Behavior Analysis1 (ABA) programs daily communication needs are met through handwritten daily notes to parents and other practitioners. Qualitative data in special education programs provides a description of the child 1

Applied Behavioral Analysis is an approach to teaching behaviors and cognitive skills to children with autism and other developmental disabilities that uses careful monitoring and positive reinforcement or prompting to teach each step of a skill. Data collection typically consists of a designation as to whether a response is correct, incorrect, correct but prompted or if no response was given. Qualitative notes are also taken to communicate major difficulties or successes [Lovaas 1987]. © 2007 Developing Minds Software

in context. Qualitative data is frequently the only way to capture the complexity and the transactional interaction between the setting and the student’s performance or behavior [Schwartz & Olswang 1996]. Observational comments reflect the practitioner's attempt to create a written account of what he or she hears, sees, experiences, and thinks in the course of observing the child in a particular context. DDtrac allows the capture of observation notes related to instructional activities, during social interactions and following behavior episodes. These observation notes are stored in a studentcentric blog that includes the date and time they were recorded, the name of the practitioner making the comment, and the name of the student the comment is being made about. The observational notes captured in DDtrac can be both descriptive and interpretive. The descriptive notes represent the practitioner's best efforts to record what is occurring in the context of the therapy session (e.g. describe student

5

mood and overall performance on tasks). The qualitative observations can also include the practitioner’s interpretation of how or why certain behaviors unfolded as they did [Schwartz & Olswang 1996]. DDtrac automatically displays the five most recent days of observations about a particular student as soon as the practitioner selects a student to work with. These qualitative notes are an important mechanism for communicating recent changes in the child and in the child’s educational programs between distributed team members. Commenting. Similar to traditional blogs, the commenting feature allows DDtrac users to comment on observations made by others. The comments are attached to a particular student-centric blog post and allow users to share experiences and provide suggestions related to issues raised about a student. It allows for a dialog between users so that the approaches that work best with a given student can be identified and adopted by the entire education team. The ability to comment and share insights is critical to DDtrac's support of the collective intelligence of special education groups. Tagging. In paper-based special education data collection environments much of the qualitative data found in the daily comments are lost within days of capturing it. The volume of data generated in the special education programs of developmentally disabled children can be overwhelming. For example, more than 100 pages of notes were generated in the home therapy data for a single student with autism during a one month period. Performing any meaningful analysis of this qualitative data (which can be accumulated over periods of years) is virtually impossible. Tagging is one mechanism that can be used in collective intelligence applications to improve the usefulness of both qualitative and quantitative data. For example, the DDtrac system supports different types of tagging for different types


of data. Every time a practitioner works on a particular objective with a student they take quantitative data related to the student’s performance on individual instructional targets. In addition, they can “tag” the entire work activity to describe the child’s overall mood during the activity. These predefined tags are selected from a dropdown list and are designed to allow practitioners to quickly convey the student’s mood to others and allow later assessment of the impact of mood on the student’s performance. Frequently special education students with intensive needs also have associated emotional and behavioral disorders [Gunter et. al. 2003]. Behavior data documents a student's patterns of behavior and is used to determine if efforts to minimize problem behaviors are effective. Tracking behaviors is often an important part of special education data collection. Behavior data collection in DDtrac includes a variety of quantitative data related to the behavior including the date, time, duration and number of behaviors counted during a behavior episode. It also allows the practitioner to define a set of tags to allow problem behaviors to be analyzed in a variety of ways. The practitioner defines tags for the types of behaviors being tracked for the student, the trigger that preceded the behavior, and the location where the behavior occurred. These tags can then be used to better understand patterns of behavior for a single student or for groups of students. For example, behavior tags can be used to generate a stacked bar chart (see Figure 3) showing how individual behaviors contribute to the number or duration of behaviors observed for a student. This allows users to see how individual behaviors are changing over time and determine if replacement behaviors are increasing and less desirable behaviors are decreasing.

6

Figure 3: Behavior Chart The final type of tagging available in DDtrac is the semantic tagging of the narrative comments taken as a part of the daily instructional, behavior or socialization observation notes taken by practitioners. These semantic tags closely resemble the tags common on many Web 2.0 sites (e.g. Flickr, Delicious, Blogger etc.). They are freely chosen keywords which allow for overlapping associations and that can be used for later retrieval and analysis of specific comments. For example, a student may exhibit a finger flicking behavior infrequently. The practitioner might note this in the daily notes along with other observations. Then, if the behavior becomes a problem, the practitioner could retrieve all of the comments tagged “flicking” to look for any patterns. Goals & Objectives Wiki. The education programs of developmentally disabled children are defined in an Individual Education Program (IEP), which establishes long-term goals and short-term objectives tailored to the needs of the individual student [Wright et. al. 2007]. The IEP also includes descriptions of the student’s current level of performance, strengths, and individual needs. In most schools this document includes input from several different people including the special education teacher, the regular education teacher, the therapy specialists, the student’s parents and external advocates. The IEP is an important document because it defines the direction for treatment to be taken for the upcoming year.


DDtrac includes a goals and objectives wiki to meet this need. The wiki utilizes a template that contains interconnected areas for goal creation, current level of performance discussions, strengths, and greatest needs. Using a wiki structure for creating IEPs allows practitioners with appropriate access permissions to add, edit, and delete goals as the new IEP evolves. It is also possible for current IEPs to reference past IEPs for the same student, as well as district standards or tests that the student should meet during the year. The ease with which wiki pages can be created and updated is essential for the success of the collaborative IEP generation tool. In addition, the ability to review changes before they are added to the document and to roll back changes that don’t meet the approval of the teacher or parents is essential in an environment where the IEP represents a legal contract between the school and the parents. The goals & objectives wiki allows IEP goals and objectives to be edited until the IEP is accepted then the approved goals and objectives are added to the data collection portion of DDtrac so data collection can begin. Support Wiki and Blog. Two other collective intelligence features were designed into the DDtrac application. A wiki is being used for all software documentation and a blog is used to communicate with users. Both are essential for feeding the collective intelligence of users back into the software. The documentation wiki allows developers to quickly add documentation related to new features. In addition, it allows users to edit the documentation themselves to clarify instructions that are not clear or to add documentation that the developers did not think to create. The blog allows the developers to post announcements about new features that have been added (essential in an environment where updates go live every two to three weeks) and solicits feedback from users on which new features they feel

7

will be most beneficial. Collective Intelligence in Practice DDtrac was deployed in an eighteen month field trial with one student with autism. The student participated in speech therapy, occupational therapy, ABA therapy, and socialization therapy known as Relationship Development Intervention in a home environment as well as receiving special education services at school. The practitioners working with the student rarely met in person and instead used the webbased DDtrac application for data collection and communication. Over the course of the eighteen month trial data was captured for 481 separate work sessions and included more than 50,000 individual pieces of data. The wiki and the blog were active for the final 6 months of the project. During the 6 months the wiki had 163 pages added and the blog had 33 posts documenting 12 software upgrades. All participants in the field study felt that DDtrac significantly enhanced their ability to take data and evaluate the performance of the student they were working with. They reported the following benefits to using DDtrac: 1. The distributed data collection features made data collection easier and faster. 2. The student centric blog enabled them to quickly understand the student’s recent behavior trends and better prepare for their own work sessions. 3. The goal & objective wiki enabled them to easily develop new goals and objectives as well as better understand the goals and objectives the student was currently working on. 4. The ability to analyze the data in a wide variety of ways enhanced their ability to assess student progress and made it easier to comply with mandatory reporting requirements.


5. The support blog and the wiki enabled them to learn more about DDtrac so that they could use it more effectively. The biggest benefit reported by the student’s parents was the ability to analyze long-term education and behavior patterns. The parents commented that they had implemented numerous interventions with their child and the data provided by DDtrac enabled them to have an unbiased measure of whether or not a particular intervention had an impact either on the educational outcomes or on the behaviors of their child. Conclusion This paper illustrates the benefits collective intelligence applications can provide in specialized domains. The DDtrac collective intelligence application allows special education data to be captured and shared more efficiently than the pencil and paper methods currently being used (Figure 4). In addition, its reporting and charting features allow this data to be analyzed comprehensively and quickly. This allows practitioners to spend more time working with their students. It will also help to provide more efficient treatment and education plans, and improve the outcomes of millions of children and adults with cognitive impairments. Use of a specialized collective intelligence application, like DDtrac, can potentially benefit organizations in a wide variety of domains (e.g. health care, outsourcing environments). The ability to apply the collective intelligence of individuals working on similar problems is an area that has just begun to be addressed by software developers; however, these systems will change the way information is shared and used and has the potential to dramatically improve decision making.

8

Figure 4: Replacing paper data collection with distributed data collection and data analysis

References 1. Deno, S.L. "Developments in CurriculumBased Measurement," The Journal of Special Education 37, 3 (March 2003) 184192. 2. Gibson, S. "Wikis are alive and Kicking in the Enterprise" eWeek.com, 20 Nov. 2006 (available at: http://www.eweek.com/article2/0,1895,2061 135,00.asp). 3. Gregg, D. "Developing a Collective Intelligence Application for Special Education," Working paper, (Sept. 2007) (available at http://carbon.cudenver.edu/~dgregg/researc h/greggCI.pdf). 4. Gunter, P. L., Callicott, K., Denny, R. K., and Gerber, B. L. Finding a place for Data Collection in Classrooms for Students with Emotional Behavioral Disorders, Preventing School Failure, 47, 1 (Fall 2003), 4-8.

5. Lovaas, O. I. Behavioral treatment and normal educational and intellectual functioning in young autistic children, Journal of Consulting and Clinical Psychology, 55, 1 (February 1987), 3-9. 6. McFedries, P. "The Web, Take Two," IEEE Spectrum (June 2006), 68 7. O'Reilly, T. “What Is Web 2.0: Design Patterns and Business Models for the Next Generation of Software,” O'Reilly Media, Inc., 30 Sept. 2005, (available at: http://www.oreillynet.com/pub/a/oreilly/tim/ne ws/2005/09/30/what-is-web-20.html) 8. Schwartz, I. S. and Olswang, L. B. Evaluating child behavior change in natural settings: Exploring alternative strategies for data collection, Topics in Early Childhood Special Education, 16; 1 (Spring 1996), 82101. 9. Wright, Peter and Wright, Pam “IEP Goals & Objectives: A Tactics and Strategy Session” 2007, (available at http://www.wrightslaw.com/advoc/articles/Ta ctics_Strategy_IEPs.html).

For more information please visit: http://developingmindssoftware.com Developing Minds Software P.O. Box 202046 Denver, CO 80220 USA 888-595-5195

Developing Minds Software™ and DDtrac™ are trademarks of Developing Minds Software, Inc.


9