Improving classroom visual accessibility with ...

Improving Classroom Visual Accessibility with Cooperative Smartphone Recordings Raja S. Kushalnagar and Brian P. Trager Information and Computing Studies Rochester Institute of Technology Rochester, NY 14623-5604 {rskics, bptnbs}@rit.edu

An earlier version of this paper was presented at the 2011 IEEE International Symposium on Technology and Society (ISTAS) at Saint Xavier University in Chicago, Illinois (and printed in the 2011 ISTAS proceedings).

Abstract We propose a cooperative approach by students in recording lecture activities such that the classroom becomes more visually accessible for everyone, especially for deaf, hard of hearing and low-vision students. Students utilize their personal camera-equipped smart phones to capture and share their views of a visually inaccessible classroom to students’ devices. We show this approach virtually retrofits a classroom to make it a more visually accessible learning environment. This approach can be used in meetings to get and share optimal views of the meeting for all participants.

Keywords: Cooperative applications, accessible technology, multiple video streams, analysis

Introduction Deaf and hard of hearing students need full visual access to classroom lectures to benefit fully from them, and federal laws mandate this equal access. Educational institutions satisfy the law and meet this need by providing accessible services that translate auditory information to visual information, such as sign language interpreters or captioners. However, deaf and hard of hearing students’ graduation rate remains abysmal at 25% [1], in contrast to the nationwide hearing student graduation rate of 56% [2]. Federal law requires educational institutions to provide equal access for deaf and hard of hearing students, therefore accessible technology research focused on utilizing institutional and classroom infrastructure. However, this approach ignores the fact that deaf students are a low incidence disability and tend to be thinly and evenly spread. Most deaf students often are the only student in their class or even in the entire institution. For a single enrolled deaf student, it is cumbersome for institutional technical staff to support specific classroom or distance learning solutions. Also, deaf students typically attend lectures in multiple classrooms, yet throughout the semester, most classrooms in a given institution will not host a deaf student. Therefore, from both an institutional and student perspective, in terms of effort, availability and cost, the accessible technology should be centered on the student rather than the classroom.

SIGCAS Computers and Society, Volume 41, No. 2, December 2011

51

Student participation has evolved from passive viewing of lecture oratories with a single blackboard to active viewing of classroom lectures with multiple sources consisting of multimedia slides, whiteboard notes and interactive student participation. Although modern classroom technology has contributed to the visual dispersion of the multiple information sources, technology can reduce these barriers for the benefit of all participants, not just the deaf.

Figure 4: Video capture approaches: a) a single high definition camera at the back of the room, versus b) multiple cameras targeted at each information source.

Related Work Presentation Capture Formal lectures are usually recorded by professional staff to capture the active information sources. The camera operator is able to capture salient classroom information by selecting and zooming into the currently active classroom source of information. However, this approach is time and labor expensive and hence not practical for universal lecture recording. Also, professional video systems are not usually portable. The alternate approach of recording the entire lecture in high definition consumes a large amount of bandwidth to capture and transmit the high resolution video from the camera to the individual students. This approach does not solve the awkward camera angle problem, and does not offer personalized views for individual students. Moreover, using a single camera to capture all information sources generates a static, boring view that is hard to follow [3]. Researchers have investigated approaches in making single camera views more lively and relevant by using automatic switching [4] and zooming approaches into regions of interest, but these approaches are not customizable to individual students and cannot handle more than one student’s preference at a time [5]. Shifting the responsibility of visual lecture access from the institution to a shared one between the teacher and students enables deaf and hearing students to collaboratively improve the accessibility of their own experiences; which is a more pragmatic solution to access problems.

SIGCAS Computers and Society, Volume 41, No. 2, December 2011

52

Figure 5: An example of a visually inaccessible classroom that illustrates the inaccessibility of one or more of lecture information sources for deaf students.

Visual Accessibility: Cooperative Sharing Classroom Visual Accessibility Most classrooms are aurally accessible, but are not typically visually accessible from all directions as shown in Figure 2. The reason for this design bias is that sound is omnidirectional, while vision is directional. Therefore, classrooms are typically optimized for one way visual communication from the teacher to the students. Other communication is usually assumed to be auditory, and therefore no additional visual access is presumed to be needed. Hearing students are able to process both the lecture audio and video streams simultaneously, because their visual and auditory memory buffers are separate, but deaf students cannot process both the video stream and video translation of audio stream simultaneously, because both streams go through the same visual memory buffer [6]. The need for minimizing visual switching between information sources therefore places even more importance on obtaining full visual access. In most classrooms, visually oriented students have limited and unsatisfactory options on where to sit in the classroom. For effective classroom discussion, deaf students tend to sit in the back of the classroom so as to see everyone. However, the accommodation personnel such as sign language interpreters and captioners normally prefer to sit in the front so as to hear the instructor and see the slides optimally. If deaf students sit in the rear, they will have difficulty reading or comprehending sign language interpreters and captions, which need to be in high resolution. Therefore the deaf student’s best option is to normally sit in the front, so as to see the information sources clearly, but this reduces their choices and autonomy in classroom learning and seating.

SIGCAS Computers and Society, Volume 41, No. 2, December 2011

53

Figure 6: An example of a more visually accessible classroom typically found in small meeting rooms or in classrooms designed for deaf students.

Another challenge for deaf students is to view the information in a way that minimizes their view switching and to have a structured view switching [7]. Previous work has shown that participants using personal laptops preferred multiple targeted smaller views of a presentation rather than to a large single high definition view [5]. Personalized multiple capture also empowers students to pick which configurations will work best for them and gives the control that no automated technique can predict [7].

Ubiquitous Smart Phones Although modern classroom technology has contributed to the visual dispersion of the multiple information sources, smart phone technology advances can reduce these barriers. In the United States, smart phones that typically include video recording and viewing capability have already captured 25% of the mobile phone market, and based on current trends, to capture 50% of the market by end of 2011 [8]. Although camera-equipped smart phones have less resolution and processing power than traditional video cameras, innovation in the consumer smart phone device field has enabled rapid gain processing and bandwidth capabilities, to the point that it is now possible to process and distribute automated standard definition multiple video feeds cheaply. These smartphone advances video capture and streaming capabilities open up the exciting possibility that hearing and deaf students can cooperatively share multiple video views without needing any additional equipment or work. Students have incentives to participate, as they will get improved visual access by capturing and sharing the multiple information sources close up and at optimal perpendicular angles as shown in Figure 7. Deaf and hearing students then can sit anywhere in the classroom and still watch any of the presentation information sources from the optimal view of the mobile devices.

SIGCAS Computers and Society, Volume 41, No. 2, December 2011

54

Figure 7: An example of a virtually visually accessible classroom with the use of five smartphones aimed at the presenter, slides whiteboard and participating students.

This virtual viewing approach shows how a visually inaccessible classroom can be virtually retrofitted by using cooperative sharing of multiple video cameras that each obtain an unobstructed, close up and perpendicular view of the lecture multiple information sources. This approach can eliminate perspective distortions which can be a serious impediment to visual intelligibility depending upon the information source that is being viewed. For example, slides can be very hard to read when viewed at an extreme angle; likewise, interpreters can be hard to follow. On the other hand, an instructor can be viewed even at right angles without affecting intelligibility significantly under most circumstances. Modern lectures often have various different communication structures during lecture. Sometimes the lecture becomes a one-on-one conversation between the teacher and student, or between two students. At other times the communication occurs in a small group between student groups that form for problem solving or brainstorming. Finally, the communication structure could follow the traditional lecture structure in which the teacher presents information to the entire classroom. In all of these cases, the collaborative personal recording approach can be reconfigured in real-time to handle each classroom communication structure.

Seating Flexibility Evaluation To evaluate deaf students’ perceptions of seating flexibility obtained through multiple capture devices, we recorded a typical multimedia lecture that had multiple information sources. We recorded the instructor slides, captions and sign language interpreter so as to capture close up views on their personal laptops as shown in Figure 8.

SIGCAS Computers and Society, Volume 41, No. 2, December 2011

55

Figure 8: An example of a deaf student viewing the video streams from multiple smart phones that capture the multiple lecture information sources.

We recruited 18 deaf participants ages 20-45 (11 female), who typically requested accommodations in the classroom (interpreters or captions) for the study. Each participant was presented with two balanced conditions: a single high definition recording view or a multiple view that consisted of four views of the presenter, power point, captions and sign language interpreter. After viewing both conditions that were randomly assigned, each participant was asked to respond to three preference questions using a Likert scale from 1 to 5. They were asked to rate the views they watched on the basis of the following questions: 1) “What is your rating for the single view versus the multiple views?”, 2) “Is the video angle good?”, and 3) “Do you feel confident you can sit at the back of the classroom?”. The respondents gave similar ratings for all three questions; For the question “What is your rating for the single view versus the multiple views?” there was a significant preference improvement from viewing the single high definition video obtained from a single high definition camera towards viewing the multiple videos obtained from personal smart phones: χ2 = 16.81, p < 0.001. Similarly, for the question on whether the video angle was good, there was a significant preference improvement of χ2 = 12.81, p < 0.002 towards viewing multiple views over single view. Finally, for the question on whether students felt confident of sitting in the back of the classroom, again there was a significant preference improvement of χ2 = 10.37, p < 0.005 in viewing multiple views over single views.

Conclusion We propose a cooperative sharing approach by students using personal camera equipped mobile devices can improve visual accessibility and seating options for all students, especially those students who are deaf and hard of hearing. We show that the ability to capture and stream online presentations from preferred locations to students in the classroom enables everyone to share visually accessible of the classroom. This benefits deaf students who otherwise would not get more inclusive access in typical classrooms. This approach also has the advantage in that it utilizes personal smart phones that result in flexible, shareable and easily deployable views compared with traditional video SIGCAS Computers and Society, Volume 41, No. 2, December 2011

56

streaming. We also show that deaf students prefer this approach over traditional video capture. We note that personal smart phones are now ubiquitous and cheap, and can be effectively used to record and stream, when combined with context-aware and scalable approaches that can handle the bandwidth and resolution limitations of personal smart phone devices. Finally, we note that capturing multiple views can reduce total information captured and displayed when compared with traditional presentation capture approaches.

Future Work Cooperative sharing of ubiquitous personal smart phones by students in the classroom shows great potential in empowering deaf students to take control of their own learning experience in mainstreamed classrooms. Related research approaches have implemented real-time region-of-interest cropping procedures [9], which in turn can be extended to detect borders using contextual cues and zoom in according to the student’s predefined preferences. Finally, the ability of placing personal camera-equipped smart phones close up and perpendicular to lecture information sources can enhance the accuracy of existing optical character recognition (OCR) programs which may allow them to accurately generate text from the video views of the slides or captions.

References [1]

H.G. Lang, “Higher education for deaf students: research priorities in the new millennium,” Journal of Deaf Studies and Deaf Education, vol. 7, Jan. 2002, pp. 267-80.

[2]

National Center for Higher Education Management Systems, “Progress and Completion: Graduation Rates: Six-Year Graduation Rates of Bachelorʼs Students: 2008.”

[3]

A. Ranjan, J. Birnholtz, and R. Balakrishnan, “Improving meeting capture by applying television production principles with audio and motion detection,” Proceedings of the 26th Annual CHI Conference on Human Factors in Computing Systems - CHI ’08, New York, New York, USA: ACM Press, 2008, pp. 227-236.

[4]

F. Lampi, S. Kopf, M. Benz, and W. Effelsberg, “A Virtual Camera Team for Lecture Recording,” IEEE Multimedia, vol. 15, Jul. 2008, pp. 58-61.

[5]

R.S. Kushalnagar, A.C. Cavender, and J.-F. Pâris, “Multiple View Perspectives: Improving Inclusiveness and Video Compression in Mainstream Classroom Recordings,” Proceedings of the 12th International ACM SIGACCESS Conference on Computers and Accessibility - ASSETS ’10, Orlando, FL: ACM Press, 2010, pp. 123-130.

[6]

M. Marschark, P. Sapere, C. Convertino, and R. Seewagen, “Access to postsecondary education through sign language interpreting.,” Journal of Deaf Studies and Deaf Education, vol. 10, Jan. 2005, pp. 38-50.

[7]

A.C. Cavender, J.P. Bigham, and R.E. Ladner, “ClassInFocus,” Proceedings of the 11th International ACM SIGACCESS Conference on Computers and Accessibility - ASSETS ’09, New York, New York, USA: ACM Press, 2009, pp. 67-74.

SIGCAS Computers and Society, Volume 41, No. 2, December 2011

57

[8]

NielsenWire, “Android Soars, but iPhone Still Most Desired as Smartphones Grab 25% of U.S. Mobile Market,” NielsenWire, 2010.

[9]

J. Foote, D. Kimber, and B.S. Manjunath, “Region of interest extraction and virtual camera control based on panoramic video capturing,” IEEE Transactions on Multimedia, vol. 7, Oct. 2005, pp. 981-990.

SIGCAS Computers and Society, Volume 41, No. 2, December 2011

58