Pixels, Numbers and Programs: A Freshman Introduction to Image ...

Pixels, Numbers and Programs: A Freshman Introduction to Image Processing Steven L. Tanimoto University of Washington [email protected] Abstract Image processing and computer vision form an increasingly relevant subject area not only for students of computer science, but for all citizens in the 21st century information society. This paper describes a freshman seminar that introduces image processing as an interdisciplinary subject connecting mathematics, computing, art, and experimental psychology. For some students, it's also a quick introduction to programming. In addition to bringing information technology and computer vision concepts to some students earlier than they might otherwise get them, it can serve as an informal curricular laboratory and a means to help a few capable students get into research well before the stage of senior projects and capstone courses.

1. Introduction 1.1 Relevance of Image Processing to Freshmen Both information technology (IT) and image processing are subject areas that are increasingly relevant to being a successful participant in 21st century society. Information technology fluency has been identified by a National Research Council study as important for all citizens [2]. Image processing is an important subarea of IT. Every college student should know that digital images can be represented as arrays of numbers and that they can be processed using mathematical algorithms in ways that can alter them to look better, take up less storage space, or make them seem to represent nonexistent objects and events. Image processing, traditionally taught as an advanced undergraduate course for students in electrical engineering or computer science, should now be introduced to a broader range of students and at an earlier stage in the curriculum. 1.2 Fostering a Research-Oriented Mindset Getting undergraduate students involved in research has been identified as a national goal, and it has been encouraged by such programs as the National Science Foundation’s “Research Experiences for Undergraduates.”

Getting undergraduates into research can mean changing their attitudes in several ways. Many students come to college with an impression that their learning will simply be a process of absorbing knowledge that is codified in books and lectures; getting students to make empirical inquiry a part of their model of education is one particular challenge. Another challenge is to foster students’ self confidence so that they themselves will initiate questioning, hypothesizing, and designing experiments. Large lecture courses tend not to foster such habits of mind, in part because their economics makes it difficult to give students the attention they would need to make such changes. Small and less formal seminars, on the other hand, can help to decrease the psychological distance between student and faculty member and help to foster more research-oriented attitudes to learning. 1.3 The Freshmen Seminar Program The University of Washington, a large researchoriented institution, began its freshman seminar program in approximately 1994 partly in order to create opportunities for freshman to have closer contact with faculty members. All too often, the freshman year academic experience is characterized by impersonal, large lecture classes that convey a mass-production notion of what learning is about. Freshman seminars also offer students an opportunity to sample a topic that might be far removed from the student’s prior experience. The seminars offer a chance to be part of a group organized around an intellectual pursuit, an occasion to get to know a faculty member on a first-name basis, and sometimes an opportunity to use interesting computational tools. For the instructor, a freshman seminar is an opportunity to get to know a cross-section of undergraduates at point in their education when they may be making major decisions about what areas to study. It’s a chance to teach in a more relaxed and informal setting than the large lecture, and the seminar provides a sort of curriculum laboratory in which new activities and approaches can be pioneered with a low risk of student unhappiness, teacher anguish, or failure to cover all required material.

(Freshman seminars are not a required part of the curriculum at all.) Students sign up for seminars on a voluntary, firstcome/first-served basis. In a typical academic quarter, approximately 12 seminars are offered, enrolling an average of 15 students each. Students take them on a Credit/No-Credit basis for one unit of credit. There is one 50-minute meeting per week for 10 weeks, and students can be asked to do approximately two additional hours per week of reading or other seminar-related activity.

modification of each component. An example display of an image using this program is shown in Fig. 1. Only a small portion of the image is visible here, because the user has zoomed in on it in order to see the numeric values of the color components of its pixels.

1.4 Project on Mathematics Experiences Through Image Processing Our “Pixels” seminar is an outgrowth of the NSF funded project, “Mathematics Experiences Through Image Processing” (METIP) [1]. This project developed and tested image-processing software for middle-school and high-school students that would introduce them to the use of arithmetic and other mathematics topics in modifying digital images. The primary goal was to improve students’ attitudes towards mathematics by empowering them with mathematical tools to create interesting effects with images [5] [6]. The Pixels seminar itself has the objectives of exposing students to the interdisciplinary nature of image processing, giving them a few basic concepts, and allowing them to develop self confidence in using tools, techniques, and terminology of image processing and its allied subjects.

2. Supporting Software Much of the seminar activity consists of hands-on image processing. We use software that is specifically oriented towards learning the use of mathematics in describing image manipulations. Most of the software was developed by the METIP project, described in Section 1.4. Here is a description of the main pieces of software. Pixel Calculator: This provides a gentle introduction to image manipulation using mathematical operations rather than virtual artist's tools. Its functionality is to display images and perform arithmetic on pixels. It is limited to monochrome images in the Microsoft BMP format. Warper: This is a simple program that allows visually complicated warping to be performed on BMP images, following the Pacific Data Images algorithm. It illustrates a side of image processing that is complementary to what the Pixel Calculator can do. Color Pixel Calculator: This program functions as an extended version of the Pixel Calculator. It shows the RGB values of each pixel, and it allows selective

Figure 1. The display of pixels using the Color Pixel Calculator program. Color XFORM: This package includes the Color Pixel Calculator functionality, but contains a much more general transformation capability. It uses the same image representation method as in the Color Pixel Calculator. There is a Lisp programming interface in this package, too. In addition to the above software developed by the METIP project, the seminar also uses a program called “PixelMath,” which is a Java-based implementation of a Color XFORM like functionality, including Lisp scripting. Fig. 2 shows the PixelMath calculator interface.

Figure 2. The PixelMath calculator interface. Most important is the formula area. Other controls include RGB color component selection toggles, source and destination window choices, polar/cartesian switch, and keys for mathematical functions and values.

3. Topics Covered 3.1 Topic Selection Criteria The topics in the seminar are selected with four points in mind, and they vary somewhat from one offering to the next. The points are: (1) some important basic concepts should be introduced, (2) enough stimulating ideas should be included so that the students’ attention is captured, (3) the interdisciplinary aspects of image processing should be highlighted, and (4) the importance of mathematics for image processing should be put in evidence. 3.2 Image Representation As an important basic concept, image representation plays an important role in the seminar. Students work with images as arrays of numbers from the first meeting on. Although this is a simple idea, it has many nontrivial aspects, including sampling and quantization and issues of format and format conversion, connections to perception, etc. Another aspect of image representation is representation of color. Color mixing activities are a good way to get students involved in manipulating digital representations building on some of their intuitions (“red and blue make purple”) while challenging others (“yellow is a primary color and cannot be produced as a mixture of other colors”). The use of formulas to compute pixel values is a notion that is introduced in the second meeting and used for the rest of the academic quarter. 3.3 Transformations The concept of an image transformation is tightly connected to the mathematical concept of function [6]. However, aspects of computational transformations that go beyond the traditional mathematical notions of geometric mappings soon come up. Important are the pixel order and source-destination orientation for computing the transformation. Therefore, we discuss the distinction between Push and Pull methods of computing transformations soon after formula-based transformations have been introduced. 3.4 Coordinate Systems Coordinate systems come into play in image processing and computer vision in numerous ways. The seminar addresses such issues as locating the origin, reconciling mathematical cartesian systems with computer screen systems: alternative directions of increasing y values; continuous versus discrete spaces, polar versus cartesian

coordinates, and representation of angles in radians versus degrees. 3.5 Anamorphism Visual effects such as anamorphism illustrate how simple physical setups such as cylindrical mirrors on flat images can raise sticky geometric problems, and yet heuristics can be used to good effect. 3.6 Computer Vision Topics Edge detection is a simple idea. Yet it provides the basis for an introduction to some of the key concerns of computer vision... computing structures from pixel arrays, how to represent form, adequacy of information, context, etc. [7]. Students who become intrigued with these ideas can pursue them further with online resources or a standard text such as Shapiro and Stockman [4]. 3.7 Computer Programming In much the same way that turtle graphics has been used to introduce Logo programming, image processing can be used to introduce programming via the scripting facilities within Color XFORM and PixelMath. The LOGO approach integrates a Lisp-like general programming language with a set of graphics primitives for creating interesting 2-D patterns. In the Pixels seminar, we use the Lisp language itself to control sequences of image processing operations. While the short time available in the seminar does not permit an indepth introduction to programming, there is enough time for students to be introduced to three important aspects. These are: (a) the fact that programming can be used to automate processing sequences that otherwise require a significant amount of human work; (b) the use of loops to make a computer do a lot of work for a relatively small programming investment; and (c) the use of programmerdefined functions to provide a means to build up a powerful collection of capabilities [8].

4. Experience to Date 4.1 Challenges Teaching a seminar such as this presents both challenges and rewards. Here are some of the challenges. At the University of Washington, freshman seminars are typically treated as overloads in teaching schedules, and not seen as departmental contributions. Also, one needs to have a suitable computer classroom or laboratory. We have tried two different laboratories and found that the physical arrangement of the room is important in

catalyzing both student-to-student and seminar-wide interactions. The best arrangement seems to be to have a large room that has a seminar table and chairs in one area and four to 5 clusters of four computers each on separate tables. That permits the group to begin at the conference table and have face-to-face discussions for 10 minutes and then to move to the computers to perform the class’ lab activities. In a room without the seminar table, we tried making a circle of chairs in order to hold a discussion; but students seemed to feel more self conscious in this arrangement and the discussions were not as comfortable. 4.2 Student Background As of September 2001, we’ve run the 10-week seminar three times. There is a good mix of students who sign up. Several want to get into computing and sign up because they see this is as a computing-oriented seminar. Some are interested in graphic art and multimedia. Others just want to be in some kind of seminar and the Pixels one sounds interesting. Most of the students have used programs such as Adobe PhotoShop prior to enrolling. A few have had prior programming experience. 4.3 Group Activities In one assignment designed to encourage collaboration, the class was divided into four groups of four students each, and four different image processing problems were distributed, two per group. In this way, each group had two problems to work on, and each problem was covered by two groups. The four problems were: (1) transform a Monet daytime beach scene into a sunset scene using only formula applications; (2) turn a picture of the Nisqually

(a)

Glacier on Mt. Rainier into one that seems to have a reflecting lake in front of it, and, if you can, put ripples in the lake, using only formulas (a sample solution is shown in Fig. 3); (3) take a picture of purple asters and make a “wallpaper” image out of it, replicating the image horizontally and vertically using only formulas; and (4) take the Mona Lisa image and, using only formulas, put her in an elliptical frame. After solving their problems, the teams were asked to describe their solution method within a textual communication system called INFACT FORUM. This assignment seemed to be one of the most successful ones. Students used their imaginations in coming up with solutions to the problems, and the requirement that they articulate their strategies online may have helped them to realize both that they could be creative in their approaches and that they were responsible for completing each challenge without much instructor help.

5. Discussion 5.2 Pedagogical Issues Finding the right balance between individual and group learning activities is a continual pedagogical challenge. In groups made up from a small, diverse population, the abilities and experience of students tend to differ significantly. Group assignments create a setting in which students can easily help each other. Yet some students will accept too much help. The question of depth versus breadth is another ongoing one. Depth is needed if students are to come away with something that sticks in their minds and has significant value. Breadth and variety, however, help to

(b)

Figure 3. An illustration of a solution to the problem of simulating a reflection in a lake, using an image of Mt. Rainier with no lake: (a) original view of Mt. Rainier and its Nisqually Glacier, and (b) simulated reflection scene using the formula: if (y > h/2) then Source1(x,y) else Source1(x,(ymax - y) + 10 * sin((10 + y * (1 + (y * y / 5000))) / 10))

capture student attention. With the varying backgrounds of students and the lack of any screening method, and the single ungraded credit students receive for taking the seminar, it would be unrealistic to expect that a rigorous introduction to image processing could be delivered in the seminar. One solution is to cover enough different topics so that most students find items of interest but to keep some open-endedness to assignments. Another is to offer students choices. However, the management of the class becomes more complicated if students are engaged in too many different activities at the same time. An interesting question is the extent to which image processing can serve as a good pathway to computer programming. As mentioned above, using visual images has much in common with using turtle graphics: students like to produce visual effects, and the instant graphical feedback is a great help in debugging and in programming generally. Although the Pixels seminar currently uses Lisp as its programming language, one could try this with Java or C as well. I’ll argue that Lisp is a better choice than either of those languages because of its level of interactivity and simple syntax. On the other hand, the syntax is less important than the pedagogical context in which the programming is done [8]. 5.2 Connections to Research The Pixels seminar is serving to help bring students into research in two ways. For example, two freshmen who showed sufficient interest and ability in the seminar have gone on to do independent study, albeit at a relatively small number of hours per week. Advanced undergraduate and graduate students have occasionally taken roles in running the seminar, for example by making up and trying out an image processing learning activity, and thus they have participated in research on teaching. The seminar also furthers the METIP project’s original goal of using image processing as a means to getting more students into mathematics and computing. While some activities can work at a middle-school level, progress is needed at every level, including the college freshman level. We recently began a new research project in which the Pixels seminar plays a key role. The goal of the project is to develop and test a learning environment that uses computer-based intensive, unobtrusive assessment to improve teaching and learning about selected concepts in information technology. The seminar has become a sort of educational technology and pedagogical laboratory. The curriculum, learning software, student activities, and assessment methods are all under project investigation and control.

6. Acknowledgements Partial support from the National Science Foundation under grants MDR-9155709, CDE-9616532, and EIA0121345 is gratefully acknowledged. The author would like to thank the many programmers of the METIP project software for their work, including but not limited to Dennis Lee, James Ahrens, Alex Rothenberg, Evan McLain, Lauren Bricker, and Jeremy Baer. Thanks to Chenoah Morgan for contributions to PixelMath. Thank you to Lori Postner and Nick Benson, who have made contributions to running the seminar, and to Earl Hunt, Robin Adams, Cindy Atman, and William Winn for contributions to the new research project involving the seminar. Also, the two anonymous referees' comments were very helpful.

7. References [1] Mathematics Experiences Through Image Processing web site. http://www.cs.washington.edu/research/metip/. [2] NRC (National Research Council Committee on Information Technology Literacy): Being Fluent with Information Technology. Washington, DC: National Academy Press, 1999. [3] J.K. Ousterhout, Scripting: “Higher Level Programming for the 21st Century”, IEEE Computer, March 1998. [4] L.G. Shapiro and G. Stockman, Computer Vision, Englewood Cliffs, NJ: Prentice-Hall. 2001. [5] S.L. Tanimoto, “Exploring mathematics with image processing”, Proc. 1995 IFIP World Conference on Computers in Education, Chapman & Hall, London, pp.805-814. [6] S.L. Tanimoto, “Image processing and teaching the concept of function”, Proc.ED-MEDIA 2001, Tampere, Finland. [7] S.L. Tanimoto. “Connecting middle school mathematics to computer vision and pattern recognition”. International Journal of Pattern Recognition and Artificial Intelligence, Vol. 12, No. 8, pp.1053-1070. 1998. [8] S.L. Tanimoto and J. W. Baer, “Programming at the end of the learning curve: Lisp scripting in image processing”, Proc. 2001 Symposium on End User Programming. IEEE Computer Society Press.