multimedia support in java - Department of Computer Science and ...

29 MULTIMEDIA SUPPORT IN JAVA Joseph Celi1 and Borko Furht2 1 Internet Development Group IBM, Boca Raton, Florida 2 Department of Computer Science and Engineering Florida Atlantic University, Boca Raton, Florida 33431

1. 2.

3.

4.

5. 6.

INTRODUCTION........................................................................................................................ 606 ANIMATION SUPPORT IN JAVA............................................................................................ 607 2.1 FRAME -BASED ANIMATION ..................................................................................................... 607 2.2 SPRITE -BASED ANIMATION ...................................................................................................... 614 AUDIO SUPPORT IN JAVA....................................................................................................... 620 3.1 PLAYING AN AUDIO CLIP FROM A JAVA APPLET ...................................................................... 620 3.2 PLAYING AN AUDIO CLIP USING A MORE SOPHISTICATED APPROACH ....................................... 620 3.3 PLAYING AND STOPPING AN AUDIO CLIP ................................................................................. 622 VIDEO SUPPORT IN JAVA....................................................................................................... 622 4.1 STANDARD VIDEO ................................................................................................................... 622 4.2 STREAMING VIDEO .................................................................................................................. 623 JAVA MEDIA API ........................................................................................................................ 624 CONCLUSION............................................................................................................................ 625 REFERENCES .............................................................................................................................. 626

Abstract. In this chapter, we discuss multimedia support in Java. The object-oriented paradigm, applied in Java, allows software developers to build upon the available class libraries in order to create new custom class libraries, thus extending the language. We present Java support for animation and audio, and we also create working Java applets, which can accomplish various animation and audio tasks. The current Java class libraries do not provide support for MIDI, streaming audio, or any form of video. However, a new Java Media Application Programming Interface (API), which is briefly introduced in the chapter, defines a set of multimedia classes to support interactive media, including audio and video.

606

Chapter 29 605 1. INTRODUCTION

Java is an exciting new technology that has hit the computer industry by storm. This technology, developed by Sun Microsystems, is gaining widespread acceptance at an unprecedented pace. What makes the Java programming language stand out from the others? To answer this and other questions about Java, one must first understand exactly what Java is. Java is an object oriented programming language that allows developers to create platform independent applications [1]. The language borrows its syntax and style from C++ but does not carry the extra baggage of having to be backward compatible with a predecessor language. Java is a clear, concise language that adheres closely to the object-oriented design principles of information hiding, encapsulation, data abstraction, inheritance, and polymorphism. These features promote code reuse, which is a highly desirable feature for most software development shops. Java provides support for exceptions, which aid programs to become more reliable since they are better equipped to handle error conditions. Java is an interpreted language requiring a Java interpreter to be present on the machine running the Java program. The Java interpreter is often called the Java Virtual Machine or JVM for short. The Java compiler compiles a Java program into byte codes that reside in a .CLASS file. The .CLASS file is fed to the JVM for execution. Figure 1 depicts the areas described in this section.

Figure 1. Java environment.

Multimedia Support in Java

607

The Java program is machine independent. The machine dependencies are contained within the JVM [2]. Currently there are JVMs written for all of the popular operating systems. Java applications can be run under any operating system that supplies a JVM. Java applets are similar to Java applications in that they define an executable entity. Java applets are typically located on a Web server and run remotely on a client machine. For security reasons, Java applets have limited access to the API at run time. The built-in security features of Java make it a very attractive solution for distributed programming. Most Web browsers available today support Java applets. The portable nature of the Java language is the key element driving the industry toward this new technology [3]. The “write once, run many” philosophy that Java presents has attracted many programming organizations to this new and exciting language. 2. ANIMATION SUPPORT IN JAVA Displaying a sequence of pre-drawn images one after another creates an animation. Each image or frame in the animation is carefully drawn with respect to a previous frame to simulate movement. If the system rendering the animation is capable of displaying the images at a rate of approximately 15 frames per second, the transitions from frame to frame will seem fluid, hence simulating real movement. The Java application programming interface (API) provides support for animation by allowing users to load and manipulate images using classes and methods are included in the Java Development Kit (JDK). Java distributes the task of retrieving, decoding, and rendering images over several classes and packages. The Applet, Image, Graphics, and Toolkit classes all play a part in producing the finished product. Other classes provide additional support. Network support is provided by the URL and URL Connection classes. The MediaTracker class and the ImageObserver interface provide a means to monitor the image loading progress. These capabilities are integrated into the Java runtime system and perform their respective roles in the background. The following sections concentrate on the direct support classes and the methods that can be used to build image animations.

2.1 FRAME-BASED ANIMATION Frame-based animation, as described above, is an animation sequence, which is produced by displaying a set of frames in some predefined sequence. In the simplest case, the images are all of the same size so that each frame simply overlays the previous one. The key issues to be tackled here are performance and picture clarity. The following example demonstrates the techniques used to ensure the best possible frame-based animations. The FrameAnimation example loads three images and displays them in sequence to produce a simple animation. The animation frames are displayed in Figure 2. The frames are displayed in the following order {1, 2, 3, 2, 1}. This frame sequence, when played over in a continuous manner gives the illusion of the famous “let your fingers do the walking” animation that is often used in commercials used to advertise the yellow pages telephone book.

608

Chapter 29

frame 1

frame 2

frame 3

Figure 2. Frames uses for animation sequence. The FrameAnimation example is in the form of a Java applet, which executes within a Java enabled browser or within the applet viewer utility. The two most popular Internet browsers in use today, from Netscape and Microsoft, both support Java applets. The applet viewer utility comes as part of the JDK from Sun. The code for this applet is shown in Figure 3. The FrameAnimation applet loads the three images from its init method. The browser or applet viewer informs the applet that it has been loaded into the system and calls the init method of an applet. It is always called before the first time the start method is called. The getImage method returns an Image object that can then be painted on the screen. This method requires the caller to specify a URL pointing to the location of the image. The image name is also required to be provided as a parameter. This method will always return immediately even if the image cannot be located. For this reason one should always check for a valid return from the getImage method. The Java libraries will load the actual image data, asynchronously, under the execution of another thread. If an attempt is made to draw an image before all of the data has been transmitted, only the portion of the image that has been loaded at that point in time will be displayed. In order to reduce the initial image flickering caused by partially loaded images, the FrameAnimation applet prepares each image that it uses for the animation. The prepareImage method, contained in the Component class, will return false if the image is not fully prepared for rendering. Prepared, in this context, means that all of the bits of the image have been transmitted across the Internet and the image has been scaled to the height and width specified. In this example, no dimension parameters were provided to the prepareImage method so the image will not be scaled. Once all of the images in the animation sequence have been prepared, the program is ready to start displaying the animation. To create animation, a series of images or frames are displayed in some logical order. A pause or sleep between the rendering of consecutive images is necessary to provide the illusion of movement. The amount of time to sleep between frames must be carefully chosen. If the time interval is too short, the motion will be too fast.

Multimedia Support in Java

609

import java.awt.*; import java.applet.*; public class FrameAnimation extends Applet implements Runnable { Thread runner; Image img1, img2, img3, curImg; boolean fOkToPaint = false; public void init() { System.out.println("In init method"); img1 = getImage(getDocumentBase(), "fingers1.gif"); if(img1 == null) System.out.println("getImage failed to obtain fingers1.gif"); img2 = getImage(getDocumentBase(), "fingers2.gif"); if(img2 == null) System.out.println("getImage failed to obtain fingers2.gif"); img3 = getImage(getDocumentBase(), "fingers3.gif"); if(img3 == null) System.out.println("getImage failed to obtain fingers3.gif"); while(!prepareImage(img1, this)) { // System.out.println("Sleeping while preparing img1"); mySleep(10); } // System.out.println("Finished preparing img1"); // System.out.println("img1 = " + img1); curImg = img1; while(!prepareImage(img2, this)) { // System.out.println("Sleeping while preparing img2"); mySleep(10); } // System.out.println("Finished preparing img2"); // System.out.println("img2 = " + img2); while(!prepareImage(img3, this)) { // System.out.println("Sleeping while preparing img3"); mySleep(10); } // System.out.println("Finished preparing img3"); // System.out.println("img3 = " + img3); fOkToPaint = true; } public void start() { System.out.println("In start method"); if(runner == null)

610

Chapter 29 { runner = new Thread(this); runner.start(); }

}

Figure 3. Java applet for frame animation.

public void stop() { System.out.println("In stop method"); if(runner != null) { runner.stop(); runner = null; } } public void run() { System.out.println("In run method"); while(true) { if(curImg == img1) curImg = img2; else if(curImg == img2) curImg = img3; else curImg = img1; repaint(); mySleep(100); } } // // We will override the update method in order to // get rid of the flicker. The default update // method will always fill in the area with the // background color and then call the paint // method. // public void update(Graphics g) { paint(g); } public void paint(Graphics g) { // System.out.println("In paint method"); if(fOkToPaint) g.drawImage(curImg, 0, 0, this); } public void mySleep(long millis) { try { Thread.sleep(millis); }

Multimedia Support in Java

611

catch(InterruptedException e) {} } }

Figure 3 (cont.) Java applet for frame animation. The motion could also appear jerky if there is not sufficient time for the eye to keep up with the frames. If the time interval chosen is too long, the output will look more like a slide show than animation. The sample animation applet uses a 100-millisecond sleep time between frames. 2.1.1 Using Multiple Threads of Execution The FrameAnimation applet utilizes the multi-threading capabilities of Java to display a continuous animation. If Java did not include support for multiple threads then applets like the FrameAnimation example would have to build their own multi-threading logic to allow other tasks to run while the continuous animation was being played. The multiple thread support in Java allows programmers the ability to build sophisticated programs and games that can accept user input while performing some other complex task [4]. Overall, using multiple threads often reduces program execution time. This is depicted in Figure 4. Referring to the figure, three systems are displayed: a single threaded system, a multithreaded system running under a single CPU, and a multi-threaded system utilizing multiple CPUs. On the single threaded system, a task runs to completion before the next task is executed. If a task performs an expensive I/O operation, the CPU will remain idle until the operation has completed.

Task 1

Task 2 Task 1 Task 2 Task3

CPU

CPU

Single Threaded System

Multi-threaded Single CPU System

Task 1

CPU

Task 2

Task 3

CPU

Multi-threaded Multiple CPU System

Figure 4. Single vs. multiple threading environments.

CPU

612

Chapter 29

Of the three, the single threaded system will often yield the worst performance. The second system shows a multi-threaded system running under a single CPU. In this system the tasks are allowed to run for an interval of time called a time slice. When the time slice expires, the next task is given the CPU and the state of the previous task is saved. This system often yields better overall CPU utilization than the first, since the CPU does not have to sit idle during I/O requests. In this system, since there is only one CPU, each task may take a bit longer to run, due to time-slicing, but the overall time to execute all tasks is almost always shorter then the single threaded counterpart. The third system shows a multi-threaded system running under multiple CPUs. This system will almost always yield the best overall performance for the obvious fact that adding more CPUs will boost system performance. It is important to understand the execution process of a Java applet when writing applets, which require multiple thread support. Figure 5 shows the execution cycle of a Java applet. The arrows show the events that take place in the browser; the circles represent the corresponding functions in the Java applet class called to handle those events.

Leaving page

Init

Start

Page activated

Stop

Re-entering page

Destroy

Page discarded

Figure 5. Life cycle of a Java applet. A Java applets’ init and start method are automatically called by the JVM when the applet is started. In order to keep the Java environment responsive, a Java applet must return from the invocation of these methods in a timely manner. The animation sequence in this example continues to play until the applet is destroyed. The FrameAnimation applet implements the Runnable interface. Any class whose instances are intended to be executed by a thread can implement the Runnable interface. During the start method, the FrameAnimation applet will create a new Thread class and execute its start method. The invocation of a thread start method will cause an objects run method to be invoked. The class, provided to the Threads constructor, determines the owning object of the run method. In this example, the “this” keyword was used when the Thread was constructed. The actual line of code is displayed below: runner = new Thread(this); The “this” keyword is used to represent the current object, which in this case is the applet itself. Therefore, the run method of this applet will gain control. In that method, the images are cycled through and repainted in a loop. The mySleep method is used to sleep between the rendering of consecutive frames.

Multimedia Support in Java

613

The animation thread is halted when the stop method is invoked. The stop method of an applet is invoked when user leaves the page that contains the applet. Although the applet is still running, the stop method provides the applet writer with a means to perform some action when the applet is no longer in view. In this case, the CPU intensive animation sequence is halted. An applet will receive a destroy method before it is actually terminated by the JVM.

2.1.2 Eliminating Flicker The FrameAnimation applet employed a technique to eliminate flicker. Flicker, in this context, is an annoying visual side effect caused by the introduction of unnecessary insertion of blank frames in the animation sequence. Figure 6 graphically displays an animation sequence that produces screen flicker.

Figure 6. Animation frame sequence when default update method is being used. A good understanding of the Java painting protocol allows developers the ability to produce flicker-free animation sequences [5]. Figure 7 graphically displays the final frame sequence of a flicker free animation. In order to produce this sequence, the FrameAnimation applet subclasses the update method.

Figure 7. Animation frame sequence when default update method is subclassed. In this example, the repaint method is called from within the animation thread to signal the painting of the next frame in the animation sequence. When the repaint method is invoked, the JVM will call the corresponding update method for that object. The default update method, as contained in the Java 1.0 release is displayed below: public void update(Graphics g) { g.setColor(getBackground()); g.fillRect(0, 0, width, height); g.setColor(getForeground()); paint(g); }

614

Chapter 29

The update method clears the graphics area with the background color before calling the paint method. In this example the update method is subclassed and simply calls the paint method. The code for this method is displayed below: public void update(Graphics g) { paint(g); }

Referring to Figure 5, the default update method is causing a black rectangle to be displayed between each of the consecutive animation frames. Although this black frame is only displayed for a few milliseconds, it will cause the animation to flicker. By subclassing the update method we eliminate the flicker since the black frames will no longer be displayed between each of the consecutive animation frames. This technique works fine for this application since it is drawing the images at the same location over and over. If the images were moving or other graphics were being rendered onto the screen, then a more sophisticated version of the update method would need to be employed. 2.2 SPRITE-BASED ANIMATION In the first animation example the animation frames were simply overlayed on top of the previous one in the exact same location. The applet was able to optimize things based on the characteristics of that type of animation. Although simple frame-based animations are frequently used, a more powerful type of animation is required for applications such as computer games. A sprite is an image that has a transparent background. In computer games the sprites are moved about the game while the background scenery is preserved. How is this accomplished in Java? There are other issues involved in sprite-based animation. For example, how is the background restored when the sprite is moved across the screen? The next example will demonstrate techniques that allow programmers to build programs using sprite-based animations. The SpriteAnimation example, presented in Figure 8, draws a red bouncing ball on a blue background. The ball bounces off the edges of the applet as a real ball bounces off a wall. The source code for the SpriteAnimation applet is given in Figure 9.

Figure 8. Output of the SpriteAnimation Java applet.

Multimedia Support in Java

615

2.2.1 Transparency Java supports the GIF89a image file format. In this format an RGB color value is assigned within the GIF file to represent the transparent color [6]. When the GIF is rendered to the screen, all pixels in the GIF matching this assigned RGB color value are not drawn, hence yielding the transparent effect. Java programmers can create images with the proper transparency settings using any image editor, which supports the GIF89a image file format. The Java programmer simply uses these special images to create transparent effects. This is an improvement upon previous windowing systems, such as Windows, where the programmer must perform special instructions to draw transparent images onto the screen. 2.2.2 Double Buffering The SpriteAnimation applet uses a technique called “double buffering” to render the sprite across the screen in a fast manner. Double buffering tackles two problems inherit in spritebased animation. These problems are avoiding flicker and preserving the background. In double buffering, an application draws to memory before updating the screen. By first drawing into memory, all flicker can be eliminated since only the finished frame is rendered to the screen. This is demonstrated in Figure 10.

import java.awt.*; import java.applet.*; public class SpriteAnimation extends Applet implements Runnable { Thread runner; Image ball, offImg; int x = 0; int xInc = 1; int yInc = 1; int y = 0; int imgX, imgY; Graphics offG; public void init() { System.out.println("In init method"); ball = getImage(getDocumentBase(), "Ball.gif"); if(ball == null) System.out.println("getImage failed to obtain Ball.gif"); while(!prepareImage(ball, this)) { // System.out.println("Sleeping while preparing ball"); mySleep(10); } imgX = ball.getWidth(this); System.out.println("imgX = " + imgX); imgY = ball.getHeight(this); System.out.println("imgY = " + imgY); }

616

Chapter 29

public void start() { System.out.println("In start method"); if(runner == null) { runner = new Thread(this); runner.start(); } } public void stop() { System.out.println("In stop method"); if(runner != null) { runner.stop(); runner = null; } } public void run() { System.out.println("In run method"); while(true) { repaint(); mySleep(32); } {

Figure 9. The Java applet for sprite animation.

public void update(Graphics g) { Dimension d = size(); if(offG == null) { offImg = createImage(d.width, d.height); offG = offImg.getGraphics(); } offG.setColor(Color.blue); offG.fillRect(0, 0, d.width, d.height); offG.drawImage(ball, x, y, this); // // Now test to see if the ball hit the right or left wall. // if((x + imgX) >= d.width) { xInc = -1; } else if(x = d.height) { yInc = -1; } else if(y = d.width) { xInc = -1; } else if(x = d.height) { yInc = -1; } else if(y = d.width) { xInc = -1; } else if(x = d.height) { yInc = -1; } else if(y