ISWC 2006 Student Colloquium, (2006)
Situated Music: An Application to Interactive Jogging Nobuchika Sakata12, Takeshi Kurata12, Masakatsu Kourogi2, Hideaki Kuzuoka1 1 University of Tsukuba, Japan, 2AIST, Japan {sakata, kuzuoka}@esys.tsukuba.ac.jp,
[email protected],
[email protected] Abstract We define Situated Music as a framework that selects and plays a tune according to a situation. The selected and played tune itself is also referred to as Situated Music. In this paper, we describe Interactive Jogging, that is an application of Situated Music for jogging. Measuring pitch acceleration with an accelerometer attached on a headphone, we estimate a mileage of the jogging. And then we provide certain music, which has certain tempo based on the measured pitch and mileage, to the jogger. Due to this, Interactive Jogging may keep the jogger a runner motivated and augment an amusement aspect of jogging.
1. Introduction Portable music players storing a lot of music files have became very popular. But input and display devices in typical portable players are so small that the operations such as selecting a tune from a lot of music files by pressing a button impose burdens to users. If tunes are played, paused or changed by the music player itself according to the user's situation and surroundings, we can get rid of the burdens. For example, with two kinds of situations, which are walking and driving a car, recognizable by some sensors, it is possible first to stop playing a tune on the portable players, then to change the audio output from earphones to car speakers, and to select another tune suitable for driving. We define “Situated Music” as a framework to select/play a tune that meets what to do and where to go, and also as the selected/played tune itself by such framework. In this paper, we propose applying the Situated Music for "Interactive Jogging" described below. To get some objectives accomplished through an interactive loop consisting of stimuli and responses (Figure 1). Some stimuli in visual/auditory/tactile senses are provided for the jogger based on a jogger's
Figure 1. Interactive jogging. state and objective. If his/her state changes by reacting to the stimuli, the next provided stimuli also changes. We employ Situated Music as stimuli to the auditory sense for Interactive Jogging, and implement it on a portable system. This paper first describes the Interactive Jogging system providing Situated Music based on jogging pitch measured by an accelerometer, and then presents two types of applications Interactive Jogging. Finally, we outline the direction for future work.
2. Interactive Jogging system The interactive Jogging system proposed in this paper requires a method of estimating physical states of a jogger so as to select a tune and to apply some audio effect to tunes based on the physical state. It is also necessary that the system has a method of controlling the jogging state without strain by encouraging the user consciously or unconsciously to make objectives accomplished naturally. To obtain the user's state, our system rules on jogging pitch measured by an accelerometer. With the detected pitch, we can estimate whether the user is jogging or walking, and even how much speed the user is jogging. The detected pitch corresponds to the number of steps per unit of time. Therefore we can roughly calculate mileage of jogging with the number of steps and the mean step length measured in advance. It is known that periodic sounds induce a human to do rhythmics like walking in the same period [3]. By means of this theory, our system induces the user and
Figure 2. Sensor headphone. controls the jogging state while selecting/playing tunes that have a certain tempo suitable for each objective.
Figure 3. Acceleration in the vertical direction when a user walks and jogs.
2.1. Sensor headphone We attached an accelerometer on a headphone speaker for detecting jogging pitch (Figure 2) as in [6]. We call the headphone attached an accelerometer “sensor headphone”. The point and posture of a headphone is more fixed compared to a portable player itself, we can easily and stably measure acceleration in the vertical direction with this sensor headphone. In addition, even nowadays we can embed the accelerometer in a headphone owing to MEMS technology (Figure 2 left).
2.2. Estimating jogging states with pitch detection
Figure 4. Power spectrum of acceleration in the vertical direction when a user walks and jogs.
Our simple but stable procedure for detecting jogging pitch is as below. 1.
2. 3.
Measure a series of acceleration in the vertical direction (Strictly, the direction is not always completely vertical but in this method, that is enough). Apply FFT to the measured accelerations. Regard a frequency with the highest peak in power spectrum as jogging or walking pitch except for around 0Hz. (Acceleration of gravity appears as a kind of bias at 0 Hz zone.)
We made a prototype sensor headphone shown in Figure-2-right to verify that the procedure works adequately. In this prototype, we only use a 1-axis accelerometer at 30Hz in a Microstrain 3DM-GX attached to the right side of the headphone. Figure 3 shows measured acceleration in the vertical direction when a subject walked for about 8 seconds, jogged for about 8 seconds, and again walked for about 10 seconds with the prototype sensor
Figure 5. Power spectrum of acceleration at three different levels of jogging pitch. headphone. We can see from this graph that walking and jogging have different amplitude and frequency. Figures 3 and 4 show power spectrum of a series of acceleration obtained by applying FFT with 128sample window corresponding to about 4.2 seconds. When the subject walked, the highest peak in power
spectrum was at 1.6 Hz zone. Meanwhile, the highest peak was at 2.4 Hz zone during jogging. The same subject with the prototype sensor headphone conducted “fastest pitch”, “normal pitch” and “slowest pitch” joggings based on the subjectivity. Figure 5 shows of acceleration at the three different levels of jogging pitch. When the subject did “fastest pitch” jogging, the highest peak appeared around 2.8 Hz. During “normal pitch” jogging, it appeared around 2.5 Hz. In the case of “slowest pitch”, it appeared around 2.3 Hz. We can see from the results of preliminary experiments that the procedure was able to distinguish between jogging and walking, and can detect different types of jogging pitch. We also had experiences, even in the case of the subject inclining his head or looking around, the sensor headphone can stably detect jogging/walking pitch.
3. Application of Interactive Jogging This section describes two types of applications of Interactive Jogging. The first one is to let a user accomplish an objective by naturally inducing a user to jog at a certain pitch as shown in Figure-6-top. The second one is to augment some entertainment aspect of Interactive Jogging by letting a user listen to some Situated Music based on detected pitch as shown in Figure-6-bottom. In the first application, for example, a jogger sets a target mileage of jogging and the target time before starting jogging. If the system estimate that the user cannot achieve the objective with currently detected pitch, it induces the user to jog at faster pitch by choosing playlists comprised of fast tempo tunes or by applying an audio effect which makes a tune played faster. Each tune's tempo can be obtained from either analyzing tunes itself or the database on the Internet such as MoodLogic [4]. On the other hand, if the system estimate the user will reach the target mileage too early, it chooses a playlist comprised of slow-tempo tunes or applying an audio effect which makes a tune played slower. Also if the system estimate the user jogs at considerably fast pitch compared with the target pitch, the system chooses a playlist for walking pitch. By controlling jogging states like this, the system can prevent the user from the overload and injury before happens. A typical example of the second application to enhance entertainment aspects of Interactive Jogging is to provide the user with playlists so as to have the tempo for the playlist closer to the target tempo in a step-by-step manner only when detecting the same
Figure 6. Applications of the Interactive jogging. jogging pitch continuously. It would be able to realize another example as follow: During the beginning of jogging, the system plays only a basso of tune with a low pass filter at small volume. If the user keeps jogging at the same pitch continuously, the diapason is played at larger volume. If the user keeps jogging more continuously, additional sounds are overlapped on tunes at full volume such as "Dance Dance Revolution" [5].
4. Discussion We detected three levels of jogging pitch with the prototype system presented in this paper. If it is necessary for some application to detect more pitch levels, we can get such high resolution date only by turning up the sampling rate of accelerometer to raise Nyquist frequency. Our system calculates a mileage of jogging from detected pitch. By means of Personal Positioning System based on inertial [1] and GPS sensor, the accuracy of jogging-mileage estimation should get improved compared with the current method. The current implementation cannot estimate physical/mental work load of users by a detected pitch because of individual differences. To obtain the physical load with the current system, we need to know the correspondence of jogging pitch with heart rate and the amount of oxygen in the blood before jogging starts. However, sensors for vital observation are getting smaller enough to be embedded in the same headphone. By using such sensor headphone, the system can finely control jogging states according to the user's daily condition.
Our prototype system doesn't have any criteria on how to use explicit user input for providing Situated Music. Rekimoto proposed UniversalPlaylist [2] that has a function to make favorite playlists based on interactive machine learning just with yes/no input from the user for each incoming tune. In such interactive manners, our future system would be able to be improved so that it can select more favorite playlists compared with the current system.
5. Conclusion In this paper, we proposed and implemented the Interactive Jogging as an application of the Situated Music. We confirmed that an accelerometer attached to headphone could detect jogging pitch and walking or jogging state in the experiment. Also we proposed a method of providing the Situated Music. In the future, we will confirm how much naturally the users accomplish each objective and how much they feel entertainment aspects in the Interactive Jogging.
References [1] M. Kourogi and T. Kurata, "A Wearable Augmented Reality System with Personal Positioning based on Walking Locomotion Analysis", In Proc. ISMAR03, Tokyo, 2003, pp.260-261 [2] J. Rekimoto, “UniversalPlaylist: A Dynamic Adaptation Mechanism for Media Playback”, In Proc. Interaction 2005, written in Japanese. [3] Muto T., Herzberger B., Kobayashi Y., Muto Y., Globocnik T., Poeppel E., Hermsdoerfer J. & Miyake Y., "Interactive cuing with the Walk-Mate to improve stability and variety of gait dynamics", In Proc. HCI International 2005, pp.1-10 [4] MoodLogic. http://www.moodlogic.com [5] DanceDanceRevolution, Konami, http://wwww.konami.co.jp/ [6] S.Matsushita, “A Wearable Communication Modulator”, ISWC2002, pp.166-167
