Texture Compensation for Haptic Feedback Signal ... - CiteSeerX

Texture Compensation for Haptic Feedback Signal Compression Fernanda Brandi, Rahul Chaudhari and Eckehard Steinbach Institute for Media Technology Technische Universit¨at M¨unchen Munich, Germany {fernanda.brandi, rahul.chaudhari, eckehard.steinbach}@tum.de Abstract—Recently proposed haptic offline compression algorithms remove perceptually irrelevant haptic samples to achieve data reduction. At display-time, the irregularly subsampled haptic signal is resampled at a higher constant sampling rate using interpolation. Such algorithms, however, have an important drawback. Although they are well suited for large-amplitude quasi-static feedback forces, low-amplitude high-frequency texture information is adversely affected. This informative tactile high-frequency component, critical to convey convincing realistic haptic impressions, needs to be treated separately for compression. To this end, we extract important tactile elements crucial to texture perception from the haptic signal in the time domain and propose a method to encode them so that the overall storage requirements are minimized. We then synthesize and superimpose them onto the reconstructed signal at displaytime. Psychophysical tests confirm that the proposed approach significantly improves the texture assessment quality during playback, while reducing storage space requirements by up to 97%.

I. I NTRODUCTION The recording and replay of haptic feedback in real-world or virtual physical interaction scenarios finds applications in posterior performance analysis, training, teaching and learning as well as entertainment and documentation applications. The future trend is to expand the number of degrees-of-freedom of haptic interaction, tending towards full-body interaction, so as to improve haptic immersion. Accordingly, the number of spatial locations sensed and actuated on the body to impart comprehensive haptic feedback will rise alongwith their associated electrical signals. For example, the commercially available virtual reality force-feedback glove CyberGrasp (CyberGlove Systems LLC) has sensors distributed over the entire human hand to estimate its geometry, and actuates each one of the five fingers separately. Combined with high sampling rates typical to haptics applications, such developments will result in voluminous haptic data and significantly raise storage and transmission requirements. Offline haptic coding algorithms [1], [2], [3] process such raw data decreasing demands on available storage resources. Early approaches in [1] present alternative techniques for achieving reduced sampling of haptic data while the latest state-of-the-art approaches introduced in [2], [3] base their subsampling processes on the perceptual deadband (PD) approach inspired by the Weber’s law of just noticeable differences (JNDs) [4], [5], one of the founding laws of psy-

chophysics. The PD approach, originally proposed for online haptic interactions can be used without modification for offline recording and playback and is expounded in detail in [6] for the online case. Amplitude-dependent perceptual thresholds are defined successively over time which represent bounds within which variations in the subsequent signal trajectory are unnoticeable to an average human user. The intermediate samples constituting these variations are thus deemed irrelevant for storage and therefore dropped. The offline coding algorithms in [2], [3] work on this foundational principle and propose variants of the subsampling and reconstruction processes, more details on which follow in Section II. In their experiments, these approaches have worked upon position-based resistive force-feedback generated by a virtual environment, inherently low-frequency in nature. They are found wanting in situations where high-frequency (HF) tactile signals are important for haptic perception - e.g. haptic display of textures or highfrequency contact forces in a real teleoperation system. This is since the Weber’s law of JNDs was originally proposed for static stimuli, and is therefore unsuitable for application to dynamic HF content. To fill this gap, we propose to separate the low-amplitude HF components of the haptic feedback and encode them completely independently of the energetic low-frequency content. In this work, an algorithm for the encoding process is presented and evaluated for perceptual quality through psychophysical experiments. The data reduction during encoding essentially boils down to achieving a condensed representation of the HF tactile component. In deciding the strength of the data reduction, we base our analyses on the properties of human haptic perception to judge the fidelity of the reconstructed signal to the original uncompressed signal. A significant amount of research has been contributed in the past in the area of tactile signal modeling. Geometric modeling of micro-scale surface properties, a simple extension of algorithms for penetration-based gross force-feedback [7], suffers from computational bottlenecks on account of complex collision detection. Overlaying force maps on smooth threedimensional models with the help of look-up tables reduces computational complexity in superimposing tactile HF signals on gross force-feedback [8]. In [9], a data-driven approach to texture modeling is taken by analyzing haptic signals, specifically the acceleration signal, to extract coefficients for

LPF

f(t)

l(t)

Offline Offline perceptual coding

d(t)

e(t)

Q(.) ()

Entropy Entropy coding e(t)

 f(t)

HFcoding 0

Fig. 1. Illustration of the presented offline coding algorithm. f (t) and fˆ(t) are the originally recorded and the reconstructed haptic texture signals respectively. An example histogram of the error signal e(t) can also be seen to have a peaky distribution of sample values centered around 0.

In [3], another subsampling process was proposed to improve the compression efficiency while ensuring high fidelity of the perceived haptic sensation with the original one. A reduced number of support points are determined in a manner which ensures that the reconstructed haptic force-feedback signal lies entirely within the deadband thresholds. The reconstruction is performed by connecting the support points by line segments at display time. III. T EXTURE COMPRESSION AND COMPENSATION In general, we assume a perspective that treats texture signals as low amplitude high frequency tactile feedback, superimposed onto the underlying low frequency kinesthetic feedback. As shown in Figure 1, we obtain the texture signal as the difference e(t) between the original haptic signal f (t) and its low-frequency (LF) coded version d(t). The lowpass filtered version l(t) of the haptic signal is treated for compression using the methods described in Section II. The HF texture signal e(t) is processed separately, as described in the following paragraph. At the time of playback, the reconstructed error signal eˆ(t) is superimposed onto the reconstructed deadband compressed LF signal to obtain the fˆ(t)

f(t) (N)

The state-of-the-art approaches [2], [3] minimize the number of stored haptic samples using an irregular subsampling process. This process exploits limitations of human haptic perception, represented by the Weber’s law of psychophysics. According to this law, a stimulus I is compared with a reference stimulus Iref , ΔI being the difference between them. This difference is only perceivable if it exceeds the JND. Such a relationship can be mathematically described as ΔI/Iref = k where k denotes the deadband parameter and can be empirically determined for several types of stimuli such as force, velocity, pain and temperature [5]. Translated to haptic signal compression [6], [2], [3], whenever an estimated sample d(t) falls within the so-called deadband thresholds f (t) ± Δf (t), the original sample f (t) can be dropped and the estimate can be displayed instead. If the estimate violates the perception thresholds, the difference is said to be perceivable and the acquired sample needs to be transmitted/stored. The offline compression approach proposed in [2] employs a procedure similar to the real-time approach expounded in [6]. However, since no temporal constraints are present for the offline case, the signal is smoothed after the subsampling process through linear interpolation of the stored samples. This procedure is performed to avoid sudden intensity variations on the reconstructed signal due to the update steps. Although the approach in [2] has low-complexity, the reconstructed signal cannot be guaranteed to be completely contained within the deadband thresholds since the tested predictions are not the ones employed during the final signal reconstruction.

LFcoding

0.6 0.55 0.5 0.45 0.6

d(t) (N)

II. O FFLINE CODING OF HAPTIC SIGNALS

l(t) ()

0.55

fhat(t) (N)

a linear predictive coding (LPC) filter. Subsequently at haptic display time, haptic signals are synthesized at the output of this filter driven by white noise. The LPC filter is represented by coefficients significantly lesser in proportion to the number of original signal samples, while simultaneously achieving high correlation between the synthesized and the real world recorded signals. All these works seek to develop methods for texture modeling, synthesis and/or prediction for unseen scenarios. The model determination process and the fidelity judgment were not optimized subject to perceptual signal quality constraints. On the other hand, motivated by storage space minimization via haptic media compression, we take a fundamentally different approach - source coding for the quantized HF tactile component - where the coarseness of the quantization is optimized based on subjective quality ratings for the texture display. This paper is organized as follows. Section II describes in brief the state-of-the-art offline coding algorithms. Section III describes the method proposed to compress HF texture signals and thereby achieve overall data reduction for offline haptic signals. This is followed by a report on the subjective tests carried out to perceptually optimize the data compression process in Section IV. Finally Section V describes and analyzes the experimental results in terms of subjective quality and achieved compression performance.

0.6 0.55

time (ms)

0.5 0.45 time (ms)

0.5 0.45 200

400

600

time (ms)

800

1000

Fig. 2. Example signals for the proposed approach: f (t): original recorded haptic signal, d(t): LF deadband-compressed signal using the approach in [2] and fˆ(t): reconstructed signal ready for playback.

Description Excellent, no perceivable artifacts Good, minor artifacts Fair, some artifacts Poor, several artifacts Very Bad, too many artifacts

haptic

Rating 100 75 50 25 0

visual

TABLE I S UBJECTIVE RATING SCHEME . SensAble,www.sensable.com

CHAI3D,www.chai3d.org

Fig. 3. Experimental setup. The human operator interacts with the virtual environment through the human-system interface (haptic device on the left hand side). The virtual end-effector is illustrated as the white sphere touching the textured plane on the right hand side.

signal that is sent to the haptic device for display. Figure 2 shows examples of haptic signal snippets recorded during acquisition, the deadband coded LF signal and finally the texture compensated reconstructed signal, respectively. Furthermore, the error signal e(t) has a characteristic peaky probability distribution, centered around zero. Stochastic signal source models with such properties are especially suitable candidates for entropy coding to minimize data size by removing source redundancy. An additional lossy step in the form of quantization of the error signal before performing entropy coding improves coding performance. In our work, the strength of the quantization is tuned on the basis of subjective quality ratings obtained from psychophysical tests described next. IV. S UBJECTIVE TESTS A. Experimental setup An illustration of the experimental setup is shown in Figure 3. Using the CHAI3D library [10], a variety of textures are generated with micro-scale geometry represented by heightfields. Interaction with these textured surfaces are recorded for subsequent processing and playback. The recorded texture signals are encoded and displayed to the subjects using a PHANTOM Omni haptic device (SensAble Technologies Corp., Woburn, MA) . Each recording session, roughly 11 seconds long, comprises of exploration of the textured surface with an approximately constant velocity. The resultant forcefeedback data is processed offline and the proposed coding scheme is applied. For the LF coding part (refer to Figure 1), the approach in [2] was employed and the deadband parameter k is varied over a range of 1% (very conservative) to 50%. As regards the HF tactile component, an Huffman entropy coder [11] was utilized and the bit budget b allocated to quantize the error signal assumes the following values: 2, 5 and 8 bits. B. Procedure Ten subjects, of which 8 were male and 2 females, with ages between 20 and 30 years participated in the experiment. The participants were instructed to hold the haptic device stylus approximately at the center of the device workspace, their wrist resting on a soft pad for comfort. They held the stylus passively while perceiving the playback of a session recorded previously.

The experiment was divided into three sessions corresponding to three different textures. Each session was further divided into subsessions corresponding to various combinations of parameter settings mentioned in Section IV-A. Under every session, the subsessions were conducted in a random order. At the beginning of every session, i.e. before each new texture, the subjects were made familiar to the original recorded signal and to the perceptual effects of compression artifacts. Subjective ratings were obtained for subsessions on the basis of the perceived difference of the displayed haptic signal to the originally recorded one according to the rating scale shown in Table I. The subjects had the facility to return to the original recording at any time during the experiment to keep a fresh reference. V. R ESULTS A. Storage savings due to compression We gauge the performance of the coding scheme presented in this paper objectively by measuring the savings in storage space obtained. Accordingly, the compression ratio is calculated as the ratio of the length of the compressed bitstream to that of the original bitstream. The results of these calculations for the three different textures are shown in Figure 4 for a range of deadband parameter k values and number of quantization levels. Evidently the application of pure deadband coding (DB only) without texture compensation achieves the best compression results for all of the textures. As expected, increasing the bit budget for the HF tactile component results in worse compression performance, which manifests itself in the form of upward shifts of compression ratio curves. These results, however, have to be read in conjunction with the performance of the coding scheme in terms of subjective quality ratings for the reconstructed signals, analyzed in the next section. B. Subjective quality Subjective ratings obtained from the experiment for each subsession are plotted in Figure 5. The application of pure deadband coding without texture compensation performs the worst among all the deployed approaches. This is expected, since the important HF tactile elements present in the original recorded signal are completely rejected during reconstruction. For the coding scheme with texture compensation, it can be seen that subjective signal quality is comparable for the bit budgets b = 5 and b = 8 allocated to the HF component. Overall, for the cases covered in our experiments, the proposed

100

6

80

SQR1

CR1 (%)

8

4 2

60 40

0

20

8

100

DB only b = 2 bits b = 5 bits b = 8 bits

6 4 2

SQR2

CR2 (%)

0

DB only b = 2 bits b = 5 bits b = 8 bits

50

0

8

100

6

SQR3

CR3 (%)

0

4 2 0 0

0.1

0.2

0.3

0.4

0.5

deadband parameter k Fig. 4. Objective quantification of storage savings due to force-feedback signal compression for three different textures in terms of the compression ratio (CR). CR1: stone texture, CR2: sand texture and CR3: cork texture.

coding scheme performs very well in terms of both storage savings and signal quality if the quantizer granularity is set according to a bit budget of 5. VI. C ONCLUSION A novel perceptual offline coding approach based on quantization and source coding techniques is presented in this paper. Instead of discarding important HF tactile components of a haptic recording in favor of higher compression ratios, the proposed approach finds a good compromise between storage savings and perceived reconstructed signal quality for an end-user. At the cost of only a slight increase (3.5%) in the compression ratio, more than five-fold improvement in perceived quality is obtained. Future work will involve exploration of and comparison with alternative approaches for modeling the HF texture signal, for e.g. LPC analysis, followed by texture synthesis during playback based on the estimated LPC coefficients. The number of LPC coefficients will decide the trade-off between the compression factor and signal quality. Furthermore, the adaptation and application of such approaches to real-time online coding also needs to be studied. ACKNOWLEDGMENT This work has been supported, in part, by the EuropeanBrazilian Network for Academic Exchange (EUBRANEX) and, in part, by the German Research Foundation (DFG) under project STE 1093/4-1.

50

0 0

0.1

0.2

0.3

0.4

deadband parameter k

0.5

Fig. 5. Subjective quality rating (SQR) for playback of recorded and coded force-feedback for three different textures. SQR1: stone texture, (b) SQR2: sand texture and SQR3: cork texture. The dashed horizontal line marks a ”good” quality level corresponding to a numerical rating of 75 according to the rating scheme presented in Table I.

R EFERENCES [1] C. Shahabi, A. Ortega, and M. R. Kolahdouzan, “A comparison of different haptic compression techniques,” in Proc. of the Int. Conf. on Multimedia and Expo (ICME), Lausanne, Switzerland, Aug. 2002, pp. 657–660. [2] J. Kammerl and E. Steinbach, “Deadband-based offline-coding of haptic media,” in Proc. of the ACM Multimedia (ACMMM), Vancouver, Canada, Oct. 2008, pp. 549–558. [3] F. Brandi and E. Steinbach, “Minimizing the number of support points in perceptual offline haptic compression,” in Proc. of the IEEE Int. Workshop on Haptic Audio Visual Environments and Games (HAVE), Phoenix, USA, Oct. 2010, pp. 1–6. [4] E. Weber, Die Lehre vom Tastsinn und Gemeingefuehl, auf Versuche gegruendet. Vieweg: Braunschweig, Germany, 1851. [5] C. Sherrick and J. Craig, Tactual Perception: A Sourcebook. Cambridge University Press, 1982. [6] P. Hinterseer, S. Hirche, S. Chaudhuri, E. Steinbach, and M. Buss, “Perception-based data reduction and transmission of haptic data in telepresence and teleaction systems,” IEEE Trans. on Signal Processing, vol. 56, no. 2, pp. 588–597, Feb. 2008. [7] C. Zilles and J. Salisbury, “A constraint-based god-object method for haptic display,” in Proc. of the IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, vol. 3, Los Alamitos, USA, 1995, pp. 146–151. [8] M. Minsky and S. J. Lederman, “Simulated haptic textures: Roughness,” in Proc. of the ASME Dynamics Systems and Control Division, vol. 58, 1996, pp. 421–426. [9] J. Romano, T. Yoshioka, and K. Kuchenbecker, “Automatic filter design for synthesis of haptic textures from recorded acceleration data,” in Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA), May 2010, pp. 1815–1821. [10] F. Conti, F. Barbagli, D. Morris, and C. Sewell, “CHAI 3D: An opensource library for the rapid development of haptic scenes,” in Proc. of the IEEE World Haptics, Pisa, Italy, Mar. 2005. [11] D. A. Huffman, “A method for the construction of minimum redundancy codes,” Proc. of the IRE, vol. 40, pp. 1098–1101, 1952.