DEEP LEARNING FOR SPEECH CLASSIFICATION ...

DEEP LEARNING FOR SPEECH CLASSIFICATION AND SPEAKER RECOGNITION by Muhammad Muneeb Saleem

APPROVED BY SUPERVISORY COMMITTEE:

Dr. John H.L. Hansen, Chair

Dr. P.K. Rajasekaran

Dr. Yang Liu

c 2014 Copyright Muhammad Muneeb Saleem All rights reserved

To my parents, and to all seeking their right of education to become better world citizens

DEEP LEARNING FOR SPEECH CLASSIFICATION AND SPEAKER RECOGNITION

by

MUHAMMAD MUNEEB SALEEM, BS

THESIS Presented to the Faculty of The University of Texas at Dallas in Partial Fulfillment of the Requirements for the Degree of

MASTER OF SCIENCE IN ELECTRICAL ENGINEERING

THE UNIVERSITY OF TEXAS AT DALLAS December 2014

UMI Number: 1583645

All rights reserved INFORMATION TO ALL USERS The quality of this reproduction is dependent upon the quality of the copy submitted. In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if material had to be removed, a note will indicate the deletion.

UMI 1583645 Published by ProQuest LLC (2015). Copyright in the Dissertation held by the Author. Microform Edition © ProQuest LLC. All rights reserved. This work is protected against unauthorized copying under Title 17, United States Code

ProQuest LLC. 789 East Eisenhower Parkway P.O. Box 1346 Ann Arbor, MI 48106 - 1346

ACKNOWLEDGMENTS The author conveys sincere appreciation to his advisor, Dr. John H.L. Hansen, for giving the opportunity to work on this thesis and providing valuable guidance throughout. This thesis would not have been possible without the unconditional support and encouragement of my parents and siblings towards my pursuit of a Master’s degree. The author would like to thank all fellow lab members and postdocs for their help, interesting discussions, and for providing a great environment conducive to creative thinking and research. We also acknowledge the financial support provided by AFRL under contract FA8750-12-1-0188 for this project. December 2014

v

DEEP LEARNING FOR SPEECH CLASSIFICATION AND SPEAKER RECOGNITION Publication No. Muhammad Muneeb Saleem, MS The University of Texas at Dallas, 2014

Supervising Professor: Dr. John H.L. Hansen

Deep learning is the state-of-the-art technique in machine learning with applications in speech recognition. In this study, an efficient system is formulated to process large amounts of speech data within the deep learning framework by harnessing the parallel processing power of HighPerformance Computing oriented Graphics Processing Unit (GPU). This thesis focuses on applications of this approach to address stressed speech classification as well as discrimination between different flavors of noise-free speech under Lombard Effect. Different architectures of deep neural networks (DNN) are explored to build state-of-the-art classifiers for detection and classification of stressed speech and Lombard Effect flavors. Furthermore, applications of deep networks are explored to improve current state-of-the-art speaker recognition systems. Further integration of discriminative deep architectures is accomplished for unsupervised methods in training front-ends for Speaker Recognition Evaluation systems.

vi

TABLE OF CONTENTS ACKNOWLEDGMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

v

ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

vi

LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

ix

LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

x

CHAPTER 1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1

CHAPTER 2 DEEP LEARNING . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4

2.1

Deep Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4

2.2

Restricted Boltzmann Machines . . . . . . . . . . . . . . . . . . . . . . . . .

5

2.3

Finetuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5

CHAPTER 3 GPU ARCHITECTURE . . . . . . . . . . . . . . . . . . . . . . . . .

7

3.1

Benchmarking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8

CHAPTER 4 STRESSED SPEECH CLASSIFICATION . . . . . . . . . . . . . . .

9

4.1

SUSAS Corpus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9

4.2

Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9

CHAPTER 5 LOMBARD EFFECT DISCRIMINATION . . . . . . . . . . . . . . .

11

5.1

Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11

5.2

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11

5.3

Corpus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12

5.4

Lombard Flavor Classification . . . . . . . . . . . . . . . . . . . . . . . . . .

13

5.4.1

DNN Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . .

13

5.4.2

Deep Classifier Performance . . . . . . . . . . . . . . . . . . . . . . .

15

DNN-assisted Speaker Recognition . . . . . . . . . . . . . . . . . . . . . . .

17

5.5.1

i-Vector based Speaker Identification (SID) System . . . . . . . . . .

17

5.5.2

Training Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

18

Speaker Identification Results . . . . . . . . . . . . . . . . . . . . . . . . . .

19

5.5

5.6

vii

CHAPTER 6 UNSUPERVISED DISCRIMINATIVE MODELING . . . . . . . . . .

22

6.1

Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

22

6.2

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

24

6.3

Discriminative Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

25

6.3.1

GMM-based DNN

. . . . . . . . . . . . . . . . . . . . . . . . . . . .

25

6.3.2

RBM-based DNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

26

6.3.3

DNN Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . .

28

6.3.4

i-Vector Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . .

29

Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

30

6.4.1

Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . .

30

6.4.2

Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

33

CHAPTER 7 CONCLUSIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

35

6.4

7.1

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

35

7.2

Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

36

REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

38

VITA

viii

LIST OF FIGURES 2.1

Architecture differences in DBN and DNN, conversion from DBN to DNN is done by replacing their top-most layers. . . . . . . . . . . . . . . . . . . . . . . . . . .

6

3.1

DNN Training times on SUSAS dataset. . . . . . . . . . . . . . . . . . . . . . .

8

4.1

DNN pre-training options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10

5.1

System block diagram of i-Vector based SID for Lombard speech. Data 1 and 2 correspond to raw features for UBM and TV matrix, respectively. Here, both are the same as training data. ‘Audio data’ reflects all acoustic data used in verification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15

System level block diagram of Deep Neural Network interface with SID systems (‘+’ indicates presence of non-target speech types in test data). . . . . . . . . .

18

System block diagram of integration of GMM-based DNN with i-Vector system along with datasets used at each step. . . . . . . . . . . . . . . . . . . . . . . .

26

Top layer of a deep belief network (stacked RBM layers) is replaced by a softmax classification layer after extracting the index of the hidden node at its peak activation normalized over the entire training set. . . . . . . . . . . . . . . . . .

28

5.2 6.1 6.2

ix

LIST OF TABLES 5.1

Classification Accuracies for Neutral and Lombard speech types; Unweighted means raw accuracy on test data, while Balanced means adjusted/weighted accuracy per class. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15

5.2

Confusion matrix for 4-way classification between neutral and Lombard speech (Classification rates are in %, figures in bold refer to matched train/test conditions). 16

5.3

SID performance with DNN classifier (%EER); ‘+’ indicates a minority presence of non-target speech types in test data. ‘All’ error rates are for all 4 systems combined. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

20

5.4

EER and Relative Improvement Comparison of different training methods. . . .

21

6.1

Results on SRE-2010 Male Core Telephone Data (1024 mixtures) . . . . . . . .

32

6.2

Results on SRE-2010 Male Core Telephone Data (2048 mixtures) . . . . . . . .

33

6.3

Results on UTSCOPE Lombard Corpus . . . . . . . . . . . . . . . . . . . . . .

33

x

CHAPTER 1 INTRODUCTION Deep learning is a revolutionary new approach in machine learning and artificial intelligence. It has created new performance benchmarks in a wide range of applications. The research areas benefiting the most from this technology are image processing and speech recognition. Deep neural networks have become an integral part of state-of-the-art Automatic Speech Recognition (ASR) systems (Hinton et al., 2012), and are gradually finding use in blind feature extraction (e.g., bottleneck features) and speaker recognition/identification (SID). This thesis focuses on discovering novel applications of deep neural networks (DNN) for speech classification and speaker recognition. Deep neural networks are multi-layer perceptron (MLP) which are specially trained, usually using parallel processing hardware (i.e., graphics processing units (GPU); Chapter 3). These are successfully applied for detection and classification of stressed speech against neutral speech on actual human speech data (Chapter 4). Improved classification results are achieved together with faster processing times (Chapter 2). Similar DNN architecture are applied to classify speech under Lombard Effect. Lombard Effect is described as a type of naturally produced stressed speech produced by a speaker when exposed to a noisy environment. This changes neutral speech production in terms of several reported parameters including duration, pitch, intensity, and spectral slope (Hansen, 1988). DNN based classification are applied on simple acoustic cepstral features extracted from raw speech data. This system is further expanded to improve speaker identification (SID) for speech under Lombard Effect. A creative approach is proposed using the validation results of a neural network classifier as meta-data to train multiple speaker identification 1

2 systems with proportionally weighted training data based on the a-priori information from DNN classifier. This is discussed in (Chapter 5). Recently, researchers have been searching for approaches to benefit from the modeling power of deep architectures in speech recognition for application in state-of-the-art speaker identification (SID) systems. Modern SID systems use speaker variability based on extracted features to build i-Vector models which are ultimately used for speaker identification. This study seeks to apply deep neural networks and deep belief networks (based on Restricted Boltzmann Machines) to extract the sufficient statistics required to build speaker models. Statistics extracted using DNNs with multi-frame concatenated speech features will be shown to outperform the statistics obtained by traditional Gaussian Mixture Model based Universal Background Model (Chapter 6). Gaussian Mixture Models (GMM) are currently used in state-of-the-art i-Vector based speaker recognition systems to cluster the development data for acoustic space modeling and to build a universal background model. Each Gaussian mixture within a trained UBM represents a cluster of similar features, hence dividing the acoustic space into regions modeled as Gaussian distributions. The posterior probabilities for each feature belonging to one of these regions is obtained through this model. The UBM is trained via the unsupervised Expectation-Maximization algorithm using unlabeled development data. Once trained, it is used to predict posterior probabilities for enrollment and verification data belonging to all speakers. These posteriors, called the Baum-Welch statistics are a core element for the extraction of i-Vector features ultimately used in speaker trials. This novel approach will be shown to produce highly competitive performance, and a universal background model (UBM) based DNN will be shown to outperform the baseline by a significant margin. This work highlights the potential of deep neural networks and restricted Boltzmann machine techniques to replace the current UBM based acoustic clustering and prediction approach to i-Vector feature extraction. This thesis explores new dimensions to

3 speech processing and speaker recognition by integrating with the cutting edge techniques of machine learning which have also showed success in speech recognition systems.

CHAPTER 2 DEEP LEARNING 2.1

Deep Neural Networks

The simplest form of deep neural networks are exactly like standard multi-layer perceptron (MLP) models. The new term basically refers to several advancements in algorithm and hardware that have made these models much more successful than simple MLP models. The standard MLP training strategy is to randomly initialize weights of the whole network, and then train the weights using backpropagation algorithm with any gradient descent algorithm e.g. stochastic gradient descent, conjugate gradient descent. But, backpropagation does not work well for these networks if the weights are randomly initialized. Such training often moves to a local minimum of the loss function which is too high for practical purposes. Deep networks trained with back-propagation (without unsupervised pre-train) perform worse than shallow networks. This has limited artificial neural networks or MLPs to be limited to one or two layers. Deep learning solves this problem by unsupervised ‘pre-training’ of weights. This pretraining is usually done layer by layer, using Restricted Boltzmann Machines (RBM) or stacked Autoencoders. Such pre-training gives initial weight values to the network which are highly suitable for that dataset. Usually this pre-training creates a generative model which extracts high-level features from the input data in each layer, resulting in successively higher-level robust features that represent patterns in input frames. 4

5 2.2

Restricted Boltzmann Machines

Stacked RBMs, or deep belief networks (DBN) are trained in a greedy layer-wise pattern (Hinton et al., 2006). Each layer is generatively trained as a Restricted Boltzmann Machine using the Contrastive Divergence (CD) algorithm with Gibbs sampling. Training the network layer-wise in this manner results in a new layer of features built on lower level sub-features. For speech data, the input/visible layer nodes are trained as Gaussian nodes while all hidden layer nodes as Bernoulli nodes.

2.3

Finetuning

Once the network has been pre-trained using RBMs, the whole pre-initialized network is ‘fine-tuned’ for classification using backpropagation. For classification, a new output layer is added to the network which usually employs the Softmax function to output results as class probabilities. This layer enables the DNN to output classification results as class probabilities which sum to 1. Target classes are expressed by Y , the weight matrix and bias vector by W and b respectively.

P (Y = i|x, W, b) = softmaxi (Wx + b) eWi x+bi = P Wj x+bj je

(2.1)

The classification result is obtained by noting the index of the node with the maximum class probability:

ypredict = argmaxi P (Y = i|x, W, b).

(2.2)

6

DBN

DNN

Pre-training

Finetuning

Generative training

Discriminative training

Hidden layer

Softmax layer Hidden layer Hidden layer

Hidden layer Input frames Figure 2.1. Architecture differences in DBN and DNN, conversion from DBN to DNN is done by replacing their top-most layers.

CHAPTER 3 GPU ARCHITECTURE In addition to better weight initialization algorithms, the other major enabler of deep learning techniques is better and faster hardware architectures to support such intensive computational algorithms. Processing massive amounts of training data with large neural networks has resulted in new systems which have outperformed all previous benchmarks. This Big Data issue is often solved for deep neural networks using a Graphics Processing Unit (GPU) which has thousands of cores built for parallel processing. For the purpose of this study, minibatch-wise training was employed, which processes the massive data in large chunks or matrices. The matrix manipulations involved in backpropagation often involve calculating matrix products; which are easily optimized by parallel processing of GPUs, which were originally designed to process image matrices. A new system was built for the purpose of all research carried out under this thesis, to enable running deep learning methods efficiently. The latest Central Processing Unit (CPU) was used as a baseline, and the Nvidia GeForce GTX Titan Black model was used as the GPU. This GPU is built using the Nvidia Kepler GPU architecture for High-Performance Computing (HPC). The system has the following specifications: • Central Processing Unit (CPU): Intel i7-4820K Quad-Core, 3.7-3.9 GHz • Graphics Processing Unit (GPU): Nvidia GeForce GTX Titan Black, 1072 MHz, 2688 Cores • RAM: 16 GB • Hard-disk: 500 GB Solid-state Drive (SSD) 7

8 3.1

Benchmarking

For the purpose of benchmarking and performance comparison, DNN training times were compared by running the system only on CPU vs running the GPU-optimized functions on GPU and memory based processes on CPU. The data used was from the SUSAS corpus discussed in next section. • Total 39-dim frames in data: 358841 • Network size: 39-1000-1000-100-5, timed till 10 epochs • 5-8x speedup with GPU vs CPU

Figure 3.1. DNN Training times on SUSAS dataset.

CHAPTER 4 STRESSED SPEECH CLASSIFICATION The deep learning system is evaluated for detection of emotion and stress in speech using the SUSAS and UT-SCOPE corpora. The effect of pre-training was also observed on DNN training time.

4.1

SUSAS Corpus

The SUSAS corpus is based on Speech Under Simulated and Actual Stress (SUSAS) under different stress types. The data used for this experiment is speech from speakers while riding a roller coaster. Five different types of stressed speech are included: high stress, medium stress, neutral, freefall, scream. Input feature vectors are based on 39-dimensional Mel-frequency cepsrtal coefficients (MFCCs); containing static, delta and delta-delta coefficients including those for energy. The data division for the purpose of training was: Training 74%, Test 26%. All data was normalized over training set to unit variance and zero mean as preprocessing for DNN.

4.2

Performance

DNN models were trained both with and without unsupervised RBM-based pre-training to observe the effect on optima reached and time to train. It was observed that pre-training process speeds up learning by about 8 epochs, but no additional benefit seen as per the loss function minima reached. This was attributed to the relatively small data set compared to the large input vectors of images. The DNN architecture consists of three hidden layers and x output nodes: 39-500-500-100-x. 9

10

Figure 4.1. DNN pre-training options.

Performance of the classifier was impressive in detecting stressed speech for two tasks: binary detection and precise type-of-stress detection. For binary stress classification (i.e. High/Medium/Freefall/Scream vs Neutral), an unweighted accuracy of 87.2% whereas a weighted accuracy of 74.0% was observed. For the 5-way classification task, an unweighted accuracy for 5 classes of 82.9% was seen. This was much better than an i-Vector based reference baseline system (57.7%).

CHAPTER 5 LOMBARD EFFECT DISCRIMINATION 5.1

Background

Lombard Effect is described as a type of stressed speech produced by a speaker when exposed to a noisy environment. This changes neutral speech production in terms of several reported parameters including duration, pitch, intensity, and spectral slope (Hansen, 1988; Hansen and Varadarajan, 2009). Lombard Effect in speech data has been shown to severely impact performance of speech systems (Hansen and Varadarajan, 2009; Varadarajan and Hansen, 2006). Different compensation schemes have been proposed to counter this impact in speech recognition systems (Hansen, 1994; Chi and Oh, 1996; Boril and Hansen, 2010) and a few for speaker identification (SID) systems (Hansen and Varadarajan, 2009). Deep neural networks (DNN) have been proven to work well for speech recognition tasks (Mohamed et al., 2012; Hinton et al., 2012) but have rarely been applied for stressed speech classification (Stuhlsatz et al., 2011) or speaker recognition under Lombard Effect. This study explores the capability of deep neural networks in extracting information from stressed speech under Lombard Effect. Furthermore, we explore the use of this information in building a robust SID system that is resilient towards the effects of background noise in human speech using metadata from the validation phase of DNN training.

5.2

Overview

The presence of Lombard Effect in speech is proven to have severe effects on the performance of speech systems, especially speaker recognition. Varying kinds of Lombard speech are produced by speakers under influence of varying noise types (Hansen and Varadarajan, 2009). 11

12 This study proposes a high-accuracy classifier using deep neural networks for detecting various kinds of Lombard speech against neutral speech, independent of the noise levels causing the Lombard Effect. Lombard Effect detection accuracies as high as 95.7% are achieved using this novel model. The deep neural network based classification is further exploited by validation based weighted training of robust i-Vector based speaker identification systems. The proposed weighted training achieves a relative EER improvement of 28.4% over an iVector baseline system, confirming the effectiveness of deep neural networks in modeling Lombard Effect.

5.3

Corpus

The speech data utilized in this study was drawn from the UT-SCOPE (Speech under COgnitive and Physical stress and Emotion) database. Details about the database can be found in (Ikeno et al., 2007). Speech was collected from speakers under nine different noisy environments. It must be noted that noise was played through open-air headphones so all data is noise-free clean speech. Three noise types were considered: large crowd noise (LCR) at 70, 80, and 90 dB-SPL, highway noise (HWY) in a car at 70, 80, and 90 dB-SPL, and pink noise (PNK) at 65, 75, and 85 dB-SPL. Neutral speech data was also collected from the same speakers for comparative analysis. The speech comprises of 20 phonetically balanced TIMIT sentences, five repetitions of 10 digits, and spontaneous speech. Speech files consist of an average of 3 seconds of data, which makes it challenging for speaker recognition. Subjects included 24 female and 6 male speakers. After randomizing, 75% was used as training and validation, while the rest was used as test data; both for modeling deep neural network and speaker identification system; to prevent overlap in training and testing.

13 5.4

Lombard Flavor Classification

It has been shown that speech under Lombard Effect severely deteriorates speaker identification (SID) systems (Hansen and Varadarajan, 2009). This study will focus on a novel method to significantly reduce errors in a demanding application like SID. The method employs the classification power of deep neural networks in detecting three kinds of speech under Lombard Effect and neutral speech (NEU). The classification into three different ‘flavors’ of Lombard speech and neutral speech assists in building weighted models for speaker identification. The three Lombard Effect flavors are under large crowd noise (LCR), highway noise (HWY) and pink noise (PNK), while speech without Lombard Effect is classified as neutral (NEU). 5.4.1

DNN Architecture

For features, 39-dimensional Mel-frequency Cepstral Coefficients (MFCC) are extracted, which include static, delta, and delta-delta coefficients. A 25ms Hamming window with 10ms shift was applied. The feature vectors are normalized to zero mean and unit variance to enable learning via neural networks. This normalization is done for the training set only; the mean and variance on training data is then used to scale the validation and test data. This paper uses the effectiveness of deep neural networks in extracting deeper meanings from simple cepstral features. A deep neural network is randomly initialized for classification purposes without generative pre-training. Pre-training using Restricted Boltzmann Machines (Hinton et al., 2006) was found to result in suboptimal results for Lombard Effect classification. The architecture comprises of a Multi-layer Perceptron with sigmoid activation functions in the hidden layers. The visible layer consists of nodes for feature vector input. The number of hidden layers tested ranged from 1 to 11, with increased number for increased classification complexities. The output layer consists of logistic regression nodes employing the softmax function.

14 This layer enables the DNN to output classification results as class probabilities which sum to 1. Target classes are expressed by Y , the weight matrix and bias vector by W and b respectively.


(5.1)

The classification result is obtained by noting the index of the node with the maximum class probability:

ypredict = argmaxi P (Y = i|x, W, b).

(5.2)

Minimization of cross-entropy error is set as the objective function, which maximizes target class membership probabilities on training data. The loss function is expressed as negative log-likelihood,

`(θ = {W, b}, D) = −

|D| X

log(P (Y = yi |xi , W, b)).

(5.3)

i=0

Mini-batch Stochastic Gradient Descent (Zhang, 2004) is used to train the DNN under the backpropagation algorithm. To introduce better regularization in the DNN model so that it performs better on test data, L2-norm regularization is applied. Also called ‘weight decay’, this regularization method prevents overfitting by preventing the weight parameters from becoming very large. Training such a large network on a large corpus was made possible by harnessing the massive parallel processing power of a Graphics Processing Unit (GPU) (Bergstra et al., 2010).

15

Audio Data Raw Feature extraction Front-end

UBM

Data UBM1

Total Variability Matrix Data 2

i-vector postprocessing

PLDA

Scores

Back-end

Figure 5.1. System block diagram of i-Vector based SID for Lombard speech. Data 1 and 2 correspond to raw features for UBM and TV matrix, respectively. Here, both are the same as training data. ‘Audio data’ reflects all acoustic data used in verification.

Table 5.1. Classification Accuracies for Neutral and Lombard speech types; Unweighted means raw accuracy on test data, while Balanced means adjusted/weighted accuracy per class. Classification Neutral/ Neutral, Neutral, Noise Type Lombard Noise-type -type/level Classes 2 4 10 Hidden layers 6 8 11 Unweighted 95.7 69.1 60.0 Balanced 94.9 66.0 49.4

5.4.2

Deep Classifier Performance

Lombard and Neutral Speech Classification The normalized acoustic features are submitted to the network in batches, and the network is trained to classify them into neutral speech or Lombard Effect. For binary classification between neutral speech and any of the Lombard Effect flavors, 95.7% accuracy was achieved using a 6-hidden-layer neural network (number of nodes: 39-3000-2000-1000-500-100-502), raising existing benchmarks. A balanced accuracy of 94.9% per class was achieved as mentioned in Table 5.1. This is a huge improvement over the results reported in (Hansen and Varadarajan, 2009) using a GMM based classifier; where an average accuracy of 81.5% is reported for binary classification task on female-only speech.

16

Table 5.2. Confusion matrix for 4-way classification between neutral and Lombard speech (Classification rates are in %, figures in bold refer to matched train/test conditions). Test Condition NEU LCR HWY PNK

NEU 94.2 5.2 4.6 6.2

LCR HWY 0.7 1.4 43.3 21.7 17.3 62.8 18.1 12.5

PNK 3.7 29.8 15.3 63.2

Lombard Noise-type and Neutral Speech For the four-way classification task into neutral speech and three noise-independent Lombard flavors (LCR, HWY, PNK), accuracy as high as 69.1% was achieved with DNN. Classification performance was validated against varying DNN depths by varying the number of hidden layers from 2 to 12. Best performance was achieved with 8 hidden layers, resulting in 10 total layers including visible and output nodes. Table 5.2 shows the confusion matrix for classification results. The resulting classifier is used for adaptation of a state-of-the-art SID system in the next section.

Lombard Noise-type, Noise-level and Neutral Speech Classification was also performed on the same data by further refining the classified Lombard Effect flavor into the 3 different noise levels behind each of them. An overall accuracy of 60% was achieved by a single DNN model in classifying all 9 Lombard flavors (3 noise levels against 3 noise types) and neutral speech.

Results and Analysis Referring back to Table 5.1, it shows DNN classification performance over different combinations of Lombard Effect flavors. Unweighted accuracy is for all samples in testing data which contain unbalanced classes. Balanced accuracy is calculated to balance class biases.

17 It is evident that the classifier performs well even with slightly biased training for neutral speech and all Lombard Effect flavors. The relatively larger gap in accuracy when additionally classifying the type of noise shows that noise-level is more sensitive to classification compared to noise-types. Varying levels of depths were required to achieve effective classification. Increasing number of hidden layers were employed as classification complexity increased from binary to 10way classification. The lower balanced accuracy for 10-way classification experiment stresses the need for balanced training and testing. The results show that after careful tuning of neural network parameters (learning rate, momentum, regularization, nodes per layer, and number of hidden layers), even complex phenomena such as Lombard Effect can be effectively modeled.

5.5 5.5.1

DNN-assisted Speaker Recognition i-Vector based Speaker Identification (SID) System

The classification system includes feature extraction and back-end modeling, which is illustrated in Fig. 5.1. MFCC features are referred to as raw features, since they can be further processed into refined features such as i-Vectors. An i-Vector based system is the state-ofthe-art platform for acoustic event identification, such as SID (Dehak et al., 2011a), and language ID (Martınez et al., 2011; Zhang et al., 2014; Liu et al., 2012, 2011). However, it has not been explored for Lombard speech. i-Vectors are extracted following factor analysis (Dehak et al., 2011a). The i-Vector model is represented by:

M = m + Tω

(5.4)

where T is the total variability matrix, ω is i-Vector, m is the universal background model (UBM) mean super-vector, and M is the super-vector derived from raw features. The ex-

18

UT-SCOPE Database

Mean, Variance Normalization on training set

Deep Neural Network

NEU+

SID System

LCR+

SID System

HWY+

SID System

PNK+

SID System

Figure 5.2. System level block diagram of Deep Neural Network interface with SID systems (‘+’ indicates presence of non-target speech types in test data).

traction converts frame length-varied spectral features matrix into a fixed-dimension features vector for each speech utterance. All available training data are employed to train both the UBM and total variability matrix using the EM algorithm. Next, the i-Vectors for both training and test sets are extracted with the total variability matrix. 100-dimensional i-Vectors are used for the purpose of this experiment which suits the relatively small database used. The extracted i-Vector of each speech utterance contains both inter-speaker and intraspeaker variabilities. Therefore, the PLDA classifier is employed in SID systems (Liu et al., 2013; Kenny, 2010). PLDA is also adopted as back-end classifier here (Fig. 5.1). 5.5.2

Training Methods

Four separate SID systems are trained for each type of speech; under the three Lombard Effect flavors and one for neutral speech. Test data is classified by the DNN as belonging to either of these four categories. Based on results from classifier, the test data classified in each class is forwarded to the SID system trained with the respective class data. The overall system is illustrated in Fig. 5.2. Two kinds of approaches are analyzed in this paper. Fixed Training The first method forks speech training data and uses speech under only a single Lombard Effect or only neutral speech to train each of the four SID systems. It uses neutral speech

19 to train one SID system, training data of Lombard Effect speech under large crowd noise in second SID system, and so on to train all four SID systems.

Weighted Training A second more innovative approach is proposed in our study. The weighted approach makes use of validation results from DNN to build SID models better adapted for each test dataset subsequently classified by the DNN. DNN classification result on validation data is monitored to observe the percentage of non-target class samples present in each of four classified sets of data. Since this validation data more closely resembles the practical results on test data by the classifier, this class distribution is used to add non-target Lombard Effect speech samples in the training data for each of the four SID systems. The additional data for training is added so that training set classes are probabilistically in the same proportion as the validation set. This makes each of the four speaker identification systems robust towards samples from another class, be it a Lombard flavor or neutral speech. This method outperforms the already high performing baseline trained on all classes as shown in next section.

5.6

Speaker Identification Results

Baseline The baseline SID system is trained with the full set of training data including neutral speech and all 3 noise-free Lombard Effect flavors. The same i-Vector based SID system was used for all training methods. The baseline gives an upper bound on performance because Lombard Effect speech has also been included in the train data.

20

Table 5.3. SID performance with DNN classifier (%EER); ‘+’ indicates a minority presence of non-target speech types in test data. ‘All’ error rates are for all 4 systems combined. Training Fixed Weighted

NEU+ LCR+ HWY+ 1.30 0.20 0.76 0.75 0.17 0.47

PNK+ All 0.71 1.27 0.47 0.53

Fixed Training Fixed training does not exploit all information from DNN classification. This is because of the presence of around 31% test samples belonging to other flavors or neutral speech. This method adversely affects neutral speech in particular because of the adverse impact of Lombard speech. Table 5.3 shows the error rates for each set of test data classified by the DNN as belonging to a particular type of Lombard or neutral speech. The test data in each column represents a majority of samples belonging to a speech type along with misclassified samples belonging to the other 3 classes. Weighted Training Each of the four SID systems are trained with one of the Lombard flavors or neutral speech, and a part of non-target class training data (for each of the three remaining classes) is included in proportion to the validation data classification results by DNN. This proportional inclusion prevents the SID system from being blind to other possible flavors, and thus avoids overfitting. Table 5.3 highlights the improvement over other systems, which is due to the inclusion of training data for DNN-misclassified speech in training the SID system. The proposed system outperforms the baseline system without DNN classification by +28.4%. Discussion Table 5.4 highlights overall error rates for the weighted and unweighted models in the presence of a DNN classifier, and the baseline. The probabilistically weighted training method

21

Table 5.4. EER and Relative Improvement Comparison of different training methods. DNN classifier Training EER (%) Rel. Imp. (%)

Absent Present Baseline Fixed Weighted 0.74 1.27 0.53 -71.6 +28.4

exhibits an overall improvement of +28.4% in EER for the final task of speaker identification in presence of Lombard speech. Since the database contains three different noise levels behind each kind of Lombard speech produced, the proposed system performance also exhibits its resilience towards varying levels of background noise, induced Lombard Effect. Fixed, single-class training is unable to provide good results since it enhances the impact of Lombard Effect by narrowing training to a single type of speech, which leaves the system vulnerable to misclassified test data belonging to other speech types.

CHAPTER 6 UNSUPERVISED DISCRIMINATIVE MODELING 6.1

Background

Universal background modeling (UBM) is currently used in state-of-the-art i-Vector based speaker recognition/identification (SID) systems to cluster development data for acoustic space modeling (Dehak et al., 2011b). Each Gaussian mixture within a trained UBM represents a cluster of acoustically similar features. The posterior probabilities for each feature belonging to one of these regions is obtained through this model. The trained UBM is then used for estimating Baum-Welch statistics which are a soft alignment between acoustic features and UBM mixtures (Lei et al., 2014; Yu et al., 2014). However, due to the mismatch between data for training UBM and those for enrollment and verification, the posterior probabilities can sometimes become unreliable and therefore affect the extracted i-Vectors (Lei et al., 2013; Yu et al., 2014). In recent studies (Lei et al., 2014; Kenny et al., 2014), tied-state-triphones (senones) have been used instead of UBM mixtures and predicted with deep neural network (DNN). Previous authors have reported significant improvement over a traditional i-Vector approach on the telephone speech condition of NIST SRE 2012 database. However, the training and prediction of senones requires a significant amount (1100 hours) of transcribed data, and thus has a limitation when applied to speaker recognition in some situations (e.g., low resource languages). Motivated by this, we propose to use a UBM for space division as with conventional i-Vector based SID, but use a DNN for posterior prediction. Taking such an approach avoids the need for transcripts while still benefiting from DNN’s robust discriminative feature 22

23 learning advantages. A very similar approach has been proposed during our study exploring unsupervised methods using RBM and GMM (Campbell, 2014). While competitive results were reported, no improvement was found. The major difference in our approach is the use of contextual information (multi-frame features) as an input instead of single-frame features. We argue that the use of contextual information is critical for the application of DNNs in prediction of UBM states as it enables the learning of more invariant features in its deeper hidden layers (Yu et al., 2013). In our experiments, we observe similar results as in (Campbell, 2014) when using single-frame input features, while much better performance is observed when multi-frame input features are used. In this study, we focus on the application of unlabeled data in a discriminative fashion. Specifically, we explore the hard alignment between UBM and input features in the development data by assigning only one Gaussian mixture having the highest posterior probability to each input feature vector. The aligned mixture labels are later used to discriminatively predict the alignment between feature frames and the corresponding mixture with a DNN in the enrollment and verification data. The input features during training and prediction are concatenated multi-frame features instead of single-frame features. This is based on the fact that features used for Baum-Welch statistics estimation can be independent of those used for i-Vector extraction (Lei et al., 2014; Kenny et al., 2014). Finally, the prediction probability comes from the softmax layer of the DNN, which is then used as the main component for computing Baum-Welch statistics in i-Vector extraction. While this approach is capable of predicting the UBM states using a DNN with labels provided by the UBM itself, the acoustic space division still relies on the UBM. In order to investigate whether the DNN itself could achieve some form of feature space clustering, we employ a stacked restricted Boltzmann machines (RBM) as a way to achieve unsupervised clustering in our second approach. The RBM is a generative model where each node in the hidden layer is activated according to certain patterns of input features. In other words, the

24 activation patterns of higher layers in the RBM correspond to distinct patterns within the input features. Motivated by this, we use indices of the nodes with the highest normalized activation probability in the top layer of a stacked RBM as labels for training the DNN. The prediction part is accomplished via this DNN which is then used for Baum-Welch statistics estimation as in the first approach. The proposed two algorithms are evaluated on two different databases namely NIST SRE-2010 dataset and the UT-SCOPE Lombard (Hansen and Varadarajan, 2009) corpus.

6.2

Overview

A universal background model (UBM) trained as a Gaussian mixture model (GMM) is used in state-of-the-art i-Vector based speaker recognition systems for acoustic space division and prediction. The main purpose of such acoustic space clustering is to constrain the acoustic comparison in small regions where between-speaker differences are the main source of variability. In this study, we investigate two unsupervised discriminative approaches as an alternative to the UBM for feature space clustering and prediction. In our first approach, a deep neural network (DNN) is directly used to estimate the mixture-wise posterior probabilities of the UBM. In this approach, while the UBM is still used for acoustic space division, the prediction is performed using DNN. The motivation for using a DNN for such prediction is the ability to learn invariant higher level features discriminatively. In our second approach, a stacked restricted Boltzmann machine (RBM) is used instead of the UBM for both acoustic space clustering and prediction. In this approach, the clustering is performed based on the activations of each node in output layer of the stacked RBM. The stand alone system using our first approach (UBM-DNN) improves overall performance by +14% in minDCF10 when evaluated on NIST SRE 2010. While similar improvements were not observed when evaluated

25 on UT-SCOPE Lombard corpus, feature-level fusion of the proposed approach with baseline i-Vector system shows significant improvement over baseline results. Moreover, this feature space clustering is used in the extraction of Baum-Welch statistics which are critical for i-Vector extraction. This study takes an unsupervised discriminative approach to using this acoustic model for alignment of unseen speech data to one of the mixtures or clusters in feature space. A deep neural network (DNN) is used to replace the GMM as a robust discriminative classifier to predict one of the Gaussian mixtures. The resulting posterior probabilities improve the overall system performance by 14%. Another discriminative approach is proposed to bypass the GMM training, by unsupervised feature learning using stacked Restricted Boltzmann Machines (RBM) followed by conversion to a deep neural network by using the learned features as labels for discriminative training.

6.3

Discriminative Training

6.3.1

GMM-based DNN

This algorithm is based on a conventional i-Vector extraction framework where the BaumWelch statistics estimation method is replaced. In traditional Baum-Welch statistic estimation, a UBM is first trained from a development dataset and then this UBM is further used for mixture-wise posterior probability estimation. The estimated mixture-wise posterior probabilities become the main component of the Baum-Welch statistics. In our approach, we still train the UBM from the development dataset, but instead of estimating the mixture-wise posterior probability using the trained UBM, the Baum-Welch statistics are estimated by treating the Gaussian component having the highest posterior probabilities as class labels for DNN training and prediction as shown in Fig. 6.1. There are two major advantages with this approach: (i) the prediction of the mixture-wise posterior probability can be discriminatively achieved, and (ii) the features used for DNN training

26 and prediction can be independent from those used for training the UBM. Furthermore, we implement this approach using multiple frames of speech features as input for training the DNN to incorporate context and make the prediction aspect robust to minor variations in the cepstral domain. Each concatenated multi-frame feature vector is used as input to the DNN to learn the label extracted by noting the mixture to which the central frame in this vector belongs to.

Dev

Mixture Labels

Deep Neural Network Zero order

GMM-UBM

First order

Baum-Welch Statistics

i-Vector Extraction

Development Enrollment Verification

Figure 6.1. System block diagram of integration of GMM-based DNN with i-Vector system along with datasets used at each step.

6.3.2

RBM-based DNN

Here, we propose to train and use a Deep Belief Network (DBN) consisting of stacked Restricted Boltzmann Machine (RBM) for UBM-like acoustic space division. The division based labels can then be used to train a DNN with pre-initialized parameters rolled over from the DBN. A novel approach is used to cluster the development data which tries to replace the GMM-UBM based method. The approach is unsupervised, requiring no prior text content knowledge, since it is comprised of training multiple layers of an RBM stacked

27 on top of each other to make a DBN. A sigmoid activation function is used for all hidden layers. It is understood that each node in the hidden layer of an RBM tries to learn a distinct higher level feature from the data in previous layer. In this experiment it is desired to learn as many distinct classes of high level features as are present in the GMM-UBM. Each higher level feature can then be treated as a criterion to cluster input features. In a trained DBN, hidden output nodes have varying levels of activation for each input feature. Our approach aims to pick only one node to act as a label for subsequent discriminative training with DNN. To accomplish this for each input frame, such node is selected that expresses its highest activation in the scope of the whole utterance for that specific input, even if the absolute activation value is much lower than one. In other words, after training is complete, the selection of cluster or label is made based on the normalized activation values of each node in the top layer. Moreover, this leads to a more homogeneous distribution of the data into cluster bins and avoids the problem of having a few neurons firing most of the time. The sigmoid activations of each node c, (ac ), are normalized over all frames in each utterance ˆc . Gaussian normalization (Eq. 6.1) is used to suppress outliers and to cluster to produce a input frames based on each node’s peak activation value. This normalization was found to yield better results than using sparsity constraint for training RBM layers.

ˆc = a

ac − µa c σac

(6.1)

These labels are extracted for all development data to train a DNN in a discriminative, supervised fashion. The labels extracted from the DBN are used to train a deep neural network in a supervised way with pre-initialized or pre-trained weights borrowed from the DBN. A new output layer is added for the DNN which consists of Softmax function nodes, as is typically used in classification tasks. This layer replaces the top layer of DBN from which labels were extracted. The output of the softmax layer is then used as posterior probabilities

28

DBN

DNN

Label selection

Node activation normalization

Hidden layer

Discriminative training

Softmax layer Hidden layer Hidden layer

Hidden layer Input frames Figure 6.2. Top layer of a deep belief network (stacked RBM layers) is replaced by a softmax classification layer after extracting the index of the hidden node at its peak activation normalized over the entire training set.

for the i-vector statistics after training the DNN with mini-batch stochastic gradient descent. The projection of the DBN outputs to a label for DNN training is illustrated in Fig. 6.2.

6.3.3

DNN Architecture

The top layer of DNN consists of nodes employing the softmax function. This function enables the DNN to output class probabilities for each node which sum to 1.

29


(6.2)

where the target mixtures are expressed as Y , and the weight matrix and bias vector by W and b respectively. Cross-entropy minimization is employed as the objective loss function, which maximizes target mixture membership probabilities for the training data.

`(θ = {W, b}, D) = −

|D| X

log(P (Y = yi |xi , W, b)).

(6.3)

i=0

Mini-batch Stochastic Gradient Descent (Zhang, 2004) is then used to train the DNN with backpropagation.

6.3.4

i-Vector Extraction

The i-Vector features are extracted via factor analysis (Dehak et al., 2011b). The i-Vector model is represented by: M = m + T ω,

(6.4)

where T is the total variability matrix, ω is the i-Vector, m is the UBM mean super-vector, and M is the super-vector derived from acoustic features. The process converts frame variable length spectral features matrix into a feature vector with fixed dimensions for each speech utterance. The output layer nodes of the final trained DNN provide the posterior probabilities for each frame Xt which are used to calculate the Baum-Welch statistics after extraction of the total variability matrix. The zeroth (Ns ) and centralized first order (Fs ) Baum-Welch statistics are: NsC=c =

X t

P (c|Xt , θU BM ),

(6.5)

30

FsC=c =

X

P (c|Xt , θU BM )(Xt − µc ).

(6.6)

t

These two equations above usually require a UBM to extract these statistics. In our case, the parameters θU BM for this UBM are artificially estimated based on the posterior probabilities of DNN for each frame Xt . The mean µc , covariance Σc , and weights ωc for each mixture c are estimated by collecting all the frames assigned to a particular class c by the DNN i.e. the class corresponding to the output node with maximum posterior probability for each frame. The generation of new GMM-UBM parameters was found to be fruitful since it aligns the system with the new means and variances of DNN-based statistical model.

θU BM = {ωc , µc , Σc }

(6.7)

Finally, using the posteriors, the i-Vector is extracted as usual using: ws∗ = (T 0 Ns Σ−1 T + I)−1 T Σ−1 Fs 6.4

(6.8)

Evaluation

6.4.1

Experimental Setup

The DNN based Baum-Welch statistics estimation method is evaluated on the core telephone speech condition of NIST SRE 2010. The development data is composed of telephone channel audio from NIST SRE 2004, 2005, 2006, and Switchboard Phase II corpora totalling about 300 hours. Mel-frequency cepstral features (MFCC) with velocity and acceleration coefficients (36-dimensional) are used for all experiments, although with this approach different features can be used for i-Vector extraction and posteriors estimation. i-Vectors with 400 dimensions are used followed by PLDA scoring (Liu and Hansen, 2014). A GMM-UBM is first trained for 1024 mixtures using the standard Expectation Maximization algorithm. After the UBM is trained on development data, IDs of the Gaussians

31 in the mixture are noted for each frame of development speech data. These IDs are treated as labels for their respective frames. These labels which are obtained in an unsupervised manner are used to train the neural network in a regular semi-supervised discriminative fashion. Both single-frame and multi-frame concatenated MFCC features are considered as potential inputs to the DNN. No unsupervised pre-training is necessary for single-frame vectors, but after concatenation of 7 frames, the 252-dimensional input vector requires generative pre-training, for optimal training of the DNN. Pre-training of DNN using stacked denoising autoencoders (SDA) is used to solve the added complexity of high-dimensional data. The DNN employed is composed of 3 hidden layers with 1000 hidden units in each hidden layer. The entire network consists of 3.3 million weight parameters. The DNN is pretrained with SDA using the development dataset. Pre-training through stacked Restricted Boltzmann Machines (DBN) was also considered but took much longer to reach similar minima. The proposed method is also evaluated using a 2048-mixture UBM and a DNN with 2048 output nodes. The RBM-based DBN-DNN approach employs a Gaussian-Bernoulli RBM (GRBM) as the first visible-hidden layer, and regular Bernoulli-Bernoulli RBM in the following three layers stacked on top of the GRBM. Trained using the fast Contrastive Divergence algorithm in a greedy layerwise manner (Hinton et al., 2006, 2012) to form a deep belief network (DBN), the frame labels are extracted by normalizing the activations of the top layer over all frames in each utterance. This ensures a near-homogeneous distribution across all clustering bins. The trained DBN is then converted to a DNN by replacing the last RBM hidden layer by a softmax classification layer on top. The top RBM layer has the same number of nodes as the top DNN softmax layer to allow training for the same number of classes as extracted by DBN.

32

Table 6.1. Results on SRE-2010 Male Core Telephone Data (1024 mixtures) System

EER(%)

minDCF10

Baseline

2.36

0.399

DNN (Singleframe)

2.56

0.371

DNN (Multiframe)

2.29

0.343

RBM-DNN (Singleframe)

3.25

0.541

RBM-DNN (Multiframe)

3.61

0.537

The best performing system based on the UBM-DNN approach was tested on the corpus UT-SCOPE Lombard to explore the impact of Lombard Effect1 on this method which is known to severely impact speaker identification systems (Hansen and Varadarajan, 2009). Inset speaker identification is performed on a set of 29 male and female native English speakers. The corpus includes clean speech for neutral conditions and speech under 3 different Lombard flavors (Large crowd, Highway, Pink noise) with 3 different levels of background noise. Our development and enrollment sets consist of neutral speech and speech under Lombard Effect with only 2 of the 3 noise-levels. The test or verification set contains neutral and speech under the open third background noise-level. It should be emphasized that all Lombard data used is noise-free, since the subjects were exposed to background noise via open-air headphones. For a relatively smaller corpus, only 128 mixtures are used for feature space clustering. 80-dimensional i-Vectors are employed followed by PLDA scoring. Similarly, a smaller 2-layer DNN is used with 700 nodes in each hidden layer. 1

Lombard Effect occurs when subjects alter their speech production in the presence of high levels of background noise.

33

Table 6.2. Results on SRE-2010 Male Core Telephone Data (2048 mixtures) System

EER(%)

minDCF10

Baseline

2.47

0.385

DNN (Multiframe)

2.18

0.351

Table 6.3. Results on UTSCOPE Lombard Corpus System

6.4.2

EER(%)

minDCF10

Baseline (GMM-UBM)

5.15

0.610

DNN (Multiframe)

5.41

0.636

Fusion

4.88

0.551

Results

The Baum-Welch statistics for the i-vector extraction framework are extracted from the trained DNN for development, enrollment and verification datasets. Table 6.1 shows that single-frame features result in slightly better performance in minDCF10 , while the multiframe features were able to beat the baseline system on the NIST SRE-2010 telephone speech core condition. A +14% relative improvement is observed for the system employing multi-frame DNN based statistics compared to the GMM-UBM baseline (1024 mixtures) on the NIST SRE2010 minDCF cost function (see Table 6.1). The results are consistent with increased number of UBM mixtures (2048) where the DNN based method shows +9% improvement compared to the new baseline in terms of minDCF10 . The EER also shows an improvement of +3% and +12% in both cases respectively.

34 The experiment employing DBN-DNN based labels also results in highly competitive error-rates on the telephone speech data. The apparent absence of improvement in the case of DBN training with multiframe input can be attributed to the smaller amount of data used for training them (30%) due to computational constraints. Although the RBM-DNN does not offer an improvement over the baseline, it does offer a novel system for use in system fusion, and a new promising direction for future research. The UT-SCOPE Lombard corpus offered a much smaller development dataset (about 13 hours) compared to the 300 hours data on telephone speech for SRE. The standalone system still performs at a competitive level, and fusion with the baseline system results in an average performance boost of +7.5% in EER and minDCF10 (Table 6.3). Simple feature-level fusion was applied by concatenation of the i-Vectors of both systems. Two new systems are proposed to extract posterior probabilities by replacing a UBM with DNN with unlabeled development data. Discriminative training is explored to build the DNN. The system using GMM-based mixtures as labels outperforms the i-Vector based system by 12%. The second system does not depend on GMM-UBM training to extract labels, rather it learns them in the form of high-level features in unsupervised manner using layered RBMs, which is subsequently discriminatively trained by a DNN as a classifier, resulting in competitive performance. Basic fusion results on speech under Lombard Effect show the benefit of having a DNN-based statistics estimation component in the system.

CHAPTER 7 CONCLUSIONS 7.1

Summary

Deep neural networks have been shown to outperform traditional classifiers in distinguishing between neutral speech and speech under different kinds of Lombard Effect under various background noise-types and noise-levels. The resulting accuracies are 95.7% for 2-way, 69.1% for 4-way, and 60.0% for 10-way classification using cepstral features. Both DNN assisted training methods have proven to be effective in combating Lombard speech in SID. The proposed probabilistically weighted system uses validation data classification as a priori information and this results in a more robust training of SID system. The distributed system is highly efficient for all kinds of speech under Lombard Effect as well as for neutral speech, resulting in an overall improvement of +28.4% on the UT-SCOPE database. Deep neural networks have also been shown to be productive in generating high-level features from cepstral coefficients. This effort confirms the impact of these feature extractors in distinguishing between different kinds of Lombard Effect, and subsequently contribute to effective SID. In this study, we have investigated the application of DNN for unsupervised Baum-Welch statistics estimation. Two systems, UBM-DNN and DBN-DNN, were proposed where the traditional acoustic space division and prediction for i-Vector extraction is accomplished with deep learning techniques. The experimental results for UBM-DNN show a relative improvement of +14% in terms of minDCF10 when evaluated on the telephone condition of the NIST SRE-2010 speaker recognition task. 35

36 However, similar improvement is not observed when evaluated on the UT-SCOPE Lombard corpus. We attribute this to the relatively small amount of data (about 13 hours) for training the DNN compared to the NIST SRE database with around 300 hours of speech. While the standalone UBM-DNN system does not show improvement over baseline system, the feature-level fusion of UBM-DNN system with baseline i-Vector system still indicates very effective improvement. On the other hand, the RBM-based DBN-DNN approach did not show improved results on SRE although it is in the same order of magnitude.

7.2

Future Work

Future work to improve SID systems for speech affected by Lombard Effect can be achieved by appending the proposed system to background noise reduction algorithms which can result in improved robustness for other corpora. The additional validation based information can also be used as metadata for calibration (Ferrer et al., 2012). Future research can focus on using this system to counter both environment and channel mismatch for speaker recognition. Although the proposed UBM-DNN system shows some improvement over a baseline system, the relative improvement is much less compared to the study in (Lei et al., 2014) where tied-state triphones along with DNN were used. The study in (Kenny et al., 2014) indicates that such an approach is sensitive to the amount of data used for training the DNN; where around 1100 hours of data are required for DNN training in order to obtain the same level of relative improvement. Because of the limited data resources, around 300 hours of data from previous NIST SRE telephone conditions were used in our UBM-DNN system. Therefore, the natural extension of current study will be the evaluation of UBMDNN method with larger amounts of training data. Adding context to a central frame clearly results in better performance with DNNs. Extended context of more than three adjacent frames can be explored. Furthermore, better methods to extract the labels from a trained DBN and better training algorithms can be

37 explored. Use of Convolutional Neural Networks (CNN) in place of DNN helps model the acoustic feature space with robustness to minor shifts in features both in frequency and temporal domains.

REFERENCES Bergstra, J., O. Breuleux, F. Bastien, P. Lamblin, R. Pascanu, G. Desjardins, J. Turian, D. Warde-Farley, and Y. Bengio (2010, June). Theano: a CPU and GPU math expression compiler. In Python for Scientific Computing Conference (SciPy). Boril, H. and J. H. L. Hansen (2010, Aug). Unsupervised equalization of Lombard effect for speech recognition in noisy adverse environments. IEEE Transactions on Audio, Speech, and Language Processing 18 (6), 1379–1393. Campbell, W. (2014). Using deep belief networks for vector-based speaker recognition. In ISCA Interspeech-2014. Chi, S. and Y. Oh (1996, Oct). Lombard effect compensation and noise suppression for noisy Lombard speech recognition. In ICSLP 1996, Volume 4, pp. 2013–2016. Dehak, N., P. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet (2011a). Front-end factor analysis for speaker verification. IEEE Transactions on Audio, Speech, and Language Processing 19 (4), 788–798. Dehak, N., P. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet (2011b). Front-end factor analysis for speaker verification. IEEE Transactions on Audio, Speech, and Language Processing 19 (4), 788–798. Ferrer, L., L. Burget, O. Plchot, and N. Scheffer (2012). A unified approach for audio characterization and its application to speaker recognition. In Proc. of Odyssey Workshop. Hansen, J. H. L. (1988, July). Analysis and compensation of stressed and noisy speech with application to robust automatic recognition. Ph. D. thesis, Georgia Inst. Tech., Atlanta, GA. Hansen, J. H. L. (1994, Oct.). Morphological constrained feature enhancement with adaptive cepstral compensation (mce-acc) for speech recognition in noise and Lombard effect. IEEE Transactions on Speech and Audio Processing 2 (4), 598–614. Hansen, J. H. L. and V. Varadarajan (2009, Feb). Analysis and compensation of Lombard speech across noise type and levels with application to in-set/out-of-set speaker recognition. IEEE Transactions on Audio, Speech, and Language Processing 17 (2), 366–378. Hinton, G., L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury (2012, Nov). Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine 29 (6), 82–97. 38

39 Hinton, G., S. Osindero, and Y. Teh (2006, July). A fast learning algorithm for deep belief nets. Neural Computation 18 (7), 1527–1554. Ikeno, A., V. Varadarajan, S. Patil, and J. H. L. Hansen (2007, March). UT-SCOPE: Speech under Lombard effect and cognitive stress. In Aerospace Conference, 2007 IEEE, pp. 1–7. Kenny, P. (2010, June). Bayesian speaker verification with heavy-tailed priors. In Odyssey. Kenny, P., V. Gupta, T. Stafylakis, P. Ouellet, and J. Alam (2014). Deep neural networks for extracting baum-welch statistics for speaker recognition. In ISCA Odyssey Speaker and Language Recognition Workshop. Lei, Y., L. Burget, and N. Scheffer (2013). A noise robust i-vector extractor using Vector Taylor Series for speaker recognition. In IEEE ICASSP, pp. 6788–6791. Lei, Y., N. Scheffer, L. Ferrer, and M. McLaren (2014). A novel scheme for speaker recognition using a phonetically-aware deep neural network. IEEE ICASSP, Florence, Italy. Liu, G. and J. H. L. Hansen (2014, Dec). An investigation into back-end advancements for speaker recognition in multi-session and noisy enrollment scenarios. IEEE Transactions on Audio, Speech, and Language Processing 22 (12), 1978–1992. Liu, G., T. Hasan, H. Boril, and J. H. L. Hansen (2013, May). An investigation on back-end for speaker recognition in multi-session enrollment. In ICASSP 2013, pp. 7755–7759. Liu, G., S. Sadjadi, T. Hasan, J. Suh, C. Zhang, M. Mehrabani, H. Boril, A. Sangwan, and J. H. L. Hansen (2011, December). UTD-CRSS systems for NIST Language Recognition Evaluation 2011. NIST 2011 LRE Workshop. Liu, G., C. Zhang, and J. H. L. Hansen (2012, June). A linguistic data acquisition front-end for language recognition evaluation. Odyssey, 224–228. Martınez, D., O. Plchot, L. Burget, O. Glembek, and P. Matejka (2011). Language recognition in ivectors space. Interspeech, 861–864. Mohamed, A., G. Dahl, and G. Hinton (2012, Jan). Acoustic modeling using deep belief networks. IEEE Transactions on Audio, Speech, and Language Processing 20 (1), 14–22. Stuhlsatz, A., C. Meyer, F. Eyben, T. ZieIke, G. Meier, and B. Schuller (2011, May). Deep neural networks for acoustic emotion recognition: Raising the benchmarks. In ICASSP 2011, pp. 5688–5691. Varadarajan, V. and J. H. L. Hansen (2006). Analysis of Lombard effect under different types and levels of noise with application to in-set speaker ID systems. In INTERSPEECH.

40 Yu, C., G. Liu, S. Hahm, and J. H. L. Hansen (2014). Uncertainty propagation in front end factor analysis for noise robust speaker recognition. In IEEE ICASSP, Florence, Italy. Yu, C., G. Liu, and J. H. L. Hansen (2014). Acoustic feature transformation using ubm-based lda for speaker recognition. ISCA Interspeech, Singapore. Yu, D., M. Seltzer, J. Li, J.-T. Huang, and F. Seide (2013, May). Feature learning in deep neural networks - studies on speech recognition. In Inter. Conf. on Learning Representations. Zhang, Q., G. Liu, and J. H. L. Hansen (2014, June). Robust language recognition based on hybrid fusion. In Odyssey, pp. 152–157. Zhang, T. (2004). Solving large scale linear prediction problems using stochastic gradient descent algorithms. In 21st Inter. Conf. on Machine Learning. ACM.

VITA Muhammad Muneeb Saleem is pursuing a Master of Science degree in Electrical Engineering with a focus on Signal Processing & Communications at the University of Texas at Dallas (UTD). He earned his Bachelor of Science in Electronics Engineering degree from Ghulam Ishaq Khan Institute (GIKI) in 2011 from Pakistan. He also has work experience of one year each in the telecommunication and oil & gas industries. Muneeb is a Graduate Student Member of the IEEE. He works primarily in the area of Machine Learning for Speaker Recognition in the Center for Robust Speech Systems (CRSS) with advisor Professor John H.L. Hansen, IEEE Fellow. Research interests include innovative techniques in Deep Learning, Machine Learning, High Performance Computing using GPU, Emotion and Stress Classification in Speech, and Speaker Recognition.