Remote vibrometry vehicle classification

Remote vibrometry vehicle classification

a*

a

Ashley Smith , Steve Goleyb, Karmon Vongsyc, Arnab Shawa, and Matthew Dierkingc

Wright State University; bEtegent Technologies, Ltd.; cAir Force Research Laboratory/Sensors Directorate ABSTRACT

In vehicle target classification, contact sensors have frequently been used to collect data to simulate laser vibrometry data. Accelerometer data has been used in numerous literature to test and train classifiers instead of laser vibrometry data [1] [2] . Understanding the key similarities and differences between accelerometer and laser vibrometry data is essential to keep progressing aided vehicle recognition systems. This paper investigates the contrast of accelerometer and laser vibrometer data on classification performance. Research was performed using the end-to-end process previously published by the authors to understand the effects of different types of data on the classification results. The end-to-end process includes preprocessing the data, extracting features from various signal processing literature, using feature selection to determine the most relevant features used in the process, and finally classifying and identifying the vehicles. Three data sets were analyzed, including one collection on military vehicles and two recent collections on civilian vehicles. Experiments demonstrated include: (1) training the classifiers using accelerometer data and testing on laser vibrometer data, (2) combining the data and classifying the vehicle, and (3) different repetitions of these tests with different vehicle states such as idle or revving and varying stationary revolutions per minute (rpm). Keywords: laser vibrometry, accelerometers, aided target recognition, vehicle classification

1. INTRODUCTION Accelerometer data has frequently been utilized in place of laser vibrometer data. Accelerometer data can be collected in much higher volumes than laser vibrometer data. Dozens of accelerometers can be easily placed across a vehicle to collect data simultaneous while the laser vibrometer must be operated singularly multiple times across a vehicle. Understanding the effects contact and non-contact sensors have on classification performance is important toward improving, benefiting, and conducting future data collections. The goal of this work is to determine how accelerometer data, laser vibrometer data, and a combination of the two affect the classification performance for different vehicles. Previously, several different approaches have been taken in identifying vehicles, or information about the vehicles from the vibrations. Four vehicles, collected with directional microphones, in a small set of traffic patterns and with different speeds were studied and classified using linear prediction coefficients (LPC) and time delay neural networks (TDNN) [3]. The vehicles were identified with almost 100% accuracy. Wireless accelerometers and magnetometers were used to classify vehicles based on their calculated number of axles [4]. Axle counts, arrival and departure times, speed estimates, and the number of vehicles that passed over the sensors were recorded for classification. Classification rates around 99% were achieved even during high traffic conditions. Another paper focused on generating aided target recognition (AiTR) algorithms to classify three vehicles and a power transformer with accelerometer data [5]. Stevens and his coauthors utilized eleven features from similar literature and selected the best features with ReliefF and classified using four different classifiers. An average classification rate between 74% and 98% was seen for the vehicles [5]. The authors published a second paper using a different set of data and explored a new process using fewer features [6]. The authors focused on determining trait similarities between the vehicles using k-nearest neighbor classification with different k values. Kangas and his coauthors examined manifold learning for dimension reduction of accelerometer data as a novel approach to vehicle target recognition [1]. Principle component analysis and diffusion maps were utilized to reduce data dimensionality for three vehicles, the same as in [6], and classified using four different classifiers. Classification rates were greater than 90% with both dimension reduction methods [1]. The authors of [7] used peak detection methods and compared the peaks of the accelerometer data across the vehicles to determine if there were similarities. Their work determined that most of the vehicle’s energy is lower in frequency and is relatively stable across the vehicle [7]. Sigmund and coauthors published on distinguishing between gas and diesel piston vehicles [2]. They looked at the cepstrums of 16 different vehicles and classified them as gas or diesel with varying levels of success for each vehicle and throttle condition. Another paper compared classifying eight different vehicles with individual classifiers and with a hierarchy of

Ground/Air Multisensor Interoperability, Integration, and Networking for Persistent ISR VI edited by Tien Pham, Michael A. Kolodny, Proc. of SPIE Vol. 9464, 94640S © 2015 SPIE · CCC code: 0277-786X/15/$18 · doi: 10.1117/12.2180633 Proc. of SPIE Vol. 9464 94640S-1 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 05/22/2015 Terms of Use: http://spiedl.org/terms

three levels of classification [8]. The hierarchy consisted of engine type, engine configuration, and serial number. Performance improvement was seen when using a hierarchy over an individual classifier. Lastly, several of the authors here developed an end-to-end process based on the work in [1], [5], and [6]. Accelerometer data from three vehicles were classified using the end-to-end process that produced classification rates greater than 70% [9]. All of these papers approached the task of vehicle classification differently. They focused on different operating conditions, such as sensor type, traffic patterns, and engine conditions, to accomplish classification. The methods performed in this paper are a combination of those done in [1], [5], and [6]. Additionally, this work is an extension of the work done in [9] and [10]. This work adds classification of remote vibration data as previously contact sensors were used to represent the data.

2. COMPARING LASER VIBROMETERS & ACCELEROMETERS 2.1. Accelerometers Accelerometers are contact sensors that measure the vibrations of a target in terms of acceleration at the sensor location. Accelerometer data has historically been collected instead of laser vibrometer data for two reasons. The first reason accelerometer data is collected in large amounts of accelerometer data can be collected at a time, unlike laser vibrometer data. Accelerometer data is also expected to outperform laser vibrometer data; accelerometers are connected directly to the vibrating surfaces. For this research, single and triaxial AC response accelerometers were used to collect data. Piezoelectric components, in this case quartz crystals, are used to measure acceleration [11]. The piezoelectric components produce a voltage output when the accelerometer experiences a change in load due to the vibrations [11]. These voltage outputs are converted to acceleration using each sensors unique parameters. Accelerometers are attached to vehicles with a variety of different methods. Magnets, beeswax, and super glue or epoxy have all been used to attach accelerometers. Each of the attachments has advantages and disadvantages. For example, beeswax is a reusable attachment but it cannot be used for moving tests. For this work, super glue and epoxy were used to attach the accelerometers to the vehicles. 2.2. Laser vibrometers Laser vibrometers are non-contact sensors that allow vibrations of a target to be measured in terms of velocity or displacement using the Doppler Effect. The velocity and displacement are measured as frequency or phase shifts that occur from light scattered off of the target(s). For this research a continuous wave single point laser vibrometer was used. Continuous wave laser vibrometers are frequently used in commercial and industrial environments. These laser vibrometers measure the frequency shifts caused by the vibrations with an interferometer. An interferometer splits a laser’s beam into a reference and measurement beam with a series of mirrors and beam splitters [12]. Figure 1 shows the optical component layout of a typical laser vibrometer. The reference beam goes through a Bragg cell, an acousto-optic modulator for frequency shifting, and onto the photo detector [12]. This shift in the reference beam allows for magnitude and direction of the vibrations to be measured. The measurement beam is passed to the target and is reflected back to the photo detector. The measured Doppler shift is proportional to the velocity over wavelength, v/λ. An interference signal on the photo detector is created from the measurement and shifted reference beam. The velocity of the target’s vibrations is directly related to the interference signal over time [12]. Beam Splitter 1

Beam Splitter 2 Target

Laser

Mirror

Beam Splitter 3

Photo Detector

Figure 1. Optical layout of a continuous wave laser vibrometer.

Proc. of SPIE Vol. 9464 94640S-2 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 05/22/2015 Terms of Use: http://spiedl.org/terms

2.3. Siggnal Comparisson Accelerometters and laser vibrometers v botth collect vibraation data. Though the methods to collect thhe vibration datta differ, a relationshipp between the two t sensors exxists. Velocity and a acceleratioon are related by b integration; acceleration inntegrates to velocity. Integrating I thee accelerometeer data inherenntly reduces thhe noise in thee signal. The data d is scaled to show vibration in µm/s. Figuree 2 shows thhe overlap the power specctral density (PSD) of sim multaneously collected c accelerometeer and laser vibrometer v datta from the same s location on a 2010 Honda H Odysseey. Though thhere are discrepanciess in the frequeency magnituddes, similar peeaks can be seeen in the dataa. Thus colleccting large amounts of accelerometeer data in place of laser vibrrometer data iss plausible givven the relationnship between these sensors. On the other hand, a remote and non-invasive lasser vibrometerr allows for colllecting data inn situations whhere acceleromeeter data cannot be obtained. o Incorrporating the weighting of the accelerom meters and thhe atmosphericc effects on the t laser vibrometer could be used to o better match the two types of o data. Comparison af LVS and Accele erometer PSDs -LVS - Accelerometer

irq

i1 V

`1 I I

-4

0

50

loo

150

200

250

300

350

400

450

500

Frequency (Hz)

Figure 2. Comparison C of spectral s content of o laser vibromeeter and acceleroometer.

3.. THE END D-TO-END PROCESS P The end-to-eend process used in this expeeriment is based on the worrk done in [1] and [5]. The process p has prreviously been explainned in depth in n [9] and [10]. The process consists of loading raw dataa, preprocessinng the data, exxtracting features, seleecting the best features, classifying, and anaalyzing the ressults, shown in Figure 3. The process is genneralized in order to teest multiple com mbinations of features f and claassifiers to devvelop the best performance. p

Figure 3. Simple S block diaagram of the end-to-end process.

The vibrationn data from thee laser vibromeeters and accellerometers are segmented intoo one second samples. s Preproocessing is completedd differently deepending on thhe sensor. The accelerometerr data is integrated to reducee noise and connvert the acceleration data to velocitty data. A highh pass filter, with w a cutoff freequency of 5 Hz, H is applied to t the laser vibbrometer data to reduce low frequeency noise; thee acquistion syystem does thiis for acceleroometer data soo it is repeatedd during [10] preprocessingg. Data from both b sensors is zero centered through mean subtraction annd normalized by b Euclidean norm n .


Five windows are created from each sample and a power spectral density (PSD) is taken from each window. Features are calculated and averaged in each window. Feature selection is performed to reduce dimensionality. For this research, all features were selected thereby feature selection is not explained in detail. Previous feature selection methods can be found in [9] and [10]. Data is separated into testing and training segments based on sensor location to avoid overfitting. Classification is performed with these sets and analyzed with a human-in-the-loop method [10]. The Waikato Environment for Knowledge Analysis (WEKA) and MATLAB are used in this end-to-end process. MATLAB is used to preprocess and extract features from the data samples. WEKA is used for feature selection and classification. 3.1. Feature extraction Features are extracted to represent relevant information from the target data in reduced dimensionality [9]. Maximizing the amount of information retained and minimizing the number of dimensions needed to retain the information is the goal of feature extraction. The features used in this research were selected by the authors of [5] from speech processing, seismology, and structural analysis research. Eleven features were selected from these areas to test with vibration data. Complexity, root mean square (RMS), linear prediction coefficients (LPC), dominant frequencies, flux, Mel-frequency cepstral coefficients (MFCC), peak counting, rolloff, spectral ratios, and spectral centroid were utilized in the original experiment [5]. Ten of the features were utilized for this experiment; LPC was the excluded feature. A short explanation of each feature and the equations used are shown in Table 1. Further explanations on each feature can be found in [9] and [10]. Table 1. Features used in the end-to-end process.

Feature

Equation

Zero Crossings: the number of times a zero centered signal crosses the y-axis; often found in speech processing [13, 14, 15]

1 ( 1 ( 0

= ,

Complexity: the comparison of seismic energy in different parts of seismic waves

( )

=

[16, 17]

Root Mean Square (RMS): the average intensity measure; often used in speech processing

∀ Spectral Flux: normalized rate of change between signal frames; often used in speech and music processing [14, 18] Mel-Frequency Cepstral Coefficients (MFCC): the condensed representation of the frequency spectrum; frequently used in speech processing [18, 19] (10 coefficients)

1

|

= , , ℎ

=

( )=

|

∀

|

∀

|

|

|

|

|

|

> |

log |


|

|

|

> > |

| (

|

|

−

|

|

| (

|

( )|

( ) 1

< 0) > 0)

( )

/

=

Dominant Frequency: the N frequencies with the highest magnitudes; often used in speech processing [9] (N=3)

> 0) ∨ ( < 0) ∨ ( ℎ

|

(

| ( )|

( − 1) ( )|

|

| |

| > |

) |

|

>

|

) |

)

Feature

Equation

Number of Peaks: the number spectral peaks higher than a specified threshold [20]

=

0 Spectral Rolloff: the frequency were P percentage of the total energy of the spectrum is contained; often used in music processing [14, 21] Spectral Centroid: the balancing point of the spectrum; often used in speech processing and seismology [14, 21]

|

|

1

,

| (

=

|

)

ℎ

=

=

|

|

=

Spectral Ratio: the comparison of energy in different frequency bands; often used in seismology [17] (0-100 Hz; 0-500 Hz; & 01000 Hz)

>

∗

|

|

|

|

|

|

/

/

|

|

/

|

|

3.2. Classification and performance metrics Extracted features are used to determine the separability of targets with classification. Four classifiers are utilized to determine the performance variation between accelerometer data and laser vibrometer data. Confusion matrices and kappa statistics are used to compare the performances. The confusion matrices allow the correct and incorrect classifications of each target to be analyzed and displayed easily. The kappa statistic is a measure of reliability as it expresses the level of true agreement made by the classifier. The kappa statistic range is between 0 and 1 with 1 considered as perfect agreement. For this research, a kappa statistic greater than 0.8 is considered successful. Training and testing sets are created to prevent bias in the classification. Some accelerometer and laser vibrometer locations are only used for training and others are only used for testing. The classifiers are trained using the data from the front and rear of the vehicle and tested using the data from the middle of the vehicle. The training and testing data are partitioned to have 66.67% of the vibration data in the training set and 33.33% of the data in the testing set. The four classifiers used included Naïve Bayes, decision trees, support vector machines (SVM) and k-nearest neighbors (KNN). These classifiers were selected from those used in [1] and [5] to provide a diverse range of classifiers. Naïve Bayes is a probabilistic classifier referred to as NaiveBayes in WEKA. The probability that a target belongs to each of classes is calculated and the class with the highest probability is selected [22]. KNN is a voting algorithm referred to as IBk in WEKA. The Euclidean distance between each target and all of the training data is calculated. The classes of the k nearest neighbors are examined and the majority class is selected. Decision tree classifiers are multi-level decision makers. The specific decision tree classifier used here is known as C4.5 and is referred to as J48 in WEKA. C4.5 compares information gains for all threshold splits of each feature to determine when and where to branch or stop [23]. Lastly, SVM is a linear classifier referred to as SMO in WEKA. SVM maps the features to a higher dimensionality and divides classes by determining a hyperplane that maximizes the distance between classes [24].

4. DATA Three different data collections were examined in this experiment. Three vehicles from each data collection, for a total of nine vehicles, are utilized for classification. Two of the data sets include laser vibrometer and accelerometer data; only accelerometer data was available for the one of the collections. These data sets were all provided by the Sensors Directorate of the Air Force Research Laboratory (AFRL). Each collection is explained in more detail in the next three subsections. 4.1. Mountain-top/Airborne Long-Range Test & Evaluation (MALTESE) MALTESE is a data collection containing three different military vehicles over the span of a few days. Two of the engines are 12 cylinder piston engines; the remaining vehicle contains a turbine engine. The vehicles are referred to as AA, BA, and CA. MALTESE took place in the desert with each vehicle being interrogated with a series of contact accelerometers and a long range laser vibrometer; however only the accelerometer data was provided for this collection. The data was collected on each of the vehicles while stationary and moving. Currently, only the stationary data is being


used as the moving data brings in additional information in the vibration signatures that are not yet completely understood. The stationary data was collected on idle engine vibrations, engine revving, and different constant engine revolutions per minute (rpm). The data was collected at 10 kHz using single axis and triaxial accelerometers that were positioned across the vehicles. Several dozen trials were collected in 30 second intervals and the best quality data was selected. The best quality data was considered to be data without noise spikes, saturation, signal dropouts, and missing values. Each vehicle provided approximately 3000 one second samples of idle and non-idle stationary data, for a total of 6000 samples per vehicle. A total of 18000 samples of data were used during experimentation. 4.2. Laser Vibrometer & Accelerometer Simultaneous Collection (LAVA-SC) LAVA-SC contains data from multiple sensors on four civilian vehicles. Contact triaxial accelerometers and a single point laser vibrometer were used to collect data from: 1) 2004 Chevrolet Cavalier; 2) 2003 Dodge Ram 2500; and 3, 4) 2010 Honda Odyssey EX-L (different serial numbers), all shown in Figure 4. Tachometer data was also collected to record the true engine rpm. Only three vehicles were exploited from the LAVA-SC dataset: D (Dodge), H (Honda), and C (Chevy). All three vehicles have piston engines; the Dodge and Honda have 6 cylinder engines while the Chevrolet only has a 4 cylinder engine. Each vehicle was fitted with 30 accelerometers and interrogated by a Polytec OFV 503 sensor head and Polytec OFV-5000 acquistion box. Most of the accelerometers were placed on the driver side of the vehicle to collect accelerometer and laser vibrometer data in the same location simultaneously. The stationary data collected for each of the vehicles include idle, rpm sweeping, and constant engine rpm. Idle and constant rpm data were used for this experiment. Between 70 and 120 trials of data were collected in 15 second intervals for each vehicle with a sampling rate of 10 kHz. Each vehicle had about 3000 one second samples for idle and non-idle stationary accelerometer data and 250 samples of idle and non-idle stationary laser vibrometer data. A total of approximately 19,500 one second samples of data were used in this analysis.

2010 Honda Odyssey EX-L

2003 Dodge Ram 2500

111

2004 Chevrolet Cavalier

2010 Honda Odyssey EX-L

Figure 4. Vehicles from LAVA-SC collection.

4.3. Summer 2014 Data Collection The Summer 2014 Data Collection is a thirteen vehicle multi-sensor collection. Accelerometers, a laser vibrometer and tachometers were used to collect data simultaneously. Only three vehicles, shown in Figure 5, were selected to use for this experiment from the summer collection: 1) 2005 International 8600 (I); 2) 2009 Pontiac G6 (P); and 3) 1999 Toyota Camry LE (T). Each of these vehicles has a piston engine; the International and Pontiac have 6 cylinder engines and the Toyota has a 4 cylinder engine. Five triaxial accelerometers and a Polytec 505 sensor head/ Polytec 5000 acquisition system were used to collect data. Tachometers were also used to record the true rpm of the engine and sometimes of the fan. Idle, sweeping rpm, and constant rpm data was collected for these vehicles with a sampling rate of 10.28 kHz. Only the idle and constant rpm data were used for this experiment. Approximately 40-60 30 second trials of data were


collected for each vehicle and engine condition. For this analysis, approximately 1500 one second samples of accelerometer data were used for idle and constant rpm for each vehicle. Approximately 500 samples of laser vibrometer data were used for each vehicle. A total of approximately 6.000 one second samples were used from this collection for analysis. .

2005 International 8600

2009 Pontiac G6

Si--1:#c

1999 Toyota Camry LE Figure 5. Vehicles used from the Summer 2014 data collection.

5. RESULTS MATLAB and WEKA were leveraged to classify nine vehicles for this research. MATLAB was used to load and organize the raw data, preprocess, and extract features. WEKA was employed for classification. The predefined classification parameters in WEKA are used to allow for repeatability of the experiments and to prevent over-tuning. Accelerometer data and laser vibrometer data were both analyzed to determine performance differences. Idle, non-idle stationary, and a combination of the data were utilized to classify the nine vehicles using accelerometer data and six vehicles using laser vibrometer data. Accelerometer data was tested first with data from all nine vehicles. Approximately 1500-4500 one second samples of accelerometer data were used for each vehicle and each engine condition. Two-thirds of the data was used to train the classifiers and the final third was used for testing. For the idle and non-idle stationary classifications, each classifier was trained with approximately 20000 samples and tested with 10000 samples. The classification on idle and non-idle stationary data was trained using approximately 40000 samples and tested using 20000 samples. 5.1. Accelerometer Classification Results The average classification performance of each of the accelerometer tests are shown in Table 2. The best classification performance seen for the accelerometer data was approximately 68%. The classification rates were relatively consistent regardless of the engine state. None of the classifiers performed with high confidence; all of the kappa statistics were lower than 0.8. Two of the confusion matrices are shown in Table 3. Vehicle BA is frequently identified as AA. Vehicles P & T are frequently misidentified as Vehicle I nearly every time. Vehicle H was also frequently identified as Vehicle C. The poor classification performance of these four vehicles (BA, H, P, and T) drastically reduce the overall classification performance. Some of the statistics of the features were examined in JMP, statistical analysis software, to examine why the performance is poor.


Table 2. Average classification results for accelerometer data.

Classification using Idle Accel Data Classifiers Classification Rates Naive Bayes 66.16% Decision trees 60.57% KNN (k=3) 67.85% SVM (SMO) 65.77%

Classification using Stationary Accel Data Classifiers Classification Rates Naive Bayes 48.98% Decision trees 54.16% KNN (k=3) 61.09% SVM (SMO) 67.64%

Classification using Idle & Stationary Accel Data Classifiers Classification Rates Naive Bayes 48.71% Decision trees 54.07% KNN (k=3) 67.57% SVM (SMO) 58.11%

Table 3. SVM and KNN confusion matrices for idle accelerometer data. SVM AA BA CA C D H I P T

AA 96.2 36.67 0 0 2.333 0 0 0 0

KNN AA BA CA C D H I P T

AA 100 37.54 0 0 0.333 0 0 0 0

BA 3.8 55.09 18.13 0 0 0 0 0 0

CA

C

D

H

I

0 0.351 77.92 0 0 0 0 0 0

0 0 0.069 99.87 5.333 40 0 0 0

0 7.895 3.819 0.067 74.8 0.267 0 0 0

0 0 0.069 0.067 17.53 59.73 0 0 0

0 0 0 0 0 0 100 100 71.67

P 0 0 0 0 0 0 0 0 0

T 0 0 0 0 0 0 0 0 28.33

BA

CA

C

D

H

I

P

T

0 61.4 3.333 0 0 0 0 0 0

0 0.702 94.93 0 0 0 0 0 0

0 0 0.556 100 3.8 40 0 0 0

0 0.351 0.417 0 86.93 0.267 0 0 0

0 0 0.764 0 8.933 59.73 0 0 0

0 0 0 0 0 0 100 94.17 91.67

0 0 0 0 0 0 0 0.119 1.296

0 0 0 0 0 0 0 5.714 7.037

5.2. Laser Vibrometer Classification Results The laser vibrometer data was classified next for six vehicles (C, D, H, I, P & T). Each vehicle was represented with approximately 300 samples per engine condition. Each classifier was trained using the features from 200 samples and tested with the remainder for the idle and non-idle stationary data. While classifying the combination of the two sets of data the classifiers were trained using 400 samples and testing on 200 samples. The training and testing data was separated to ensure that they did not contain same beam location data. The laser vibrometer data was frequently classified with accuracy greater than 90%. The average classification performance for each test is shown in Table 4. The kappa statistics in all of the classification results, except for Naïve Bayes, were higher than 0.8. The classification performance was 20-30% higher than the accelerometer classification performance; this increased classification performance was not expected. Two confusion matrices are shown in Table 5 for the idle laser vibrometer data to future investigate the increased performance. For the laser vibrometer data, minimal misclassification was seen between vehicles. Vehicles P and T were correctly identified with the laser vibrometer data most of time, unlike with accelerometer data.


Table 4. Cllassification resu ults for laser vibrrometer data.

Classification using g Idle LVS Data Classifiers Classification Rates Naive Bayes 71.59% Decisiion trees 99.33% KNN (k=3) ( 98.95% SVM (SMO) ( 95.92%

Classificatiion using Stattionary L LVS Data Classifiication Classifiers Rattes Naive Bayees 74.662% Decision trrees 80.11% KNN (k=3)) 86.665% SVM (SMO O) 94.449%

Classification n using Idle & Stationaryy LVS Data Classificatioon C Classifiers Rates N Naive Bayes 60.45% D Decision trees 91.19% K KNN (k=3) 97.33% SVM (SMO) 92.66%

Table 5. SV VM and KNN co onfusion matricees for idle laser vibrometer v data.

SV VM C H I P D T

C 100 15.53 0 0 0 0

H 0 83.5 0 0 2 0

I 0 0 100 0 0 0

P 0 0 0 1 100 0 6

D 0 0.971 0 0 98 0

T 0 0 0 0 0 94

KNN C H I P D T

C 100 0 0 0 0 0

H 0 99.03 0 0 1.333 0

I 0 0 100 0 0 0

P 0 0 0 100 0 4

D 0 0.971 0 0 98.67 0

T 0 0 0 0 0 96

Figures 6, 7, and 8 display the distributioons by class forr spectral centrroid (SC), dom minant frequenccy (DF1), and MFCC4 for idle data created using the t statistical analysis a softwaare JMP®. SC and DF1 weree selected to deetermine their effect e on the classificaation of these vehicles v as theey have previouusly been seleccted as useful features for acccelerometer data [9, 10]. MFCC4 wass selected as it displayed chaanges in vehiclle separability between accelerometer and laser vibromeeter data. SC and DF1 alone do not show an increasse in the separaability of the frequently fr conffused vehicles P and T. Vehiccles D & H become more m separable with w laser vibrrometer data when w examiningg SC as the varriance of rangee of values is laarger for vehicle D thaan vehicle H. For F MFCC4, however, h the raange of valuess for vehicles P and T overlaap much less with w laser vibrometry data d than with accelerometer a d data. This decrrease in overlapp may play a roole in the increeased performaance. Labels aSensor

550

4k

It

533

453

el

,e1

Assel

LK

/r¢I

LK

/r¢I

LK

LK

/r¢I r

Labels/Sensor

Figure 6. Feature distributio ons for spectral centroids for acccelerometer (Accel) and laser vibrometer (LVS)) data.


NI vs. Lanoeasonso.

OW

dab

100

_,._

6_

LK

/r¢I

0

Ac..

Aeeel

Ac..

Accel C4

AA

/r¢I

LK

Ir¢I 0

LK

/r¢I

LK

r

H

Labels/Sensor

Figure 7. Feature distrib butions for the dominant d frequenncy (DF1) for acccelerometer (Acccel) and laser vibrometer v (LVS S) data. MFCC4 vs_ L.abels

& Sensor

-

;

:

t

I i ti

tI LYS C

Acc

Au

CM

CBA

4 LVS

D

Acc

LVS

Acc

LVS

Acc

LVS

H

Acc

LVS T

Labels/Sensor

Figure 8. Feature F distributiions for MFCC44 for accelerometter (Accel) and laser l vibrometerr (LVS) data.

5.3. Numberr of Cylinder Classification C Results The acceleroometer and lasser vibrometer data were givven different classes c to reprresent the num mber of cylindders. The vehicle CA was w excluded from f the testingg as it is a turbbine engine. C and T represennted the 4 cylinnder class. D, H, H I, and P defined thee 6 cylinder claass. Lastly, AA A and BA charracterized the 12 1 cylinder claass. All three cllasses were tessted with the accelerom meter data; onlly 4 and 6 cyliinder data coulld be tested wiith the laser viibrometer. Thee data used to train t and test the classsifier is the com mbination of thhe idle and nonn-idle stationarry data mentionned in the prevvious two expeeriments.


Grouping the data by number of cylinders improved the classification performance of the accelerometer data seen in Table 6. The vehicles with 4 cylinders were frequently identified as 6 cylinder vehicles, seen in Table 7. The same problem was seen with the laser vibrometer data with less frequency. Table 6. Cylinder count classification results for accelerometer and laser vibrometer data.

Classifying # of Cylinders with Accelerometer Data Classifiers Classification Rates Naive Bayes 55.37% Decision trees 71.94% KNN (k=3) 77.48% SVM (SMO) 69.07%

Classifying # of Cylinders with LVS Data Classifiers Classification Rates Naive Bayes 65.36% Decision trees 96.16% KNN (k=3) 95.88% SVM (SMO) 79.97%

Table 7. SVM and KNN confusion matrices for idle accelerometer (left) and laser vibrometer data (right).

SVM Four Six Twelve

Four 26.87 17.09 2.384

Six 72.99 82.91 0.197

Twelve 0.138 0 97.42

KNN Four Six Twelve

Four 57.45 17.67 0.177

Six 42.18 82.16 6.994

Twelve 0.368 0.171 92.83

SVM Four Six

Four 65.82 5.882

Six 34.18 94.12

KNN Four Six

Four 92.09 0.321

Six 7.91 99.68

5.4. Accelerometer and Laser Vibrometer Classification Results The last test performed involved utilizing both the accelerometer and laser vibrometer data. The first test was training on accelerometer data and testing on laser vibrometer data. The second test involved training and testing on both accelerometer data and laser vibrometer data. Idle and non-idle stationary data was used for each of these tests. For the first test all 40000 samples of accelerometer data were tested and all 600 samples of laser vibrometer were trained for C, D, H, I, P & T. The second test trained on 40000 samples of accelerometer data and 400 samples of laser vibrometer data and tested on 20000 samples of accelerometer data and 200 samples of accelerometer data. All nine classes were used for the second test; however, only accelerometer data was used to test the three vehicles without laser vibrometer data. The performance of the classifiers in both of these tests shown in Table 8 was poor. When training on accelerometer and testing on laser vibrometer data C, D, H, and I were frequently misclassified. Training and testing on both accelerometer and laser vibrometer data did not improve the overall classification when compared to using accelerometer and laser vibrometer data independently. Table 8. Classification results for accelerometer and laser vibrometer data. Classification with Training on Accelerometer Data and Testing on LVS Data (Idle & Stationary Data) Classifiers Classification Rates Naive Bayes 29.36% Decision trees 32.13% KNN (k=3) 15.9% SVM (SMO) 22.79%

Classification with Accelerometer and LVS Data (Idle & Stationary Data) Classifiers Classification Rates Naive Bayes 48.76% Decision trees 60.99% KNN (k=3) 63.3% SVM (SMO) 62.26%


6. CONCLUSIONS & FUTURE WORK In this paper, the classification performance of accelerometer and laser vibrometer data is compared. An end-to-end process developed from [1] and [5] is utilized in order to classify the vibration signatures of nine vehicles. The process involves loading time series accelerometer and laser vibrometer data, integrating the accelerometer data and high pass filtering the laser vibrometer data, and preprocessing with zero centering and normalizing in MATLAB. Ten different features are also extracted from the data in MATLAB. Finally, the features are used to classify the vehicles in WEKA. Several experiments were performed using accelerometer and laser vibrometer data. Accelerometer data and laser vibrometer data were first classified separately using idle, non-idle stationary, and a combination to classify vehicles. The accelerometer data classified nine vehicles with a maximum of approximately 68% correct classification. Laser vibrometer data classified six vehicles more than 80% of the time. The laser vibrometer data appears to outperform the accelerometer data. More data is needed to further test the classification performance of the laser vibrometer data. Further testing is also needed in order to determine the effects of increasing the distance from the target and varying the beam spot size on the target. The accelerometer data should also be reexamined for errors not seen during typical quality coding. The performances of the two sensors were expected to be similar or reverse of what was seen. Division of data for testing and training, number of data samples, and unseen issues in the data could all be causing the performances currently seen. The final experiment performed involved grouping the vehicles by the number of cylinders within piston engines. Three groups were created: 4 cylinders, 6 cylinders, and 12 cylinders. The grouping of the vehicles improved the overall average classification performance of the accelerometer data. However, the laser vibrometer classification performance decreased slightly. The last experiment performed investigated training on accelerometer data and testing on laser vibrometer data to compare the testing and training of both types of data. Training on accelerometer data and testing on laser vibrometer data performed the worst of the tests with the maximum correct classification rate at 32%. Testing on both accelerometer data and laser vibrometer data did not improve the overall classification. Throughout these tests the same misclassifications are frequently seen. The 4 and 6 cylinder vehicles are misclassified. The Dodge Ram is commonly misclassified as the Honda Odyssey and the Pontiac G6 is repeatedly misclassified as an International 8600. Features that preserve the diversity between 4 and 6 cylinder piston engines and gas and diesels engines are needed in order to prevent these misclassifications. Improving the model to compare laser vibrometer to accelerometer data should also be investigated in order to produce higher classification results when utilizing both types of data. Future work for this research includes several tasks. Improving the features in the end-to-end process is important in order to reduce misclassifications. Developing features to improve classification performance are underway. Another task is to improve the model that relates accelerometer and laser vibrometer data. The process is currently simplified and can be improved to produce better correlated data. Lastly, developing a hierarchical type classifier is in the works. Previous results have shown that the classification of vehicles is well suited to a hierarchical model [8].

ACKNOWLEDGEMENTS The authors would like to thank Lindsay Cain and Scott Clouse from AFRL and Andrew Lingg and Scott Kangas from Etegent Technologies for their assistance with code and constant feedback. This paper was approved for public release via 88ABW-2015-1792.

REFERENCES [1] Kangas, S., Mendoza-Schrock, O. and Freeman, A., "Applying manifold learning to vehicle classification using vibrometry signatures," Proc. SPIE 8751 (2013). [2] Sigmund, K., Shelley, S., Bauer, M. and Heitkamp, F., "Analysis of vehicle vibration sources for automatic differentiation between gas and diesel piston engines," Proc. SPIE 8391 (2014). [3] Nooralahiyan, A., Lopez, L., Mckewon, D. and Ahmad, M., "Time-delay neural network for audio monitoring of


road traffic and vehicle classification," Proc. SPIE 2902 (1997). [4] Ma, W., Xing, D., McKee, A., Bajwa, R., Flores, C., Fuller, B. and Varaiya, P., "A wireless accelerometer-based automatic vehicle classification prototype system," Proc. IEEE Transactions on Intelligent Transportation System 15(1), 104-111 (2014). [5] Stevens, M. R., Snorrason, M. and Petrovich, D., "Laser vibrometry for target classification," Proc. SPIE 4726 (2002). [6] Stevens,M. R., Stouch,D. W., Snorrason, M. and Heitkamp, F., "Mining vibrometry signatures to determine target separability," Proc. SPIE Automatic Target Recognition XIII (2003). [7] Crider, L. and Kangas, S., "Exploiting vibration-based spectral signatures for automatic target recognition," Proc. SPIE 9079, Ground/Air Multisensor Interoperability, Integration, and Networking for Persistent ISR V (2014). [8] Nolan, A., Lingg, A., Goley, S., Sigmund, K. and Kangas, S., "Laser vibrometry exploitation for vehicle identification," Proc. SPIE 9079 Ground/Air Multisensor Interoperability, Integration, and Networking for Persistent ISR V (2014). [9] Smith, A., Mendoza-Schrock, O., Kangas, S., Dierking, M. and Shaw, A., "An end-to-end vehicle classification pipeline using vibrometry data," Proc. SPIE 9079, Ground/Air Multisensor Interoperability, Integration, and Networking for Persistent ISR V (2014). [10] Smith, A., End-to-End Classification Process for the Exploration of Vibrometry Data (Electronic Thesis) (2014). [11] Kulwanoski, G. and Schnellinger, J., "Acceleration/vbration: the principles of piezoelectric accelerometers," Sensors Online, 1 February 2004. http://www.sensorsmag.com/sensors/acceleration-vibration/the-principlespiezoelectric-accelerometers-1022. [Accessed 18 March 2015]. [12] "Polytec: Basic Principles of Vibrometry," Polytec GmbH Waldbronn, 2014. http://www.polytec.com/us/solutions/ vibration-measurement/basic-principles-of-vibrometry/. [Accessed 10 March 2014]. [13] Theodoridis, S. and Koutroumbas, K., "Chapter 6," [Pattern Recognition] Academic Press (2008). [14] Scheirer, E. and Slaney, M., "Construction and evaluation of a robust multifeature speech/music discriminator," Proc. IEEE, Acoustics, Speech and Signal Processing 2 (1997). [15] Rabiner, L. R. and Sambur, M. R., "An algorithm for determining the endpoints of isolated utterances," The Bell System Technical Journal 54(2), 297-315 (1975). [16] Tiira, T., "Discrimination of nuclear explosions and earthquakes from teleseismic distances with a local network of short period seismic stations using artificial neural networks," Physics of the Earth and Planetary Interiors 97, 247268 (1996). [17] Sobel, P. A. and von Seggern, D. H., "Analysis of Selected Seismic Events from Asia in a Seismic Discrimination Context," Teledyne Geotech (SDAC-TR-78-5), Alexandria, VA (1978). [18] Tzanetakis, G. and Cook, P., "A framework for audio analysis based on classification and temporal segmentation," in EUROMICRO 25 (1999). [19] Muda, L., Begam, M. and Elamvazuthi, I., "Voice recognition algorithms using Mel frequency cepstral coefficient (MFCC) and dynamic time warping (DTW) techniques," Journal of Computing 2(3), 138-143 (2010). [20] Kim, H. and Holmes, W. H., "Nonparametric peak feature extraction and its applications to speech signals," Proc. 8th Australian International Conference of Speech Science and Technology (2000). [21] Lartillot, O. and Toiviainen, P., "A Matlab toolbox for musical feature extraction from audio," Proc. 10th International Conference on Digital Audio Effects (2007). [22] Mitchell, T. M., [Machine Learning (Draft)], 1-17 (2010). [23] Quinlan, J. R., [C4.5 Programs for machine learning], London: Morgan Kaufmann Publishers Inc. (1988). [24] Robnik-Šikonja, M. and Kononenko, I., "Theoretical and empirical analysis of ReliefF and RReliefF," Machine Learning Journal 53, 23-69 (2003).
