SS symmetry Article
Threats of Password Pattern Leakage Using Smartwatch Motion Recognition Sensors Jihun Kim and Jonghee M. Youn * Computer Engineering, Yeungnam University, Gyeongsan, Gyeongbuk 38541, Korea;
[email protected] * Correspondence:
[email protected]; Tel.: +82-53-810-2552 Academic Editor: Laurence T. Yang Received: 3 March 2017; Accepted: 26 June 2017; Published: 30 June 2017
Abstract: Thanks to the development of Internet of Things (IoT) technologies, wearable markets have been growing rapidly. Smartwatches can be said to be the most representative product in wearable markets, and involve various hardware technologies in order to overcome the limitations of small hardware. Motion recognition sensors are a representative example of those hardware technologies. However, smartwatches and motion recognition sensors that can be worn by users may pose security threats of password pattern leakage. In the present paper, passwords are inferred through experiments to obtain password patterns inputted by users using motion recognition sensors, and verification of the results and the accuracy of the results is shown. Keywords: side-channel attack; smartwatch; motion sensor; keystroke
1. Introduction Before smartwatches were released onto the market, the watches were simply intended to identify time. However, thanks to the release of smartwatches embedded with a processor and an operating system, watches began to include instruments (smartwatches) that are used not only to identify time, but also to perform diverse functions simultaneously, and have been becoming more and more convenient to use [1]. One of representative functions of smartwatches is the fitness function, such as the measurement of step counts and calorie consumption [2]. The fitness function of smartwatches operates by calculating the rotation, direction, and movements of watches using gyroscopes that sense movements, measurement sensors such as acceleration sensors, gravity sensors, and terrestrial magnetism sensors that sense directions. Application developers utilize these measurement sensors as motion sensors to understand users’ motions, that is, as motion recognition sensors, which recognize users’ motions to perform certain functions, and are used as controllers, such as those that turn off the screen or maintain the screen dark as a default, and brighten the screen when the user has raised his/her hand [3]. The present paper noted the fact that most smartwatches support motion recognition sensors as described above, while being wearable devices that are worn on users’ wrists [4]. Since smartwatches are worn on users’ wrists, users’ wrist movements can be exposed to motion recognition sensors. The exposure of wrist movements can easily break down individuals’ privacy and important security elements [5]. This is closely related to modern trends where analog devices are replaced by digital devices. For instance, most private houses’ front doors are opened and locked using digital door locks by entering passwords instead of using mechanical keys, and banking is done through automatic teller machines (ATMs) instead of through bank clerks using paper bankbooks. In a digitalized society, attackers intending to make ill use of the convenience of digital systems are most interested in ‘passwords’ that are individuals’ unique character strings based on agreements made in advance for security. Passwords for digital door locks on front doors or those used for banking are four-digit
Symmetry 2017, 9, 101; doi:10.3390/sym9070101
www.mdpi.com/journal/symmetry
Symmetry 2017, 9, 101
2 of 21
Symmetry 2017, 9, x FOR PEER REVIEW numbers without any character
2 of 21 string in most cases, and the positions of numbers on password input panels are invariable in most cases. Furthermore, security keys such as passwords or personal keys such as passwords or personal identification numbers (PINs) are input directly with the users’ identification numbers (PINs) are input directly with the users’ fingers. To input security keys, users fingers. To input downward, security keys, users and make upward, downward, leftward, and rightward wrist make upward, leftward, rightward wrist movements, and the possibility of leakage movements, and the possibility of leakage of these movements through motion recognition sensors of these movements through motion recognition sensors in smartwatches is a major subject in the in smartwatches is a major subject in the present paper. present paper. To Toconfirm confirmthe theissue issueas assuch, such,in inthe thepresent presentpaper, paper,after afterusers userswearing wearinga asmartwatch smartwatchinput input passwords into password input panels of machines such as automated teller machines, the patterns passwords into password input panels of machines such as automated teller machines, the patterns of of the wrist movements was analyzed to estimate the passwords entered by the users. the wrist movements was analyzed to estimate the passwords entered by the users. The composed as as follows. follows. In In Section 2, proposed the proposed method devised to Thepresent presentpaper paper is is composed Section 2, the method devised to achieve achieve the objective are explained and in Section 3, experiments conducted based on the method the objective are explained and in Section 3, experiments conducted based on the method proposed in proposed in the present paper are explained and the results are shown. The results are verified in the present paper are explained and the results are shown. The results are verified in Section 4 and Section 4 and in Section 5, motion recognition sensor related works are mentioned and conclusions in Section 5, motion recognition sensor related works are mentioned and conclusions are drawn in are drawn in Section 6. Section 6.
2. Proposed Methods 2. Proposed Methods Figure 1 is a diagram of the method proposed in the present paper. The method proposed in Figure 1 is a diagram of the method proposed in the present paper. The method proposed in the the present paper consisted a smartwatch, a smartphone connected to smartwatch, the smartwatch, a present paper consisted of aof smartwatch, a smartphone connected to the and aand server server for prediction of passwords. We identified the wrist movements relevant to password input for prediction of passwords. We identified the wrist movements relevant to password input using using the measured values the accelerometer of the smartwatch the measured the measured values from from the accelerometer sensorsensor of the smartwatch and theand measured values of values of the gyroscope sensor. The smartwatch was used to simply transmit the relevant measured the gyroscope sensor. The smartwatch was used to simply transmit the relevant measured values values the smartphone. the smartphone, the password was converted an based image to theto smartphone. In theIn smartphone, the password patternpattern was converted into an into image on based on the measured value transmitted from the smartwatch, and the relevant image was the measured value transmitted from the smartwatch, and the relevant image was transmitted to the transmitted the server. The server predicted based image on the by server. Theto server predicted the password basedthe on password the transmitted bytransmitted applying theimage algorithm applying the algorithm to the image. to the image.
Figure 1. Proposed method structure. Figure 1. Proposed method structure.
2.1. Identifying User Behaviors 2.1. Identifying User Behaviors When seen from the perspective of an attacker trying to steal the password of a user wearing When seen from the perspective of an attacker trying to steal the password of a user wearing a a smartwatch, accurate password pattern cannot easily obtained if the movements the smartwatch, the the accurate password pattern cannot be be easily obtained if the movements of ofthe smartwatch are continuously collected because the user wearing a smartwatch walks and conducts smartwatch are continuously collected because the user wearing a smartwatch walks and conducts other activities of daily living, such as typing on a computer keyboard and eating meals. Therefore, we other activities of daily living, such as typing on a computer keyboard and eating meals. Therefore, inferred certain behavior patterns of users when they pressed their password at an ATM, and assumed we inferred certain behavior patterns of users when they pressed their password at an ATM, and the points of the start and end of transmission. assumed the points of the start and end of transmission. Raise the arm 1. 1. Raise the arm 2. 2. Take the actions to input a password Take the actions to input a password 3. 3. Lower the arm Lower the arm
The sensor‐measured values for the movements of the smartwatch immediately following the behavior set forth above were transmitted to the smartphone.
Symmetry 2017, 9, 101
3 of 21
The sensor-measured values for the movements of the smartwatch immediately following the behavior set forth above were transmitted to the smartphone. Symmetry 2017, 9, x FOR PEER REVIEW 3 of 21 2.2. Start Transmission and Measurement Values 2.2. Start Transmission and Measurement Values Figure 22 shows anan example of the that indicates the start ofstart sensor assumed Figure shows example of behavior the behavior that indicates the of transmission sensor transmission in Section 2.1. The figures on the top show the process of raising the arm to enter the password, assumed in Section 2.1. The figures on the top show the process of raising the arm to enter and the the smartwatch screens on the bottom show thebottom related show states the of the smartwatch with to the password, and the smartwatch screens on the related states of the regard smartwatch transmission values. Table 1 shows that The English displayed on English the smartwatch with regard of to the the sensor transmission of the sensor values. Table words 1 shows that The words screen explain the following states. displayed on the smartwatch screen explain the following states.
Figure 2. States of the smartwatch according to user motions. Figure 2. States of the smartwatch according to user motions. Table 1. States of smartwatches according to the time points of measurement recording. Table 1. States of smartwatches according to the time points of measurement recording.
State Stable State Send Stable Send
Explanation Explanation The measured values were not transmitted. The measured values were not transmitted. The measured values were transmitted. The measured values were transmitted.
2.3. Transmission of Sensor Measured Values 2.3. Transmission of Sensor Measured Values In the present paper, among motion recognition sensors, only accelerometer sensors were used for recording of measured values because accelerometer sensors are fusion sensors that can used also In the present paper, among motion recognition sensors, only accelerometer sensors were sense gravity and angular speed, and they can produce the results of many sensors without using for recording of measured values because accelerometer sensors are fusion sensors that can also sense other sensors, through the inclusion of several calculation methods. gravity and angular speed, and they can produce the results of many sensors without using other sensors, through the inclusion of several calculation methods. 2.3.1. Acceleration Sensor 2.3.1. Acceleration Sensor In general, the dictionary definition of ‘acceleration’ is the degree to which speed changes over In general, the dictionary definition of ‘acceleration’ is the degree to which speed changes over time [6,7]. For instance, acceleration is related to the feeling of the body being pushed into the car time [6,7]. For instance, acceleration is related to the feeling of the body being pushed into the car seat when the accelerator pedal has been pressed down. Therefore, it may be difficult to be certain seat when the accelerator pedal has been pressed down. Therefore, it may be difficult to be certain of acceleration acting downwards on a mass even when it remains still due to the effects of gravity. of acceleration acting downwards on a mass even when it remains still due to the effects of gravity. However, in the formula F = ma, acceleration a is associated with force F through the proportional However,termed in the formula F = ma, acceleration is associated with force F through the proportional constant mass. Therefore, acceleration a proportional to force exists everywhere that force constant termed mass. Therefore, acceleration proportional to force exists everywhere that force exists. exists. In addition to the above theory, there is a kind of acceleration that we always feel on earth, In addition to the above theory, there is a kind of acceleration that we always feel on earth, termed termed gravitational acceleration [7 8]. gravitational acceleration [7,8]. Therefore, sensors embedded in devices also measure an acceleration of 9.8 m/s when they are in the direction of gravity. Given the foregoing, it can be seen that the concepts of gravity sensors and acceleration sensors are not apart from each other [9,10]. Because acceleration is physically composed of vector quantities that have sizes and directions, in the present paper, the sizes and directions of movements of smartwatches are measured based on the measured values of acceleration sensors along the x‐, y‐, and z‐axes.
Symmetry 2017, 9, 101
4 of 21
Therefore, sensors embedded in devices also measure an acceleration of 9.8 m/s2 when they are in the direction of gravity. Given the foregoing, it can be seen that the concepts of gravity sensors and acceleration sensors are not apart from each other [9,10]. Because acceleration is physically composed of vector quantities that have sizes and directions, in the present paper, the sizes and directions of movements of smartwatches are measured based on the measured values of acceleration sensors along the x-, y-, and z-axes. Symmetry 2017, 9, x FOR PEER REVIEW 4 of 21 on device The measurements of coordinate systems and movements are also calculated based coordinate systems as shown in Figure 3. As shown in Figure 3, if the device is moved left, the x-axis The measurements of coordinate systems and movements are also calculated based on device measured Symmetry 2017, 9, x FOR PEER REVIEW value willsystems decreases, andin conversely, ifshown the device is moved x-axisleft, measured value Figure 3. As 4 of 21 coordinate as shown in Figure 3, if the right, device the is moved the x‐axis measured decreases, and conversely, if the device is moved right, ±the will increase (Figure 3a).value Here,will a general range of measurement is approximately 4. x‐axis With regard to The measurements of coordinate systems and movements are also calculated based on device measured value will increase (Figure3a). Here, a general range of measurement is approximately ± up-down movements, that is, changes in location along the y-axis, if the device is moved upward, the coordinate systems as shown in Figure 3. As shown in Figure 3, if the device is moved left, the 4. With regard to up‐down movements, that is, changes in location along the y‐axis, if the device is y-axis measured value will increase and if the is moved downward, theright, y-axis value x‐axis measured value will decreases, and device conversely, if the device is moved the measured x‐axis moved upward, the y‐axis measured value will increase and if the device is moved downward, the will decrease (Figure 3b). measured value will increase (Figure3a). Here, a general range of measurement is approximately ± y‐axis measured value will decrease (Figure3b). 4. With regard to up‐down movements, that is, changes in location along the y‐axis, if the device is moved upward, the y‐axis measured value will increase and if the device is moved downward, the y‐axis measured value will decrease (Figure3b).
(a)
(b)
Figure 3. Figure 3. ChangesChanges in acceleration sensor coordinate systems according to smartwatch movements. in acceleration sensor coordinate systems according to smartwatch movements. (a) (b) value. (a) Decrease in the x coordinate system value. (b) Increase in the y coordinate system value. (a) Decrease in the x coordinate system value. (b) Increase in the y coordinate system Figure 3. Changes in acceleration sensor coordinate systems according to smartwatch movements. 2.3.2. Simple Movements of Smartwatches in the Process of Inputting Passwords
2.3.2. Simple Movements of Smartwatches in the Process of Inputting Passwords (a) Decrease in the x coordinate system value. (b) Increase in the y coordinate system value.
The left drawing in Figure 4 (Figure 4a) is a view of movements when the user inputs a
four‐digit password 5980, 4the central 4a) drawing is the of smartwatch movement for the user’s The left drawing in Figure (Figure is a view movements whenpattern the user inputs a four-digit 2.3.2. Simple Movements of Smartwatches in the Process of Inputting Passwords resultant changes in the acceleration sensor’s measured and the right drawing is the passwordmovements, 5980, the central drawing is the smartwatch movement pattern for the user’s movements, The left drawing in Figure 4 (Figure in Figure 4b, user’s movements to input passwords and 4a) is a view of movements when the user inputs a values. As can be seen from the drawings and the right drawing is the5980, resultant changes in the acceleration sensor’s pattern measured values. As can be four‐digit password drawing smartwatch for other. the user’s the related movements of the the central smartphone can is be the regarded to be movement quite similar to each The seen from movements, the drawings in Figure 4b, user’s movements to input passwords and the related movements the resultant changes in the acceleration sensor’s measured and the right drawing is related acceleration sensor’s measured values are recorded using the x‐ and y‐axes measured values values. As can be seen from the drawings in Figure 4b, user’s movements to input passwords and because the movements are simple upward, downward, leftward, and rightward movements. of the smartphone can be regarded to be quite similar to each other. The related acceleration sensor’s the related movements of the smartphone can be regarded to be quite similar to each other. The measured values are recorded using the x- and y-axes measured values because the movements are related acceleration sensor’s measured values are recorded using the x‐ and y‐axes measured values simple upward, downward, leftward, and rightward movements. because the movements are simple upward, downward, leftward, and rightward movements.
Figure 4. User’s password input and related movements of the smartwatch. (a) Smartwatch movement. (b) Changes in measured values of the acceleration sensor. Figure 4. User’s password input and related movements the smartwatch. Smartwatch movement. Figure2.3.3. Smartwatch Movements when Password Pressing Actions are Taken 4. User’s password input and related movements of theof smartwatch. (a) (a) Smartwatch movement. (b) Changes in measured values of the acceleration sensor. (b) Changes in measured values of the acceleration sensor. If the user presses a pattern that is drawn only in the horizontal direction, such as 1333, the
movements of the smartwatch and the acceleration sensor’s measured values will be recorded as 2.3.3. Smartwatch Movements when Password Pressing Actions are Taken shown in Figure 5. In this case, although the pattern of the password can be deduced, the password 2.3.3. Smartwatch Pressing Actions are direction, Taken such as 1333, the If the Movements user presses a when pattern Password that is drawn only in the horizontal cannot be easily estimated. movements of the smartwatch and the acceleration sensor’s measured values will be recorded as If the user presses a pattern that is drawn only in the horizontal direction, such as 1333, the shown in Figure 5. In this case, although the pattern of the password can be deduced, the password movements of the smartwatch and the acceleration sensor’s measured values will be recorded as cannot be easily estimated.
Symmetry 2017, 9, 101
5 of 21
Symmetry 2017, 9, x FOR PEER REVIEW
5 of 21
shown in Figure 5. In this case, although the pattern of the password can be deduced, the password Symmetry 2017, 9, x FOR PEER REVIEW 5 of 21 cannot be easily estimated.
Symmetry 2017, 9, x FOR PEER REVIEW
5 of 21
Figure 5. Movement of the smartwatch when there are overlapping numbers. (a) Movement of the Figure 5. Movement of the smartwatch when there are overlapping numbers. (a) Movement of the smartwatch. (b) Changes in measured values of the acceleration sensor. Figure 5. Movement of the smartwatch when there are overlapping numbers. (a) Movement of the smartwatch. (b) Changes in measured values of the acceleration sensor. smartwatch. (b) Changes in measured values of the acceleration sensor.
To solve such exceptions, motions made when passwords are pressed were identified. To this were end, To decreases in the z‐axis values of the acceleration sensor were utilized. That is, as shown in solve such exceptions, motions made when passwords are pressed identified. To this To solve such exceptions, motions made when passwords are pressed were identified. To this Figure 6, pressing actions were recorded when the measured value of the z‐axis had decreased and Figure 5. Movement of the smartwatch when there are overlapping numbers. (a) Movement of the end, decreases in the z-axis values of the acceleration sensor were utilized. That is, as shown end, decreases in the z‐axis values of the acceleration sensor were utilized. That is, as shown in in smartwatch. (b) Changes in measured values of the acceleration sensor. increased thereafter. Figure 6, pressing actions were recorded when the measured value of the z-axis had decreased and Figure 6, pressing actions were recorded when the measured value of the z‐axis had decreased and increased thereafter. To solve such exceptions, motions made when passwords are pressed were identified. To this increased thereafter. end, decreases in the z‐axis values of the acceleration sensor were utilized. That is, as shown in Figure 6, pressing actions were recorded when the measured value of the z‐axis had decreased and increased thereafter.
Figure 6. Pattern acquisition utilizing the z-axis of the acceleration sensor. (a) Movement of the Figure 6. Pattern acquisition utilizing the z‐axis of the acceleration sensor. (a) Movement of the smartwatch. (b) Changes in measured values of the acceleration sensor. smartwatch. (b) Changes in measured values of the acceleration sensor. Figure 6. Pattern acquisition utilizing the z‐axis of the acceleration sensor. (a) Movement of the Figure 6. Pattern acquisition utilizing the z‐axis of the acceleration sensor. (a) Movement of the 2.4. Terminate Transmission a Measurement Values smartwatch. (b) Changes in measured values of the acceleration sensor. smartwatch. (b) Changes in measured values of the acceleration sensor.
2.4. Terminate Transmission a Measurement Values In the2.4. Terminate Transmission a Measurement Values present paper, the transmission of the measured values was terminated at the motion of the 2.4. Terminate Transmission a Measurement Values In the present paper, the transmission of the measured values was terminated at the motion of user to ‘lower the arm’ after taking the motion to enter the password. In the present paper, the transmission of the measured values was terminated at the motion of the user to ‘lower the arm’ after taking the motion to enter the password. In the present paper, the transmission of the measured values was terminated at the motion of Figure 7 shows the time point of termination, that is, the user’s motion to lower the arm and the user to ‘lower the arm’ after taking the motion to enter the password. Figure 7 shows the time point of termination, that is, the user’s motion to lower the arm and the user to ‘lower the arm’ after taking the motion to enter the password. resultant statesFigure 7 shows the time point of termination, that is, the user’s motion to lower the arm and of the smartwatch. While the user was inputting the password, the measured values of resultant states of the smartwatch. While the user was inputting the password, the measured values resultant states of the smartwatch. While the user was inputting the password, the measured values Figure 7 shows the time point of termination, that is, the user’s motion to lower the arm and the smartwatch were recorded and transmitted as a ‘Send’ state. However, at the time point where of the smartwatch were recorded and transmitted as a a ‘Send’ ‘Send’ state. However, at the time point of the smartwatch were recorded and transmitted as state. However, at the time point resultant states of the smartwatch. While the user was inputting the password, the measured values the arm was completely lowered, the smartwatch was put into a stable state and the recording of arm was completely lowered, the the smartwatch smartwatch was put put into a stable and the and the where the where arm the was completely lowered, was into a state stable state of the smartwatch were recorded and transmitted as a ‘Send’ state. However, at the time point measured values was terminated. recording of measured values was terminated. recording of measured values was terminated. where the arm was completely lowered, the smartwatch was put into a stable state and the recording of measured values was terminated.
Figure 7. States of the smartwatch according to the motion of the user (termination).
Figure 7. States of the smartwatch according to the motion of the user (termination). 2.5. Password Estimation
Figure 7. States of the smartwatch according to the motion of the user (termination). Figure 7. States of the smartwatch according to the motion of the user (termination).
2.5. Password Estimation
2.5. Password Estimation
Symmetry 2017, 9, 101
6 of 21
Symmetry 2017, 9, x FOR PEER REVIEW
6 of 21
2.5. Password Estimation In the present paper, password patterns were acquired through the movements of users In the paper,when password were acquired throughdownward, the movements of users wearing wearing a present smartwatch they patterns pressed passwords. Upward, leftward, rightward, aand smartwatch when they pressed passwords. Upward, downward, leftward, rightward, and diagonal diagonal movements were recorded as the movements of the acceleration sensor along the movements werey‐axis, recorded asthe the actions movements of thepasswords accelerationwere sensor along thewith x-axischanges and the y-axis, x‐axis and the and to press measured in the and the actions to press passwords were measured with changes in the acceleration sensor’s positions acceleration sensor’s positions along the z‐axis. However, there were difficulties in estimating along the z-axis. However, there were difficulties in estimating actual numbers pressed by users when actual numbers pressed by users when based on only these movement patterns. basedTo onsolve only these movement this problem, in patterns. the present paper, acquired patterns were used as information to To solve this problem, in the present paper, acquired patterns were used as information to estimate estimate the numbers actually pressed by users. the numbers actually pressed by users. Figure 8 shows the method proposed in the present paper that was used to estimate pressed Figure 8 shows the method proposed in the present paper that was used to estimate pressed passwords based on acquired patterns. Let us assume that a pattern of a four‐digit password was passwords onin acquired Let us assume thaton a pattern of a indicate four-digitactions password was acquired as based shown the left patterns. figure. Here, the numbers the pattern taken to acquired as shown in the left figure. Here, the numbers on the pattern indicate actions taken to press press the panel, that is, the sections in which the z‐axis measured values decreased and increased the panel, that is, the sectionsdo in which the z-axis values decreased pressed and increased thereafter thereafter and the numbers not indicate the measured numbers in the password by the user but and the numbers do not indicate the numbers in the password pressed by the user but the order of the order of pressing. That is, the user took actions to pressed the password input panel in order of pressing. That is, the user took actions to pressed the password input panel in order of the numbers 1, the numbers 1, 2, 3, and 4. 2, 3, and 4.
Figure Assumption of of patterns. patterns. (a) (a) Acquired Acquired password password pattern; pattern; (b) (b) Password Password input input panel; panel; (c) (c) Figure 8. 8. Assumption Password pattern on a grid plate. Password pattern on a grid plate.
In the the present present paper, paper, the user’s password input panel assumed as shown the central In the user’s password input panel waswas assumed as shown in thein central figure figure in Figure 8. The password input panel as such was simply regarded as a grid plate. In the in Figure 8. The password input panel as such was simply regarded as a grid plate. In the present present paper, the pattern was shown on a grid plate, as shown in the right figure in Figure 8. paper, the pattern was shown on a grid plate, as shown in the right figure in Figure 8. Before estimating Before numbers, estimating the sizes of range coordinates, that is, the range used for in actual the actual sizes ofnumbers, coordinates, that is, the to be used in calculations, wasto setbe simply calculations, was set simply for the accuracy of estimated numbers. the accuracy of estimated numbers. Figure 9 shows the method of setting the range of estimation. First, 1 is located at the left end Figure 9 shows the method of setting the range of estimation. First, 1 is located at the left end and 2 is located at the right end. The horizontal range of estimation is set based on 1 and 2, that is, and 2 is located at the right end. The horizontal range of estimation is set based on 1 and 2, that is, the the left end and the right end. In addition, 1 is located not only at the left and but also at the top and left end and the right end. In addition, 1 is located not only at the left and but also at the top and 4 is 4 is located the bottom. The vertical range of estimation is set on that 1 and 4, that is, the located at theat bottom. The vertical range of estimation is set based onbased 1 and 4, is, the uppermost uppermost and the lowermost coordinates. and the lowermost coordinates.
Symmetry 2017, 9, 101 Symmetry 2017, 9, x FOR PEER REVIEW
7 of 21 7 of 21
Figure 9. Setting of ranges of estimation. (a) Leftmost coordinate; (b) Rightmost coordinate; (c) Range of estimation (horizontal direction); (d) Uppermost coordinate; (e) Lowermost coordinate; (f) Range of estimation (vertical direction).
The final range of estimation can be indicated as shown by the left drawing of Figure 10 which shows the common segment of the horizontal range of estimation and the vertical range of estimation. Next, the estimations of actually pressed numbers are calculated. In the present paper, the estimations were calculated based on the coordinates where pressing actions were taken. In particular, the leftmost, rightmost, uppermost, and lowermost coordinates are used as important Figure 9. Setting of ranges of estimation. (a) Leftmost coordinate; (b) Rightmost coordinate; (c) Range information for estimation. For let (a) us Leftmost estimate coordinate; the actual (b) numbers pressed by the Figure 9. Setting of ranges of instance, estimation. Rightmost coordinate; (c) user of estimation (horizontal direction); (d) Uppermost coordinate; (e) Lowermost coordinate; (f) Range of from Range of estimation (horizontal direction); (d) Uppermost coordinate; (e) Lowermost coordinate; (f) the first number to the last one, based on the acquired pattern shown in Figure 10. First, estimation (vertical direction). Range of estimation (vertical direction). calculations as such can be easily understood by assuming that the pattern is drawn on the central figure in Figure 8 that is, the password input panel. Let us think about the location where the first The final range estimation can belocation indicatedwhere as shown by output the left and drawing of Figurewhere 10 which The final range of estimation can be indicated as shown by the left drawing of Figure 10 which pressing action was oftaken, that is, the 1 was the location the shows the common segment of the horizontal range of estimation and the vertical range of estimation. shows the common segment of the horizontal range of estimation and the vertical range of second pressing action was taken, that is, the location where 2 was output. estimation. Next, the estimations of actually pressed numbers are calculated. In the present paper, the estimations were calculated based on the coordinates where pressing actions were taken. In particular, the leftmost, rightmost, uppermost, and lowermost coordinates are used as important information for estimation. For instance, let us estimate the actual numbers pressed by the user from the first number to the last one, based on the acquired pattern shown in Figure 10. First, calculations as such can be easily understood by assuming that the pattern is drawn on the central figure in Figure 8 that is, the password input panel. Let us think about the location where the first pressing action was taken, that is, the location where 1 was output and the location where the second pressing action was taken, that is, the location where 2 was output.
Figure 10. Final range of estimation. Figure 10. Final range of estimation.
Since the location where the last pressing action was taken, that is, the location where 4 was Next, the estimations of actually pressed numbers are calculated. In the present paper, the output, is located between locations 1 and 2, the password input numbers 1 and 3, 4 and 6, and 7 estimations were calculated based on the coordinates where pressing actions were taken. In particular, and 9 are expected for the locations 1 and 2. Next, the location where the third pressing action was the leftmost, rightmost, uppermost, and lowermost coordinates are used as important information for taken, that is, the location where 3 was output, is located below the location where 2 was output, estimation. For 9 instance, us estimate thepanel actualare numbers pressed bysame the user from the first number and 3, 6, and in the let password input located on the vertical row. Therefore, to the last one, based on the acquired pattern shown in Figure 10. First, calculations as such can be easily understood by assuming that the pattern is drawn on the central figure in Figure 8 that is, the password input panel. Let us think about the location where the first pressing action was taken, that is, the location where 1 was output and the location where the second pressing action was taken, that is, Figure 10. Final range of estimation. the location where 2 was output. Since is, the 4 was Since the the location location where where the the last last pressing pressing action action was was taken, taken, that that is, the location location where where 4 was output, is located between locations 1 and 2, the password input numbers 1 and 3, 4 and 6, and 7 and 9 output, is located between locations 1 and 2, the password input numbers 1 and 3, 4 and 6, and 7 are expected for the locations 1 and 2. Next, the location where the third pressing action was taken, and 9 are expected for the locations 1 and 2. Next, the location where the third pressing action was taken, that is, the location where 3 was output, is located below the location where 2 was output, and 3, 6, and 9 in the password input panel are located on the same vertical row. Therefore,
Symmetry 2017, 9, 101
8 of 21
that is, the location where 3 was output, is located below the location where 2 was output, and 3, 6, and 9 in the password input panel are located on the same vertical row. Therefore, regardless of whether the estimated number pressed by the second pressing action is 3 or 6, the estimated number pressed by the third pressing action becomes 6 or 9. In the same manner, since 1, 2, or 3 is located at the location where the last pressing action was taken, that is, the location where 4 was output, 1 was output on the left and 2 and 3 were output on the right, the password input numbers 8 or 0 can be expected. The estimated numbers explained as above are set forth in Table 2. According to the acquired pattern shown in Figure 8, the finally expected passwords are 1368, 1360, 1390, and 4690. Table 2. Estimation of passwords according to the locations where pressing actions were taken. Estimated Number First and second Third Fourth Finally estimated number
(1, 3) 6 8 1368
(1, 3) 6 0 1360
(1, 3) 9 0 1390
(4, 6) 9 0 4690
(7, 9) ← none none
2.6. Computational Complexity Prediction of passwords in the present paper was calculated based on the obtained pattern images. Coordinate values were judged to have been pressed on the grid plate, and the areas above, below, left, right to the stored nodes were explored to predict the password. If the password was pressed on one coordinate value only, as with 3333, the computational complexity was shown as O(1) because there was no need to explore the areas above, below, left, or right of the coordinate value, and even in the worst case where all the areas above, below, left, right to the coordinate value were explored, the computational complexity was shown as O n2 . 2.7. Filter and Inertia Pure measured values of sensors involved errors such as noises, drifts, and zero offset. Various filters were necessary to improve the accuracy and precision of the sensors’ measured values. 2.7.1. Low-Pass Filter Low-pass filters are used in acceleration sensors to remove gravity acceleration [11]. Since the measured values used in the present paper required pure acceleration removed of gravitational acceleration, gravitational acceleration was removed from measured values of acceleration. Low-pass filters as such were used during data equalization to minimize noise. A general method of applying low-pass filters for data equalization is used by obtaining the newest value through weighting of the previous average value. The equalization parameter a is used as follows Formula (1):
(6= w value) = ( previous value) + xi × a − ( previous value) × a
(1)
where x (the value collected the most recently) weighted by a is added to the value calculated previously, and the previous value weighed by a is deducted to obtain the difference. If a is close to a value of 1, the new value will be x, and if a is close to zero, the new value in the calculation formula will not be different from the previous value. Here, x can have the desired level of effects on the new value. 2.7.2. Kalman Filter Purely measured sensor values cannot be easily utilized accurately, due to interference and noises. To solve this problem, an optimum mathematical calculation process was introduced that enabled the prediction of locations after a certain time, by analyzing existing measured values mixed with noises through the least-squares method. This calculation process is called the Kalman Filter [12]. To mention
Symmetry 2017, 9, 101
9 of 21
Symmetry 2017, 9, x FOR PEER REVIEW Symmetry 2017, 9, x FOR PEER REVIEW
9 of 21 9 of 21
with expansion, the Kalman Filter is based on measurement progression over time. More accurate results can be expected from the Kalman Filter than from using the results of measurements conducted measurements conducted only at relevant moments. The Kalman Filter recursively processes input measurements conducted only at relevant moments. The Kalman Filter recursively processes input only at relevant moments. The Kalman Filter recursively processes input data states. including noise, and data including noise, enables optimum statistical deduction of the current states. entire data including noise, and and enables optimum statistical deduction of the current The The entire enables optimum statistical deduction of the current states. The entire algorithm can be divided into algorithm can be divided into two parts; prediction and update. The prediction refers to algorithm can be divided into two parts; prediction and update. The prediction refers to the the two parts; prediction and update. The prediction refers to the prediction of the current states and the prediction of the current states and the update refers to the enabling of more accurate prediction by prediction of the current states and the update refers to the enabling of more accurate prediction by update the refers to the enabling of morein accurate prediction by including the observed in including the measurements observed in the current state. As shown in measurements Figure measured including measurements observed the current state. As shown in Figure 11, 11, the the measured the current state. As shown in Figure 11, the measured values of the smartwatch accelerometer sensor values of the smartwatch accelerometer sensor before applying the Kalman Filter pose difficulties in values of the smartwatch accelerometer sensor before applying the Kalman Filter pose difficulties in before applying the Kalman Filter pose difficulties in accurate measurement due to noise. This poses accurate measurement due to noise. This poses difficulties in acquiring the smartwatch’s movement accurate measurement due to noise. This poses difficulties in acquiring the smartwatch’s movement difficulties in acquiring the smartwatch’s movement patterns, and the application of the Kalman Filter patterns, and the application of the Kalman Filter reduces noise in measured values to assist in the patterns, and the application of the Kalman Filter reduces noise in measured values to assist in the reduces noise in measured values to assist in the acquisition of accurate patterns. acquisition of accurate patterns. acquisition of accurate patterns.
(a) (a)
(b) (b)
Figure 11. Application of the Kalman Filter to the measured values of the accelerometer sensor. (a) Figure 11. Application of the Kalman Filter to the measured values of the accelerometer sensor. (a) Figure 11. Application of the Kalman Filter to the measured values of the accelerometer sensor. Before applying the Kalman Filter. (b) After applying the Kalman Filter. Before applying the Kalman Filter. (b) After applying the Kalman Filter. (a) Before applying the Kalman Filter. (b) After applying the Kalman Filter.
2.7.3. Inertia 2.7.3. Inertia 2.7.3. Inertia Inertia refers to the tendency to maintain the state of movements, and the resistance of objects Inertia refers to the tendency to maintain the state of movements, and the resistance of objects Inertia refers to the tendency to maintain the state of movements, and the resistance of objects when the state of movements is changed. All movements on the earth follow this law of inertia and when the state of movements is changed. All movements on the earth follow this law of inertia and when the state of movements is changed. All movements on the earth follow this law of inertia and the movements of sensors are not exceptional. the movements of sensors are not exceptional. the movements of sensors are not exceptional. After moving a smartwatch laid onto a plane, the values as shown in Figure 12 were measured After moving a smartwatch laid onto a plane, the values as shown in Figure 12 were measured After moving a smartwatch laid onto a plane, the values as shown in Figure 12 were measured in in real life. in real life. real life.
[x, y, z] [x, y, z] [0.2, 0, 0] [0.2, 0, 0] [0.5, 0, 0] [0.5, 0, 0] [2.3, 0, 0] [2.3, 0, 0] [5.0, 0, 0] [5.0, 0, 0] [2.2, 0, 0] [2.2, 0, 0] [−0.2, 0, 0] [−0.2, 0, 0] [−5.1, 0, 0] [−5.1, 0, 0] (a) (a)
(b) (b)
Figure 12. Movement of smartwatch and measured values of acceleration sensor in relation Figure 12. 12. Movement of of smartwatch and measured values of acceleration acceleration sensor relation to to Figure Movement smartwatch and measured values of sensor in in relation to inertia. inertia. (a) Changes in the measured values of the acceleration sensor; (b) Record of movement. inertia. (a) Changes in the measured values of the acceleration sensor; (b) Record of movement. (a) Changes in the measured values of the acceleration sensor; (b) Record of movement.
As shown in the right‐hand figure in Figure 12, the smartwatch’s movement, which should be As shown in the right‐hand figure in Figure 12, the smartwatch’s movement, which should be As shown in the right-hand figure in Figure 12, the smartwatch’s movement, which should be recorded with one line, was actually recorded by two lines. recorded with one line, was actually recorded by two lines. recorded with one line, was actually recorded by two lines. Changes in the acceleration sensor’s measured values, and the portion actually relevant to the Changes in the acceleration sensor’s measured values, and the portion actually relevant to the Changes in the acceleration sensor’s measured values, and the portion actually relevant to the present paper, are shown in Figure 13. present paper, are shown in Figure 13. present paper, are shown in Figure 13.
Symmetry 2017, 9, 101 Symmetry 2017, 9, x FOR PEER REVIEW Symmetry 2017, 9, x FOR PEER REVIEW
10 of 21 10 of 21 10 of 21
Figure 13. Changes in measured values of acceleration along the x‐axis in relation to inertia. Figure 13. Changes in measured values of acceleration along the x‐axis in relation to inertia. Figure 13. Changes in measured values of acceleration along the x-axis in relation to inertia.
Since the smartwatch was moved to the right, the values of movement along the x‐axis should Since the smartwatch was moved to the right, the values of movement along the x‐axis should Since the smartwatch was moved to the right, the values of movement along the x-axis should be positive numbers. moved to the right, among the measured positive numbers, only the be positive numbers. When When the right, right, among among the the measured measured positive positive numbers, numbers, only only the the be positive numbers. When moved moved to to the maximum value was recorded, and other values were omitted. maximum value was recorded, and other values were omitted. maximum value was recorded, and other values were omitted. 3. Experiment 3. Experiment 3. Experiment In this section, the objective of the present paper is presented and methods that were planned In this section, the objective of the present paper is presented and methods that were planned In this section, the objective of the present paper is presented and methods that were planned and and implemented in the present paper were proposed. and implemented in the present paper were proposed. implemented in the present paper were proposed. 3.1. Experimental Environment 3.1. Experimental Environment 3.1. Experimental Environment First, we used the LG Urbane smartwatch (LG Electronics, Seoul, South Korea) [13], and the First, First, we we used used the the LG LG Urbane Urbane smartwatch smartwatch (LG (LG Electronics, Electronics, Seoul, Seoul, South South Korea) Korea) [13], [13], and and the the Samsung Galaxy 6 (Samsung Electronics, Suwon, South Korea) [14] as a smartphone connected to Samsung Galaxy 6 (Samsung Electronics, Suwon, South Korea) [14] as a smartphone connected to Samsung Galaxy 6 (Samsung Electronics, Suwon, South Korea) [14] as a smartphone connected to the smartwatch. In addition, we made an effort to perform the experiment in an environment very the smartwatch. In addition, we made an effort to perform the experiment in an environment very the smartwatch. In addition, we made an effort to perform the experiment in an environment very similar to the actual environment where passwords are entered into an actual ATM. To this end, we similar to the actual environment where passwords are entered into an actual ATM. To this end, we similar to the actual environment where passwords are entered into an actual ATM. To this end, we first checked the exterior specifications of ATMs. The ATM selected for the study was the ATM of first checked the exterior specifications of ATMs. The ATM selected for the study was the ATM of the first checked the exterior specifications of ATMs. The ATM selected for the study was the ATM of the bank used the most frequently in the area where we were located, and we selected one of the bank used the most frequently in the area where we were located, and we selected one of the most the bank used the most frequently in the area where we were located, and we selected one of the most crowded places containing ATM users. crowded places containing ATM users. most crowded places containing ATM users. Figure shows an environment assumed to an Figure shows an experimental environment assumed to configure an environment maximally Figure 1414 14 shows an experimental experimental environment assumed to configure configure an environment environment maximally close to a real ATM. In the place where the actual user was standing, the height of the close to a real ATM. In the place where the actual user was standing, the height of the screen was maximally close to a real ATM. In the place where the actual user was standing, the height of the screen was approximately 85 cm. Therefore, a test password input device tablet was placed on approximately 85 cm. Therefore, a test password device tablet was placed on a was table,placed which on wasa screen was approximately 85 cm. Therefore, a input test password input device tablet a table, which was approximately 85 cm high to replicate the same height for the test. In addition, the approximately 85 cm high to replicate the same height for the test. In addition, the screen used to enter table, which was approximately 85 cm high to replicate the same height for the test. In addition, the screen used passwords 7 cm and long, the experimental actual was 7actual cm wide and 10 cmwas long, the experimental tablet used screen passwords used to to enter enter actual passwords was 7 and cm wide wide and 10 10 cm cm application long, and and for the the experimental application for the tablet used in the test was also made in the same size. The password input in the test was the same size. password input device of actual ATMs was tilted by application for also the made tablet inused in the test The was also made in the same size. The password input ◦ device of actual ATMs was tilted by approximately 5°. Since this is almost the same as a flat surface, approximately 5 . Since this is almost the same as a flat surface, the experimental tablet was set to be device of actual ATMs was tilted by approximately 5°. Since this is almost the same as a flat surface, the experimental tablet was set to be placed on a flat surface. placed on a flat surface. the experimental tablet was set to be placed on a flat surface.
Figure 14. View of the experimental environment. Figure 14. View of the experimental environment. Figure 14. View of the experimental environment.
Symmetry 2017, 9, 101
11 of 21
In the experiment, we wore a smartwatch on the right wrist because most people enter their password with their right hand. Most people wear the watch on the left wrist regardless of whether they are left-handed or right-handed [15,16]. However, in general, right-handed women tend to wear a smartwatch on their right wrist, and some forums argue that they generally wear a smartwatch on their right wrist [17,18]. Furthermore, according to a study conducted by [19], approximately 30% of all people are ambidextrous, and their hands where they wear accessories vary with certain working environments. In addition, manufacturers of smartwatches such as Samsung and Apple have announced that left-handed people wear smartwatches on their right wrist [20,21] and they separately support left-handed people’s mode for such users [22,23]. Moreover, among smartwatch applications, fitness applications such as those that measure the trajectory of the wrist to help the correction of exercise postures have been released. Unlike general watches, these applications should be worn on the right wrist because they are loaded on smartwatches. The smartwatch market will continue to develop, and these fitness applications are a reason for why many people wear the smartwatch on their right wrist. 3.2. Scenario In the present paper, users wearing a smartwatch input passwords as four-digit numbers into the password input panel on a plane like an ATM, the password patterns were acquired, and the passwords were estimated based on the patterns. 3.3. Test Case In the present paper, a total of 10 test cases were fabricated. Table 3 shows 10 patterns and the characteristics of the patterns. Table 3. Password test cases. Test Case
Password
Characteristics of the Patterns
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
1234 2580 1333 2000 1399 1357 1733 7597 9054 8642 75920 39547 13333 10000 79831 976154 391736 798105 800000 139712
horizontal vertical horizontal, overlapping vertical, overlapping ¬-shaped, overlapping horizontal, diagonal vertical, diagonal triangle random random 5-digits 5-digits 5-digits 5-digits 5-digits 6-digits 6-digits 6-digits 6-digits 6-digits
The passwords consisted of four-digit numbers, and for the 10 test cases, seven passwords had characteristic shapes such as horizontal, vertical, triangle, and diagonal, and in addition to the characteristic shapes, passwords that contained overlapping numbers were also included. In addition, two test cases were randomly input to conduct experiments on password pattern acquisition.
Symmetry 2017, 9, x FOR PEER REVIEW Symmetry 2017, 9, x FOR PEER REVIEW Symmetry 2017, 9, x FOR PEER REVIEW Symmetry 2017, 9, x FOR PEER REVIEW Symmetry 2017, 9, x FOR PEER REVIEW Symmetry 2017, 9, x FOR PEER REVIEW Symmetry 2017, 9, x FOR PEER REVIEW Symmetry 2017, 9, x FOR PEER REVIEW
12 of 21 12 of 21 12 of 21 12 of 21 12 of 21 12 of 21 12 of 21 12 of 21
The passwords consisted of four‐digit numbers, and for the 10 test cases, seven passwords had The passwords consisted of four‐digit numbers, and for the 10 test cases, seven passwords had The passwords consisted of four‐digit numbers, and for the 10 test cases, seven passwords had The passwords consisted of four‐digit numbers, and for the 10 test cases, seven passwords had The passwords consisted of four‐digit numbers, and for the 10 test cases, seven passwords had The passwords consisted of four‐digit numbers, and for the 10 test cases, seven passwords had The passwords consisted of four‐digit numbers, and for the 10 test cases, seven passwords had characteristic shapes such as horizontal, vertical, triangle, and diagonal, and in addition to the characteristic shapes such as as horizontal, vertical, triangle, and diagonal, and in in addition to to the the characteristic shapes such as horizontal, vertical, triangle, and diagonal, and in addition to the characteristic shapes such horizontal, vertical, triangle, and diagonal, and addition characteristic shapes, passwords that contained overlapping numbers were also included. In characteristic shapes, passwords that contained overlapping numbers were also included. In characteristic shapes, passwords that contained overlapping numbers were also included. In characteristic shapes, passwords that contained overlapping numbers were also included. In characteristic shapes, passwords that contained overlapping numbers were also included. In characteristic shapes, passwords that contained overlapping numbers were also included. In In characteristic shapes, passwords that contained overlapping numbers were also included. addition, two test cases were randomly input to conduct experiments on password pattern addition, two test cases were randomly input to to conduct experiments on on password pattern addition, two test cases were randomly input conduct experiments password pattern addition, two test cases were randomly input to conduct experiments on password pattern addition, two test cases were randomly input to conduct experiments on password pattern addition, two test cases were randomly input to conduct experiments on password pattern addition, two test cases were randomly input to conduct experiments on password pattern 3.4. Pattern Acquisition acquisition. acquisition. acquisition. acquisition. acquisition. acquisition. acquisition.
Symmetry 2017, 9, 101 shapes 12the of 21 characteristic shapes such as horizontal, vertical, triangle, and diagonal, and in addition to the characteristic shapes such as as horizontal, vertical, triangle, and diagonal, and in in addition to to the characteristic shapes such horizontal, vertical, triangle, and diagonal, and addition characteristic such as horizontal, vertical, triangle, and diagonal, and in addition to the
Table 4 shows the passwords in the test cases, the characteristics of the passwords, and the 3.4. Pattern Acquisition 3.4. Pattern Acquisition 3.4. Pattern Acquisition 3.4. Pattern Acquisition 3.4. Pattern Acquisition 3.4. Pattern Acquisition 3.4. Pattern Acquisition password patterns acquired in the experiments. of the the passwords, and the the passwords, and the the Table 4 shows the passwords in the test cases, the characteristics Table 4 shows the passwords in the test cases, the characteristics of of passwords, and the passwords, and the Table 4 shows shows the the passwords in the the test cases, the the characteristics of the Table 4 shows shows the passwords in the the test cases, the characteristics of the the passwords, and the Table 4 the passwords in test cases, the characteristics of passwords, and the Table 4 passwords in test cases, characteristics of passwords, and the password patterns acquired in the experiments. password patterns acquired in the experiments. password patterns acquired in the experiments. Table 4. Acquired patterns by test case. password patterns acquired in the experiments. password patterns acquired in the experiments. password patterns acquired in the experiments. password patterns acquired in the experiments. Table 4. Acquired patterns by test case. Table 4. Acquired patterns by test case. Table 4. Acquired patterns by test case. Table 4. Acquired patterns by test case. Characteristics Characteristics Table 4. Acquired patterns by test case. Table 4. Acquired patterns by test case. Table 4. Acquired patterns by test case. Acquired Test Case Password of the Test Case Password of the Characteristics of the Characteristics of the Acquired Test Test Test Test Characteristics of the Characteristics of the Patterns Acquired Test Test Characteristics of the Characteristics of the Acquired Test Test Characteristics of the Characteristics of the Acquired Characteristics of the Characteristics of the Acquired Test Test Characteristics of the Characteristics of the Acquired Test Test Password Password Characteristics of the Characteristics of the Acquired Test Test Password Characteristics of the Password Characteristics of the Characteristics of the Characteristics of the Acquired Test Test Password Password Acquired Test Test Characteristics of the Characteristics of the Acquired Test Test Patterns Patterns Password Password Password Password Password Password Patterns Patterns Patterns Case Case Patterns Patterns Patterns Case Case Password Password Password Password Patterns Patterns Patterns Case Case Patterns Patterns Patterns Case Case Password Password Case Case Password Case Password Case Case Case Case
1 1 1 1 1 1
1 1 1 1
1234 1234 1234 1234 1234 1234 1234 1234 1234 1234
Patterns Patterns Patterns Patterns Patterns Patterns Patterns
Patterns Patterns Patterns Patterns Patterns Patterns Patterns
Case Case Password Case Password Case Case Case Case
horizontal horizontal horizontal horizontal horizontal horizontal horizontal horizontal horizontal horizontal
11 11 11 11 11 11 11 11 11 11
2 2 2 2 2
2580 2580 2580 2580 2580 2580 2580 2580 2580 2580 2580
4 4 4 4 4
2000 2000 2000 2000 2000 2000 2000 2000 2000 2000 2000
5 5 5 5 5
5 5 5 5
6
6 6 6 6 6
1357 1357 1357 1357 1357 1357 1357 1357 1357 1357 1357
7 7 7 7 7
1733 1733 1733 1733 1733 1733 1733 1733 1733 1733 1733
8 8 8 8 8
7597 7597 7597 7597 7597 7597 7597 7597 7597 7597 7597
9 9
9 9 9 9 9
9054 9054 9054 9054 9054 9054 9054 9054 9054 9054
10 10 10 10 10 10 10 10
8642 8642 8642 8642 8642 8642 8642 8642 8642 8642 8642
6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6-digits
6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6-digits
139712 139712 139712 139712 139712 139712 139712 139712 139712 139712 139712
20 20 20 20 20 20 20 20 20 20 20
random random random random random random random random random random random
800000 800000 800000 800000 800000 800000 800000 800000 800000 800000
19 19 19 19 19 19 19 19 19 19
random random random random random random random random random random
10 1010
6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6-digits 6‐digits
798105 798105 798105 798105 798105 798105 798105 798105 798105 798105 798105
18 18 18 18 18 18 18 18 18 18 18
triangle triangle triangle triangle triangle triangle triangle triangle triangle triangle triangle
9 9 9 9
6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6-digits 6‐digits 6‐digits
8 8
391736 391736 391736 391736 391736 391736 391736 391736 391736 391736 391736
17 17 17 17 17 17 17 17 17 17 17
diagonal
vertical, diagonal vertical, vertical, diagonal vertical, diagonal vertical, diagonal vertical, diagonal vertical, diagonal vertical, diagonal vertical, diagonal vertical, diagonal vertical, diagonal
8 8 8 8
6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6-digits 6‐digits 6‐digits 6‐digits
7
976154 976154 976154 976154 976154 976154 976154 976154 976154 976154 976154
16 16 16 16 16 16 16 16 16 16 16
diagonal
horizontal, horizontal, diagonal horizontal, diagonal horizontal, diagonal horizontal, diagonal horizontal, diagonal horizontal, diagonal horizontal, diagonal horizontal, diagonal horizontal, diagonal horizontal, diagonal
7 7 7 7 7
5‐digits 5‐digits 5‐digits 5‐digits 5-digits 5‐digits 5‐digits 5‐digits 5‐digits
79831 79831 79831 79831 79831 79831 79831 79831 79831
15 15 15 15 15 15 15 15 15
¬-shaped, ㄱ‐shaped, overlapping ㄱ‐shaped, overlapping 1399 ㄱ‐shaped, overlapping 1399 ㄱ‐shaped, overlapping 1399 ㄱ‐shaped, overlapping 1399 1399 1399 1399 ㄱ‐shaped, overlapping ㄱ‐shaped, overlapping 1399 ㄱ‐shaped, overlapping 1399 ㄱ‐shaped, overlapping overlapping
6 6 6 6 6
5‐digits 5‐digits 5‐digits 5‐digits 5‐digits 5-digits 5‐digits 5‐digits 5‐digits 5‐digits 5‐digits
10000 10000 10000 10000 10000 10000 10000 10000 10000 10000 10000
14 14 14 14 14 14 14 14 14 14 14
vertical, vertical, overlapping vertical, overlapping vertical, overlapping vertical, overlapping vertical, overlapping vertical, overlapping vertical, overlapping vertical, overlapping vertical, overlapping vertical, overlapping overlapping
5‐digits 5‐digits 5‐digits 5‐digits 5-digits 5‐digits 5‐digits 5‐digits 5‐digits 5‐digits 5‐digits
4 4 4 4 4 4
13333 13333 13333 13333 13333 13333 13333 13333 13333 13333 13333
13 13 13 13 13 13 13 13 13 13 13
horizontal, 1333 horizontal, overlapping 1333 horizontal, overlapping horizontal, overlapping 1333 1333 horizontal, overlapping 1333 1333 horizontal, overlapping 1333 horizontal, overlapping 1333 horizontal, overlapping 1333 horizontal, overlapping 1333 horizontal, overlapping 1333 horizontal, overlapping overlapping
5‐digits 5‐digits 5‐digits 5‐digits 5-digits 5‐digits 5‐digits 5‐digits 5‐digits 5‐digits 5‐digits
3 3 3 3 3
39547 39547 39547 39547 39547 39547 39547 39547 39547 39547 39547
3 3 3 3 3 3
12 12 12 12 12 12 12 12 12 12 12
vertical vertical vertical vertical vertical vertical vertical vertical vertical vertical vertical
2
5‐digits 5‐digits 5-digits 5‐digits 5‐digits 5‐digits 5‐digits 5‐digits 5‐digits 5‐digits
2 2 2 2 2
75920 75920 75920 75920 75920 75920 75920 75920 75920 75920
Patterns Patterns Patterns Patterns Patterns Patterns Patterns
Acquired Acquired Acquired Acquired Patterns Acquired Acquired Acquired Acquired Acquired Acquired Acquired Patterns Patterns Patterns Patterns Patterns Patterns Patterns Patterns Patterns Patterns
3.5. Pattern Acquisition by Angle Change We tried to prepare an environment very similar to the actual ATMs. Also, the password input device of actual ATMs was tilted by approximately 5◦ , which is almost the same as a flat surface, and the experimental tablet was placed on a flat surface. Since the acceleration sensors mounted on the smartwatch are fusion sensors that can also sense gravity and angular speed, the acquired password
Symmetry 2017, 9, x FOR PEER REVIEW Symmetry 2017, 9, x FOR PEER REVIEW Symmetry 2017, 9, x FOR PEER REVIEW
13 of 21 13 of 21 13 of 21
3.5. Pattern Acquisition by Angle Change 3.5. Pattern Acquisition by Angle Change 3.5. Pattern Acquisition by Angle Change Symmetry 2017, 9, 101
13 of 21
We tried to prepare an environment very similar to the actual ATMs. Also, the password input We tried to prepare an environment very similar to the actual ATMs. Also, the password input We tried to prepare an environment very similar to the actual ATMs. Also, the password input device of actual ATMs was tilted by approximately 5°, which is almost the same as a flat surface, and device of actual ATMs was tilted by approximately 5°, which is almost the same as a flat surface, and device of actual ATMs was tilted by approximately 5°, which is almost the same as a flat surface, and pattern can be different depending on the angle of the ATM’s keypad. For this reason, we discussed the experimental tablet was placed on a flat surface. Since the acceleration sensors mounted on the the experimental tablet was placed on a flat surface. Since the acceleration sensors mounted on the the experimental tablet was placed on a flat surface. Since the acceleration sensors mounted on the smartwatch are fusion sensors that can also sense gravity and angular speed, the acquired password how the password patterns were acquired as the keypad angle increased by 10◦ . Table 5 shows smartwatch are fusion sensors that can also sense gravity and angular speed, the acquired password smartwatch are fusion sensors that can also sense gravity and angular speed, the acquired password pattern can be different depending on the angle of the ATM’s keypad. For this reason, we discussed pattern can be different depending on the angle of the ATM’s keypad. For this reason, we discussed pattern can be different depending on the angle of the ATM’s keypad. For this reason, we discussed acquired password patterns as the angle increases by 10◦ , using the same password (9054) within the how the password patterns were acquired the keypad angle increased 10°. Table 5 acquired shows how the password patterns were acquired as the keypad angle increased by 10°. 5 shows how the password acquired as as the keypad angle increased by by 10°. Table 5 shows experiment to compare.patterns As thewere angle changed, there was no significant difference inTable the acquired password patterns as the angle increases by 10°, using the same password (9054) within the acquired password patterns as the angle increases by 10°, using the same password (9054) within the acquired password patterns as the angle increases by 10°, using the same password (9054) within the password pattern, but when compared to the flat surface (0◦ ), additional lines were drawn. This was experiment to compare. As the angle changed, there was no significant difference in the acquired experiment to compare. As the angle changed, there was no significant difference in the acquired experiment to compare. As the angle changed, there was no significant difference in the acquired indicated by a hatched rectangle, and it confirmed that the greater the angle increased, the longer the password pattern, but when compared to the flat surface (0°), additional lines were drawn. This was password pattern, but when compared to the flat surface (0°), additional lines were drawn. This was password pattern, but when compared to the flat surface (0°), additional lines were drawn. This was lengthindicated by a hatched rectangle, and it confirmed that the greater the angle increased, the longer the increased. However, in this paper, we estimate the action to press passwords as changes in the indicated by a hatched rectangle, and it confirmed that the greater the angle increased, the longer the indicated by a hatched rectangle, and it confirmed that the greater the angle increased, the longer the z-axis,length increased. However, in this paper, we estimate the action to press passwords as changes in so even if such lines are added, there was still no difficulty in estimating the password. length increased. However, in this paper, we estimate the action to press passwords as changes in length increased. However, in this paper, we estimate the action to press passwords as changes in the z‐axis, so even if such lines are added, there was still no difficulty in estimating the password. the z‐axis, so even if such lines are added, there was still no difficulty in estimating the password. the z‐axis, so even if such lines are added, there was still no difficulty in estimating the password. Table 5. Pattern acquisition by angle change. Table 5. Pattern acquisition by angle change. Table 5. Pattern acquisition by angle change. Table 5. Pattern acquisition by angle change. Acquired Acquired Tablet Angle TabletTablet Angle Tablet Angle Tablet Tablet Tablet Tablet Tablet Tablet Tablet Tablet Tablet Tablet Tablet Tablet Angle Tablet Angle Tablet Angle Patterns Acquired Patterns Acquired Patterns Acquired Patterns Acquired Patterns Acquired Patterns Acquired Patterns Patterns Tablet Angle Tablet Angle Acquired Patterns Acquired Patterns Acquired Patterns Acquired Patterns Tablet Angle Acquired Patterns Acquired Patterns Angle Angle Angle Angle Angle Angle Angle Angle Angle
0◦
Angle Angle Angle
10◦10° 10° 10° 10° 10° 10°
0° 0° 0° 0° 0° 0°
30° 30° 30° 30° 30° 30◦ 30°
20° 20◦20° 20° 20° 20° 20°
40° 40° 40° 40° 40° 40◦40°
Acquired Patterns
Acquired Patterns Acquired Patterns Acquired Patterns Acquired Patterns Acquired Patterns Acquired Patterns
50° 50° 50° 50° 50° 50◦50°
3.6. Results of Password Estimation 3.6. Results of Password Estimation 3.6. Results of Password Estimation
3.6. Results of Password Estimation
Table 6 shows the results of the password estimation analysis. Table 6 shows the results of the password estimation analysis. Table 6 shows the results of the password estimation analysis.
Table 6 shows the results of the password estimation analysis.
Table 6. Estimated passwords by test case. Table 6. Estimated passwords by test case. Table 6. Estimated passwords by test case.
Table 6. Estimated passwords by test case. Test Case Test Case Test Case Password Password Password Characteristics of the Patterns Characteristics of the Patterns Characteristics of the Patterns Test Case Test Case Password Password Characteristics of the Patterns Characteristics of the Patterns Test Case Password Characteristics of the Patterns
1234, 4560, 7890 1234, 4560, 7890 1234, 4560, 7890 1234, 4560, 7890 1234, 4560, 7890 1234, 4560, 7890
1 1 1 1
2 2
2 2 2 2
3
3 3 3 3
1333 1333 1333 1333 1333 1333 1234
horizontal, overlapping horizontal, overlapping horizontal, overlapping horizontal, overlapping horizontal, overlapping horizontal, overlapping horizontal
1222, 1333, 4555, 4666, 7888, 7999, 0000 1222, 1333, 4555, 4666, 7888, 7999, 0000 1222, 1333, 4555, 4666, 7888, 7999, 0000 1222, 1333, 4555, 4666, 7888, 7999, 0000 1222, 1333, 4555, 4666, 7888, 7999, 0000 1222, 1333, 4555, 4666, 7888, 7999, 0000 1234, 4560, 7890
100 100 100 100 100100 100
4 4 4 4
2000 2000 2000 2000 2000 2000 2580
vertical, overlapping vertical, overlapping vertical, overlapping vertical, overlapping vertical, overlapping vertical, overlapping vertical
1444, 1777, 1000, 2555, 2888, 2000, 3666, 3999 1444, 1777, 1000, 2555, 2888, 2000, 3666, 3999 1444, 1777, 1000, 2555, 2888, 2000, 3666, 3999 1444, 1777, 1000, 2555, 2888, 2000, 3666, 3999 1444, 1777, 1000, 2555, 2888, 2000, 3666, 3999 1444, 1777, 1000, 2555, 2888, 2000, 3666, 3999 1470, 2580, 3690
100 100 100 100 100 100100
5 5 5 5
1399 1399 1399 1333 1399 1399 1399
ㄱ‐shaped, overlapping ㄱ‐shaped, overlapping ㄱ‐shaped, overlapping horizontal, overlapping ㄱ‐shaped, overlapping ㄱ‐shaped, overlapping ㄱ‐shaped, overlapping
6 6 6 6
2000 1357 1357 1357 1357 1357 1357
vertical, overlapping horizontal, diagonal horizontal, diagonal horizontal, diagonal horizontal, diagonal horizontal, diagonal horizontal, diagonal
1357 1733 9054 9054 9054 9054 9054 9054 7597 8642 8642 8642 8642 8642 8642 9054 75920 75920 75920 75920 75920 75920 8642 39547 39547 39547 39547 39547 39547 75920 13333 13333 13333 39547 13333 13333 13333 13333 10000 10000 10000 10000 10000 10000
horizontal, diagonal triangle triangle triangle triangle triangle triangle vertical, diagonal random random random random random random triangle random random random random random random random 5‐digits 5‐digits 5‐digits 5‐digits 5‐digits 5‐digits random 5‐digits 5‐digits 5‐digits 5‐digits 5‐digits 5‐digits 5-digits 5‐digits 5‐digits 5‐digits 5-digits 5‐digits 5‐digits 5‐digits 5-digits 5‐digits 5‐digits 5‐digits 5‐digits 5‐digits 5‐digits
1 3 4 2 4 5 3 5 4 6 6 5 7 7 6 8 8 7 9 8 9 10 910 11 1011 12 1112 13 1213 1314 14 15 1415
7 7 7 7 8 8 8 8 9 9 9 9 10 10 10 10 11 11 11 11 12 12 12 12 13 13 13 13 14 14 14 14
Password
2580 2580 2580 2580 2580 2580
1399 1733 1733 1733 1733 1733 1733
7597 7597 7597 7597 7597 7597
horizontal horizontal horizontal of the horizontal horizontal horizontal Characteristics vertical vertical vertical Patterns vertical vertical vertical
Estimation Success Rate Estimation Success Rate Estimation Success Rate Estimation Success Rate Estimation Success Rate Estimation Success Rate
1 1
Test Case
1234 1234 1234 1234 1234 1234
Estimated Passwords Estimated Passwords Estimated Passwords Estimated Passwords Estimated Passwords Estimated Passwords
¬-shaped, overlapping vertical, diagonal vertical, diagonal vertical, diagonal vertical, diagonal vertical, diagonal vertical, diagonal
15 79831 79831 15 15 15 79831 79831 79831 79831 10000
5‐digits 5‐digits 5‐digits 5‐digits 5‐digits 5‐digits 5-digits
16 16 16 976154 976154 976154 976154 16 16 976154 976154 16
6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6‐digits
17 391736 391736 17 17 17 391736 391736 391736 391736
6‐digits 6‐digits 6‐digits 6‐digits 6‐digits 6‐digits
15 17 17 16 17 18
79831 976154 391736 798105
5-digits 6-digits 6-digits 6-digits
19
800000
6-digits
20
139712
6-digits
Estimated Passwords 1470, 2580, 3690 1470, 2580, 3690 1470, 2580, 3690 1470, 2580, 3690 1470, 2580, 3690 1470, 2580, 3690
1255, 1288, 1366, 1399 1255, 1288, 1366, 1399 1255, 1288, 1366, 1399 1222, 1333, 4555, 4666, 7888, 7999, 0000 1255, 1288, 1366, 1399 1255, 1288, 1366, 1399 1255, 1288, 1366, 1399 1444, 1777, 1000, 2555, 2888, 2000, 3666, 3999 1357, 1350, 4680 1357, 1350, 4680 1357, 1350, 4680 1357, 1350, 4680 1357, 1350, 4680 1357, 1350, 4680 1255, 1288, 1366, 1399 1733, 1766, 4033, 4066 1733, 1766, 4033, 4066 1733, 1766, 4033, 4066 1733, 1766, 4033, 4066 1733, 1766, 4033, 4066 1733, 1766, 4033, 4066 1357, 1350, 4680 4264, 7297, 7597 4264, 7297, 7597 4264, 7297, 7597 4264, 7297, 7597 4264, 7297, 7597 4264, 7297, 7597 1733, 1766, 4033, 4066 6821, 6021, 9054 6821, 6021, 9054 6821, 6021, 9054 6821, 6021, 9054 6821, 6021, 9054 6821, 6021, 9054 4264, 7297, 7597 0975, 0972, 8642 0975, 0972, 8642 0975, 0972, 8642 0975, 0972, 8642 0975, 0972, 8642 0975, 0972, 8642 6821, 6021, 9054 7592075920 75920 75920 75920 75920 0975, 0972, 8642 39547, 39540 39547, 39540 39547, 39540 39547, 39540 39547, 39540 39547, 39540 75920 13333, 46666, 79999, 13333, 46666, 79999, 13333, 46666, 79999, 39547, 39540 13333, 46666, 79999, 13333, 46666, 79999, 13333, 46666, 79999, 13333, 46666, 79999, 14444, 17777, 10000, 25555, 28888, 20000 3666, 39999 14444, 17777, 10000, 25555, 28888, 20000 3666, 39999 14444, 17777, 10000, 25555, 28888, 20000 3666, 39999 14444, 17777, 10000, 25555, 28888, 20000 3666, 39999 14444, 17777, 10000, 25555, 28888, 20000 3666, 39999 14444, 17777, 10000, 25555, 28888, 20000 3666, 39999 14444, 17777, 10000, 25555, 28888, 20000 3666, 46531, 79831, 79864 46531, 79831, 79864 46531, 79831, 79864 46531, 79831, 79864 46531, 79831, 79864 46531, 79831, 79864 39999 976154 976154 976154 976154 976154 976154 46531, 79831, 79864 281714, 391736, 392825 281714, 391736, 392825 281714, 391736, 392825 281714, 391736, 392825 281714, 391736, 392825 281714, 391736, 392825 976154 281714, 391736, 392825 798105 144444, 177777, 100000, 200000, 255555, 288888, 366666, 399999 400000, 4777777, 500000, 588888, 699999, 700000, 800000 136412, 139012, 139712, 469745
100 100 100 100 100 100 Estimation 100 100 100 Success Rate 100 100 100
100 100 100100 100 100 100
100100 100 100 100 100 100 100100 100 100 100 100 100
100 100 100 100 100 100 100100 100 100 100 100 100 100 100100 100 100 100 100 100 100100 100 100 100 100 100 100100 100 100 100100 100 100 100 100100 100 100 100 100 100 100 100 100 100 100 100
100 100 100 100 100 100100 100 100 100 100 100 100
100 100 100 100
100 100 100 100 100 100
100 100
Symmetry 2017, 9, 101
14 of 21
4. Verification The reduction rate of the number of cases was calculated by Formula (2), and on reviewing the reduction rates along with the characteristics by pattern in Table 7, the following characteristics of decreases in reduction rates were found. 1−
the number o f estimated passwords total number o f estimated
(2)
Table 7. Reduction rate by test case.
Test Case
Password
No. of Digits
Characteristics of the Patterns
Total Number of Cases of Passwords
Number of Estimated Password
Number of Cases Reduction Rate
1 1234 4 horizontal 1000 3 99.700 2 2580 4 vertical 1000 3 99.700 3 1333 4 horizontal, overlapping 1000 7 99.300 4 2000 4 vertical, overlapping 1000 8 99.200 Symmetry 2017, 9, x FOR PEER REVIEW 15 of 21 5 1399 4 ¬-shaped, overlapping 1000 4 99.600 6 1357 4 horizontal, diagonal 1000 3 99.700 7 1733 4 vertical, diagonal 1000 4 of overlapping digits decreases, the range of movements of pattern increases and 99.600 the range of 8 7597 4 triangle 1000 3 99.700 locations of pressing actions would also increase, leading to narrower ranges of estimation. 9 9054 4 random 1000 3 99.700 10 8642 4 random 1000 3 99.700 11 75920 5 random 10,000 1 99.990 4.3. Reduction Rates for the Number of Cases According to Pattern Complexity 12 39547 5 random 10,000 2 99.980 13 13333 5 horizontal, overlapping 10,000 3 99.970 14 10000 overlapping 99.920 On reviewing Figures 15 5and 16 vertical, once again, it can be 10,000 seen that the 8rate of reduction of the 15 79831 5 Random 10,000 3 99.970 number of cases increased as the number of digits of passwords increased and the number of 16 976154 6 Random 100,000 1 99.999 17 391736 6 Random 100,000 3 99.997 overlapping digits decreased. This means that the rate of reduction of the number of cases 18 798105 6 Random 100,000 1 99.999 decreased when the motions taken by the user to input passwords were wider and more complex 19 800000 6 vertical, overlapping 100,000 15 99.985 20 139712 6 random 100,000 4 99.996 and the locations where the motions were taken were not overlapping and were more complicated.
Organized according to the number of digits, the reduction rates according to the complexity of 4.1. Reduction Rates for Number of Cases by the Number of Digits of Passwords user’s password patterns can be explained with Figure 17. It was shown that when the actions taken by the user to input the password were more complicated, the reduction rate was higher. Figure 15 shows reduction rates according to the numbers of digits in the passwords.
Figure 15. Reduction rates according to the numbers of digits of passwords. Figure 15. Reduction rates according to the numbers of digits of passwords.
If the number of digits of the passwords were one, the total number of cases of passwords would be 10. However, the reduction rate would be zero because the user took only one action for each password, and the number of estimated passwords would also be 10. Even when the number of digits of passwords was only increased to two, the reduction rates would increase drastically because the
user’s password patterns can be explained with Figure 17. It was shown that when the actions taken by the user to input the password were more complicated, the reduction rate was higher.
Symmetry 2017, 9, 101
15 of 21
total number of cases increases by a power of 10, but the number of estimated passwords in the present paper was not large enough to the extent that the number exceeded 10. 4.2. Reduction Rates for Number of Cases by the Number of Overlapping Digits Figure 16 shows the rates of reduction of the numbers of cases according to the numbers of overlapping digits calculated on the basis of four-digit passwords. It can be seen that as the number of overlapping digits increased, the reduction rate decreased. This is because, if all the numbers in four digits were assumed to be the same, the user would take pressing actions at the same location, the number of leading to a wider range of passwords that could be estimated. On the contrary, when overlapping digits decreases, the range of movements of pattern increases and the range of locations Figure 15. Reduction rates according to the numbers of digits of passwords. of pressing actions would also increase, leading to narrower ranges of estimation.
Figure 16. Reduction rates according to the numbers of overlapping digits (on the basis of Figure 16. Reduction rates according to the numbers of overlapping digits (on the basis of four‐digit four-digit passwords). passwords).
4.3. Reduction Rates for the Number of Cases According to Pattern Complexity On reviewing Figures 15 and 16 once again, it can be seen that the rate of reduction of the number of cases increased as the number of digits of passwords increased and the number of overlapping digits decreased. This means that the rate of reduction of the number of cases decreased when the motions taken by the user to input passwords were wider and more complex and the locations where the motions were taken were not overlapping and were more complicated. Organized according to the number of digits, the reduction rates according to the complexity of user’s password patterns can be explained with Figure 17. It was shown that when the actions taken by the user to input the password were more complicated, the reduction rate was higher.
Symmetry 2017, 9, 101 Symmetry 2017, 9, x FOR PEER REVIEW
16 of 21 16 of 21
Figure 17. Reduction rates according to the complexity of patterns. Figure 17. Reduction rates according to the complexity of patterns.
4.4. Binary Classification Result 4.4. Binary Classification Result The precision, recall factors, and F‐scores calculated in 20 test cases are as shown in Table 8. The precision, recall factors, and F-scores calculated in 20 test cases are as shown in Table 8. Table 8. Binary classification result. Table 8. Binary classification result. Test Numbers of Predicted Password Predicted Passwords Precision NumbersPasswords of Case Recall Test Case Password Predicted Passwords Predicted Precision 1234, 4560, 7890 1 1234 3 0.333 Factor Passwords 1470, 2580, 3690 2 2580 3 0.333 1222, 1333, 4555, 4666, 7888, 7999, 0000 3 1333 7 0.143 1 1234 1234, 4560, 7890 3 0.333 1.000 1444, 1777, 1000, 2555, 2888, 2000, 3666, 3999 4 2000 8 0.125 2 2580 1470, 2580, 3690 3 0.333 1.000 1255, 1288, 1366, 1399 5 1399 4 0.250 3 1357 1333 1222, 1333,1357, 1350, 4680 4555, 4666, 7888, 7999, 0000 7 0.143 1.000 6 3 0.333 4 1733 2000 1444, 1777, 1733, 1766, 4033, 4066 1000, 2555, 2888, 2000, 3666, 3999 8 0.125 1.000 7 4 0.250 5 7597 1399 1255, 1288, 1366, 1399 4 0.250 1.000 4264, 7297, 7597 8 3 0.333 6 9054 1357 1357, 1350, 4680 3 0.333 1.000 6821, 6021, 9054 9 3 0.333 0975, 0972, 8642 10 3 0.333 7 8642 1733 1733, 1766, 4033, 4066 4 0.250 1.000 75920 11 1 1.000 8 75920 7597 4264, 7297, 7597 3 0.333 1.000 39547, 39540 12 2 0.500 9 39547 9054 6821, 6021, 9054 3 0.333 1.000 13333, 46666, 79999 13 3 0.333 10 13333 8642 0975, 0972, 8642 3 0.333 1.000 14444, 17777, 10000, 25555, 28888, 20000 36666, 39999 14 8 0.125 11 10000 75920 75920 1 1.000 1.000 46531, 79831, 79864 15 79831 3 0.333 12 39547 39547, 39540 2 0.500 1.000 976154 16 976154 1 1.000 13 391736 13333 13333, 46666, 79999 3 0.333 1.000 281714, 391736, 392825 17 3 0.333 14444, 17777, 10000, 25555, 28888, 20000 36666, 798105 18 1 1.000 14 798105 10000 8 0.125 1.000 39999 144444, 177777, 100000, 200000, 255555, 288888, 366666, 399,999 400,000, 19 800000 15 0.067 15 79831 46531, 79831, 79864 3 0.333 1.000 4777777, 500000, 588888, 699999, 700000, 800000 16 139712 976154 976154 1 1.000 1.000 136412, 139012, 139712, 469745 20 4 0.250
Recall F‐Score Factor F-Score 1.000 0.500 1.000 0.500 1.000 0.250 0.500 1.000 0.222 0.500 1.000 0.400 0.250 1.000 0.500 0.222 1.000 0.400 1.000 0.4000.500 1.000 0.5000.500 1.000 0.4000.500 1.000 0.5001.000 1.000 0.5000.667 1.000 0.5000.500 1.000 1.0000.222 1.000 0.500 0.667 1.000 1.000 0.500 1.000 0.500 1.000 0.2221.000 1.000 0.5000.125
1.000 1.0000.400 281714, 391736, 392825 3 0.333 1.000 0.500 798105 1 1.000 1.000 1.000 The precision is 144444, the probability for the actual 177777, 100000, 200000, 255555, 288888,password to be included in the predicted 19 800000 400,000, 4777777, 500000, 588888, 15 0.067 1.000 0.125 passwords, the recall 366666, rate 399,999 is the probability for predicted passwords to be included in the actual 699999, 700000, 800000 20 139712 the f‐score 136412, 139712,obtained 469745 0.400 passwords, and is 139012, a score by the 4 formula; 0.250 f‐score 1.000 = 2×(precision ×
17 18
391736 798105
recall/precision+recall) [24 ]. The precision was identified to be low through calculations with values not exceeding 33% in most for cases although tothe of passwords that The precision is the probability thebecause actual password be number includedof in cases the predicted passwords, could be 1000 in the case of four digit passwords was reduced to 1~7, the actual number of the recall rate is the probability for predicted passwords to be included in the actual passwords, and passwords was fixed to 1. However, since all the groups of passwords predicted using the method the f-score is a score obtained by the formula; f-score = 2×(precision × recall/precision+recall) [24]. proposed in the present paper included actual passwords, the recall factors were identified as 100% The precision was identified to be low through calculations with values not exceeding 33% in most in all cases. cases because although the number of cases of passwords that could be 1000 in the case of four digit passwords was reduced to 1~7, the actual number of passwords was fixed to 1. However, since all 4.5. Energy Efficient the groups of passwords predicted using the method proposed in the present paper included actual passwords, the recall factors were identified as 100% in all cases. We have also set forth the energy efficiency of the smartwatch and smartphone with regard to password prediction. Figure 18 shows a graph of the amounts of energy consumed in the experiment described in the present paper. The upper side shows the energy consumed when one test case is experimented. One test case took approximately 5 s of experimental time and both the smartwatch and smartphone showed a charge amount of 99.8%. The bottom side is a report on the amount of energy
Symmetry 2017, 9, 101
17 of 21
4.5. Energy Efficient We have also set forth the energy efficiency of the smartwatch and smartphone with regard to password prediction. Figure 18 shows a graph of the amounts of energy consumed in the experiment described in the present paper. The upper side shows the energy consumed when one test case is experimented. One test case took approximately 5 s of experimental time and both the smartwatch and smartphone Symmetry 2017, 9, x FOR PEER REVIEW 17 of 21 showed a charge amount of 99.8%. The bottom side is a report on the amount of energy consumed when all the 20 test cases were experimented. Approximately 120 s of experimental time was spent consumed when all the 20 test cases were experimented. Approximately 120 s of experimental time when the 20 test cases were experimented. The smartwatch showed a charge amount of 96.8% with an was spent when the 20 test cases were experimented. The smartwatch showed a charge amount of energy consumption of 3.2%, while the smartphone showed a charge amount of approximately 98% 96.8% with an energy consumption of 3.2%, while the smartphone showed a charge amount of with a battery energy consumption of approximately 2%. The reason why the battery consumption of approximately 98% with a battery energy consumption of approximately 2%. The reason why the the smartwatch was higher was that the smartwatch was loaded with a battery with lower specification battery consumption of the smartwatch was higher was that the smartwatch was loaded with a than that of the smartphone, considering the size and specification of the smartwatch [12,13]. The fact battery with lower specification than that of the smartphone, considering the size and specification that the period of use of the smartwatch was longer than that of smartphone was also a reason for of the smartwatch [12,13]. The fact that the period of use of the smartwatch was longer than that of this effect. smartphone was also a reason for this effect.
Figure (upper side: Experiment 1 test case,case, bottom side: side: Experiment 20 test20 case). Figure 18. 18. Energy Energy efficiency efficiency (upper side: Experiment 1 test bottom Experiment test case).
5. Related Work Studies of attacks on components of smart devices such as sensors have mainly dealt with threats that may arise from using smart device sensors, rather than from the weaknesses of algorithms that can occur within the code of the application that runs the sensor [25,26]. Most smart devices are installed not with only motion sensors, but with other sensors such as various cameras,
Symmetry 2017, 9, 101
18 of 21
5. Related Work Studies of attacks on components of smart devices such as sensors have mainly dealt with threats that may arise from using smart device sensors, rather than from the weaknesses of algorithms that can occur within the code of the application that runs the sensor [25,26]. Most smart devices are installed not with only motion sensors, but with other sensors such as various cameras, global positioning system (GPS), and microphones. Previous studies have been conducted focusing on privacy infringement using information from these sensors. For instance, [27] designed a Driver Detection System (DDS) that calculates the change values of the motion sensor of the smartphone to detect the vehicle driving direction. This system first detects the NFC (Near Field Communication) and audio of the firmware installed in the vehicle with the smartphone to detect the presence of the driver in the driver’s seat nearby, and thereafter calculates the change values of motion sensors such as the accelerometer and gyroscope sensor to detect the driving speed and direction. In addition, the author of [28] injected malware that operates sensors into the smartphone to expose the privacy of user-based information leaked from sensors other than the motion sensor, that is, the microphone, GPS data, and camera. One of the interesting and sensitive pieces of information attackers aim to obtain in the field of security threats is information on passwords or pin codes. Such pieces of information can be predicted by expanding keystroke threats that can predict acts of typing on the smartphone screen. Studies conducted before the commercialization of smartwatches first used motion sensors to confirm the keystrokes on smartphones. Both studies [29,30] utilized motion sensors to predict motions for typing on the virtual keyboard of a touch screen and used the degree of change from the gyroscope sensor and accelerometer sensors. The authors of [29] have particularly conducted experiments where the users did not type at a fixed location but did type on handheld smart devices, and additionally mentioned the relationship between vibration state and keystrokes in such cases, and the author of [30] compared results from gyroscope sensors with those from accelerometer sensors to show that gyroscope sensors were superior to accelerometer sensors in terms of precision. Studies on keystrokes as such have been expanded to the prediction of passwords or pin codes. In [31], the movements of motion sensors inside the smartphones were monitored by installing a Trojan application. The affected sensors were the gyroscope and accelerometer sensors, and the application learned behaviors such as the touch events of the user, and predicted the actions conducted by the user on the touch screen based on the learning. Password stealing was one of the experiments in the current study. The current study conducted the experiment while expanding PIN codes consisting only of numbers to four, six, and eight digits, and showed a prediction rate of 80%. The author of [32] uses only accelerometer sensors and predicts passwords with character strings rather pin codes consisting of numbers. The author predicted the passwords by dividing smartphone screens into certain rectangles in proportion to the sizes of the screens and calculating the distances to coordinate values being touched. After smartwatches were commercialized, the threats of various keystrokes were demonstrated based on the movements of the smartwatch [33–36]. For instance, studies that predicted the movements of the smartwatch to predict keyboard typing logs [33] or smartphone pin codes [34] were conducted. The author of [33] predicted English words typed on the keyboard of a desktop or laptop computer based on signals from motion sensors such as the smartwatch’s accelerometer sensors and gyroscope sensors. The position of the wrist changed according to the position of alphabets on the keyboard and the degree of change, which was measured by the smart sensor's motion sensor, were used to predict the words being typed. Similarly, the author of [34] predicted PINs using motion sensors such as the accelerometer and gyroscope sensor, based on the movements of the smartwatch when the user entered the pin number of the smartphone after wearing a smartwatch. The pin number prediction of the relevant study adopted the random forest learning method, and the signals of the x-, y-, z-axes of the gyroscope and the accelerometer were used as features of the random forest learning. Similarly, the author of [35] also predicted pin numbers wearing a smartwatch. In the relevant study, the author
Symmetry 2017, 9, 101
19 of 21
made a 12-key number input device for pin number input by himself and predicted the pressed pin numbers through machine learning (deep learning). Our study was most similar to studies [34,35] in that the purpose of the study was to predict passwords such as PINs pressed while wearing a smartwatch. However, studies [34,35] predicted pin numbers based on machine learning. This means that the sensor values recorded in the smartwatch could be used again in learning, and the features used in learning could be also collected. Such processes mean that quite some effort and time should be spent in deriving predictable results. On the contrary, we predicted the passwords only through a series of calculation processes using only algorithms, without learning or feature collection, which naturally led to a reduction in the time and costs of prediction. 6. Conclusion In the present paper, the threat of sufficient leakage of user’s password patterns through the motion recognition sensors embedded in smartwatches that are prominent in wearable markets, was proved. Most smartwatches are provided with motion recognition sensors to expand the functionality and to overcome the limitations of hardware in smartwatches. However, users’ passwords can be sufficiently leaked through these motion recognition sensors. Therefore, it can be said that, ironically, wearable devices such as smartwatches released for increasing speed and convenience, may be subject to security threats such as password pattern leakage, due to that fact that those wearable devices are worn by the users. In the present paper, passwords were predicted by collecting the patterns of passwords entered by users wearing a smartwatch using accelerometer sensors, which are motion recognition sensors. Basically, if the number of digits of passwords was assumed to be four, the number of cases of possible passwords is 104 = 1000. However, according to the method proposed in the present paper, three to eight passwords were predicted for each of the four-digit passwords depending on the overlapping numbers, showing a reduction rate for the number of cases, of 99%, and the predicted passwords showed high accuracy. In the current study, the experiment was expanded to predict passwords with more than four digits. In the results of this experiment, the reduction rate for the number of cases decreased as the number of digits of the passwords increased, or as the number of overlapping numbers of passwords decreased. This means that the information for the prediction of passwords increased as the number of numbers not overlapping increased. On the contrary, the method proposed in the present paper showed a limitation in that the number of cases did not decrease at all in the case of passwords consisting of only one number of one digit passwords. However, the above limitation could be overcome because bank users do not set their important passwords as a single number or a one-digit password, pursuant to the password protection policy [37]. We plan to carry out future works with more extended scenarios. The password prediction system proposed in the present paper simply transmits the sensor values from the smartwatch to the connected smartphone, and the transmitted sensor values are again transmitted to the server connected to the smartphone. First, this may involve a weak point because it can be subject to spatial restriction in security threat scenarios, and we plan to overcome this weak point, because attackers wish to capture target information without being subject to spatial restriction. Therefore, their first goal is usually to steal the sensor values from the smartwatch within the smartphone, and secretly retransmit the sensor values from the smartphone to the cloud environment. Second, the user’s act of transmitting sensor values may be a limitation. In the present paper, sensor values were transmitted assuming that the act of inputting passwords was a prototype, and the point of transmitting sensor values was set to the point of completion of transmission. However, the act of transmitting sensor values can be a limitation because such behaviors can sufficiently occur in everyday life. To overcome the foregoing, we plan to approach the issue with a reinforced scenario such as enabling the user to obtain sensor values only when the user conducts the act of entering the password near the relevant GPS address after storing the GPS values of the coordinates of the bank and ATM.
Symmetry 2017, 9, 101
20 of 21
Acknowledgments: This work was supported by the MSIP (Ministry of Science, ICT and Future Planning), Korea, under the ITRC (Information Technology Research Center) support program (IITP-2016-R2718-16-0035) supervised by the IITP (National IT Industry Promotion Agency), the Basic Science Research Program through the NRF funded by the Ministry of Education (NRF-2015R1C1A1A02037561) and the 2016 Yeungnam University Research Grant. Author Contributions: Jihun Kim and Jonghee M. Youn conceived, designed and performed the experiments; Jihum Kim analyzed the data; Jihun Kim and Jonghee M. Youn wrote the paper. Conflicts of Interest: The authors declare no conflicts of interest.
References 1. 2.
3.
4.
5.
6.
7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17.
18.
19. 20.
Rawassizadeh, R.; Price, B.A.; Petre, M. Wearables: Has the age of smartwatches finally arrived? Commun. ACM 2015, 58, 45–47. [CrossRef] Chernbumroong, S.; Atkins, S.A.; Yu, H. Activity classification using a single wrist-worn accelerometer. In Proceedings of the 5th International Conference on Software, Knowledge Information, Industrial Management and Applications (SKIMA), Benevento, Italy, 8–11 September 2011. Hinckley, K.; Pierce, J.; Sinclair, M.; Horvitz, E. Sensing techniques for mobile interaction. In Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology, San Diego, CA, USA, 6–8 November 2000. Rawassizadeh, R.; Tomitsch, M.; Nourizadeh, M.; Momeni, E.; Peery, A.; Ulanova, L.; Pazzani, M. Energy-Efficient Integration of Continuous Context Sensing and Prediction into Smartwatches. Sensors 2015, 15, 22616–22645. [CrossRef] [PubMed] Udoh, E.S.; Alkharashi, A. Privacy risk awareness and the behavior of smartwatch users: A case study of Indiana University students. In Proceedings of the Future Technologies Conference (FTC), San Francisco, CA, USA, 6–7 December 2016. A PowerPoint Presentation by Paul E. Tippens, Professor of Physics Southern Polytechnic State University, 2017. Technical Paper. Available online: https://www.stcharlesprep.org/01_parents/vandermeer_s/ Useful%20Links/Honors%20Physics/pdf%20lectures/Acceleration.pdf (accessed on 29 June 2017). Dadafshar, M. Accelerometer and Gyroscopes Sensors: Operation, Sensing, and Applications; Maxim Integrated: San Jose, CA, USA, 2014. The Earth’s Gravitational Field. Techinical Paper. Available online: http://www-gpsg.mit.edu/12.201_12. 501/BOOK/chapter2.pdf (accessed on 29 June 2017). Takeda, R.; Tadano, S.; Todoh, M.; Morikawa, M.; Nakayasu, M.; Yoshinari, S. Gait analysis using gravitational acceleration measured by wearable sensors. J. Biomech. 2009, 42, 223–233. [CrossRef] [PubMed] Liu, M. A study of mobile sensing using smartphones. Int. J. Distrib. Sens. Netw. 2013. [CrossRef] Furutani, K.; Kato, M.; Tsuru, T. Low-Pass Filter. U.S. Patent 5,668,511, 16 September 1997. Welch, G.; Bishop, G. An Introduction to the Kalman Filter; University of North Carolina: Chapel Hill, NC, USA, 1995. LG Urbane. Available online: http://www.gsmarena.com/lg_watch_urbane_w150-7680.php (accessed on 29 June 2017). Samsung Galaxy S6. Available online: http://www.gsmarena.com/samsung_galaxy_s6-6849.php (accessed on 29 June 2017). Hardyck, C.; Petrinovich, L.F. Left-handedness. Psychol. Bull. 1977, 84, 385. [CrossRef] [PubMed] Watch Handedness. Available online: https://en.wikipedia.org/wiki/Watch#Handedness (accessed on 29 June 2017). Is It Acceptable to Wear a Watch on the Right Wrist? Available online: http://www.askandyaboutclothes. com/forum/showthread.php?116570-Is-it-acceptable-to-wear-a-watch-on-the-right-wrist (accessed on 29 June 2017). Why Wear a Watch on the Wrist Where You’re Hand Dominant? Available online: http://www.reddit. com/r/Watches/comments/1wzub5/question_why_wear_a_watch_on_the_wrist_where/ (accessed on 29 June 2017). Annett, M. Handedness and Brain Asymmetry: The Right Shift Theory; Psychology Press: Leicester, UK, 2002. Yan, S.; Soh, P.J.; Vandenbosch, G.A.E. Wearable dual-band composite right/left-handed waveguide textile antenna for WLAN applications. Electron. Lett. 2014, 50, 424–426. [CrossRef]
Symmetry 2017, 9, 101
21. 22. 23. 24.
25.
26. 27.
28.
29.
30. 31.
32.
33.
34.
35. 36.
37.
21 of 21
Why Don’t Apple Have Left Handed Watches? Available online: https://www.iphonetricks.org/5-reasonsto-wear-the-apple-watch-on-your-left-hand/ (accessed on 29 June 2017). Android Wear Left Handed Mode? Available online: https://www.reddit.com/r/AndroidWear/comments/ 3f77fs/left_handed_mode/ (accessed on 29 June 2017). How to Set Up the Apple Watch for Left-Handed Use. Available online: http://www.imore.com/how-setapple-watch-left-handed-use (accessed on 29 June 2017). Goutte, C.; Gaussier, E. A probabilistic interpretation of precision, recall and F-score, with implication for evaluation. In Proceedings of the European Conference on Information Retrieval, Santiago de Compostela, Spain, 21–23 March 2005. Nahapetian, A. Side-Channel Attacks on Mobile and Wearable Systems. In Proceedings of the 13th IEEE Annual Consumer Communications and Networking Conference (CCNC), Las Vegas, NV, USA, 9–12 January 2016. Spreitzer, R.; Moonsamy, V.; Korak, T.; Mangard, S. Systematic Classification of Side-Channel Attacks: A Case Study for Mobile Devices. arXiv 2010, arXiv:1611.03748. Chu, H.; Raman, V.; Shen, J.; Kansal, A.; Bahl, V.; Choudhury, R.R. I am a smartphone and I know my user is driving. In Proceedings of the 2014 Sixth International Conference on Communication Systems and Networks (COMSNETS), Bangalore, India, 6–10 January 2014. Cai, L.; Machiraju, S.; Chen, H. Defending against sensor-sniffing attacks on mobile phones. In Proceedings of the 1st ACM Workshop on Networking, Systems, and Applications for Mobile Handhelds, Barcelona, Spain, 17 August 2009. Cai, L.; Chen, H. TouchLogger: Inferring Keystrokes on Touch Screen from Smartphone Motion. In Proceedings of the 6th USENIX conference on Hot topics in security, San Francisco, CA, USA, 8–12 August 2011. Liang, C.; Chen, H. On the practicality of motion based keystroke inference attack. In Proceedings of the International Conference on Trust and Trustworthy Computing, Vienna, Austria, 13–15 June 2012. Xu, Z.; Bai, K.; Zhu, S. Taplogger: Inferring user inputs on smartphone touchscreens using on-board motion sensors. In Proceedings of the fifth ACM conference on Security and Privacy in Wireless and Mobile Networks, Tucson, AZ, USA, 16–18 April 2012. Owusu, E.; Han, J.; Das, S.; Perrig, A.; Zhang, J. ACCessory: Password inference using accelerometers on smartphones. In Proceedings of the Twelfth Workshop on Mobile Computing Systems & Applications, San Diego, CA, USA, 28–29 February 2012. Wang, H.; Lai, T.T.-T.; Choudhury, R.R. Mole: Motion leaks through smartwatch sensors. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking, Paris, France, 7–11 September 2015. Sarkisyan, A.; Debbiny, R.; Nahapetian, A. WristSnoop: Smartphone PINs prediction using smartwatch motion sensors. In Proceedings of the 2015 IEEE International Workshop on Information Forensics and Security (WIFS), Rome, Italy, 16–19 November 2015. Beltramelli, T.; Risi, S. Deep-Spying: Spying Using Smartwatch and Deep Learning. Master’s Thesis, IT University of Copenhagen, Copenhagen, Denmark, December 2015. Liu, X.; Zhou, Z.; Diao, W.; Li, Z.; Zhang, K. When good becomes evil: Keystroke inference with smartwatch. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, Denver, CO, USA, 12–16 October 2015. SANS Technology Institute. Password policy. Available online: https://www.sans.edu/student-files/ projects/password-policy-updated.pdf (accessed on 29 June 2017). © 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).