KINECT Sensor Gesture and Activity Recognition for

JOURNAL OF LATEX CLASS FILES, VOL. 13, NO. 9, SEPTEMBER 2014

1

KINECT Sensor Gesture and Activity Recognition for Consumer Cognitive Systems Marina Gavrilova, IEEE SM, Yingxu Wang, IEEE SM, Faisal Ahmed and Padma Polash Paul Abstract— Cognitive consumer electronics is the fast growing sector worldwide driven by machine intelligence and cognitive systems. It triggered and enabled by audio and video capturing devices, smart sensors, health and fitness monitoring devices, security and education electronics, and intelligent systems. Smart consumer sensors and cognitive systems can be synergized through Internet of Things (IoT), for optimal information sharing, communication, real-time updates, data analytics and enhanced support for decision-making. Biometric based devices, originally intended for large-scale applications in airports, border controls, disaster zones or refugee migration zones are enabling a wide range of applications in commercial and consumer sectors, as standalone systems or as part of interconnected sensor networks. This article introduces the applications of Microsoft KINECT in cognitive systems towards smart consumer electronics. A human behavior cognition technology is presented for gesture and activity recognitions using KINECT sensors. As a novel frontend of pervasive cognitive systems, challenges and applications of KINECT sensorbased system will be explored in consumer electronics including smart automobiles, health care, surveillance and activity recognition. I.

OVERVIEW OF SMART CONSUMER ELECTRONICS

Consumer electronics domain is one of the highest growing markets in North America and around the globe. The spheres covered by consumer electronics are broad, encapsulating almost all human activities related to work or leisure. Some of consumer electronics include capturing devices, such as photo and video cameras that now are increasingly integrated with communication electronics, such as mobile phones. Another rapidly developing field includes household appliances, such as smart TVs, smart fridges, smart humidifiers, and smart ovens, that allow to keep track of prescriptions, inform an owner when grocery trip is warranted, or ensuring that the given temperature and humidity setting are at a desired level through the day. A secure and reliable design of smart cars of the future can now be efficiently tested and validated in cyberworlds [18]. Another important application is healthoriented devices such as heart rate monitors, calorie burning devices, fitness gadgets (i.e. Fitbit) and stress-detecting devices, that allow to monitor a stress level and help to achieve short and long term health goals [4]. Home security applications, including cameras, alarms, and smart locks can

now include fingerprints and face recognition technologies in addition to traditional code combination. Education-oriented portable devices, including laptops, tablets, electronic books, audio and cd-players, and child developmental e-toys are popular items found in almost any household. Smart sensors for measuring water quality, air pollution, wind speed, traffic congestion, or number of bikes travelling through intersection are used by city officials, environmental agencies and civil services alike. All of those consumer electronics and sensors can be interlinked through Internet of Things (IoT), for ease of information sharing, communication, real-time updates, data analytics and enhanced decision-making capabilities, powered by the combination of machine learning and artificial intelligence [11]. Biometric-based devices, originally intended for large-scale applications in airports, at border crossings, in disaster zones or refugee migration centers, are finding more and more support in both commercial and consumer sectors. They can be used as standalone systems or as part of interconnected sensor network. Secure banking, car and social security registries, office building access, recreation facility such as swimming pools or amusement parks, mobile fund transfer, and even cloud data storage are increasingly include biometric security [26]. And finally, perhaps one of the most advanced areas of consumer electronics, with millions of users worldwide, is the game consoles. According to BBC news, in 2011 Microsoft KINECT was the fastest selling device on record, with its sales reaching 23 million units in 2013. One of the unique features of this device is that it is equally suitable for consumer market as well as for research intensive computing. The data acquisition is cheap and can be stored in a form ready for postprocessing. The sensors is of a high-resolution, and allow both image and video to be captured in a variety of settings, i.e. gray-scale, color, infrared, and depth. Multiple users can be tracked using a single device, with easily adjustable point of view. APIs are available to process data and conduct more thorough pattern analysis. Gait and gesture data can be easily acquired [35]. This articles is thus set to examine the vast variety of modern applications of one of the most popular consumer devices:


2

Microsoft KINECT, both from consumer and scientific point of view.

Household

Security

Data Capturing

Education

IoT Consumer Electronics

Communication

Fitness

Medical

Fig. 1. Smart Consumer Electronics Market Spheres.

II.

EXPLORING NOVEL APPLICATIONS OF MIROSOFT KINECT SENSORS

Aside from an obvious entertainment value, such device as Microsoft KINECT has a range of very interesting features and design elements that makes it more and more popular with consumers, engineers and researches alike. First of all, it a very cheap sensor with a wide range of data capturing capabilities, that can obtain still images, record a sequence of frames and take a continuous video. The data can be represented in multiple formats, including infrared, grey-scale, color images and video. It also can extract key properties of a human body, such as general skeleton topology and coordinates of various joints, making it a convenient tool for fast processing. It is sensitive to even small changes in the positon of the camera, can perform a real-time tracing, and can even recognize positions of individual fingers which make it suitable for gesture and activity recognition. User can be positioned at a various distance from the KINECT camera, at different degree angles, and move with different speed or remain motionless. Camera can even be moved behind the user, to capture more data. Moreover, features of multiple users can be tracked and recorded simultaneously. The set of ready to use APIs to promote development of custom applications is available from Microsoft. Applications of KINECT sensor can be found in home monitoring, health care, kinesiology, theater and arts, virtual reality and biometric security.

III.

STATE-OF-THE-ART OF KINECT-BASED RESEARCH

The features above made Microsoft KINECT not only one of the most popular consumer electronic devices, but also a thought-after sensor that can be found in many kinesiology and biometric security laboratories worldwide. Some wellknown research centers doing research on KINECT based human recognition include Prof. Nadia Magnenat-Thalmann Institute for Media Innovation (IMI), that recently published a groundbreaking research on actor’s emotion recognition using KINECT sensors [31, as well as PURDUE University, Microsoft Research, NTU Singapore, and Biometric Research Center at Hong Kong Polytechnic University. In Canada, the Biometric Technology laboratory at the University of Calgary, or BTLab, focuses its research on KINECT-based gait recognition [8], KINECT-based action recognition [9], and estimating user degree of involvement in a collaborative environment [7]. In addition, other biometric modalities related to KINECT sensor are being studied. For example, a new method was introduced in BTLab to estimate the Kinectbased facial recognition quality [38]. Recently, a multi-modal biometric system design based on KINECT gait and EEG brain wave signals, has being developed [35]. The main advantage of multi-modal system is that it can significantly increase the recognition rate by combining features from two or more different biometric modalities. In this system KINECT-base face and EEG features were combined for the first time [35]. It is practically impossible to discuss a multi-modal system architecture without mentioning one of the most powerful theories based on intelligent cognitive design. The work in cognitive domain was pioneered by Lofti Zadeh, a highly-regarded founder of fuzzy logic. It was continued at the University of Calgary, where Prof Yingxu Wang leads an innovative research building cognitive models for knowledge representation. The next sections provide a more detailed overview of some of the popular approaches to KINECT-based gait and action recognition, as well as cognitive biometric research.

Fig 2. KINECT sensor captured images: Infrared, Depth stream (Gray), Depth stream (Color) and 3D Skeleton (left to right).

IV.

BIOMETRIC GAIT AND ACTIVITY RECOGNITION AND COGNITIIVE SYSTEMS

One of the first clustering methods for Kinect-based gait recognition was developed in 2012 [1]. In the same year, a geometry-based approach that uses an ensemble of biometric features, such as human height, arm length, step-length and speed was developed [16]. The methods could achieve


relatively good recognition rate of around 70-80%. In 2015, a first feature fusion based architecture was proposed for gait recognition in BT Lab, U of Calgary [8]. It also introduced a new topology-based set of features based on scale and view independent joint relative distance (JRD) and joint relative angle (JRA). The method used a dynamic time warping (DTW)based kernel as part of a rank-level fusion, and resulted in the performance exceeding 90% accuracy. This research has led to an observation that gait and activity-related features are dependent not only on the walking style, but also on the emotional and health state of a person. Factors that may affect gait include personal health, sport injuries, or emotional state, while contextual or external factors may include a walking surface (grass, clay, asphalt, stone), a type of clothing, or a type of accessories that a person might carry (a backpack, a briefcase, a laptop, or a coffee mug). To accommodate for such variability, adaptive contextbased biometric systems were developed [21]. It was noted by researchers that gait and activity-related features are highly sensitive to changes in clothing and carrying conditions [33]. This affects gait and activity-related biometric authentication in a dynamic environment. Thus, recent gait recognition methods introduce additional modalities to increase the robustness of the resulting system [33]. Another emerging research is based on trying to predict user behaviour from their posture. For example, during a meeting in a boardroom room setup, a system can couple the sensing seat based authentication with Kinect-based skeleton data, which can boost the recognition performance [21, 8-10]. Different types of behavioural and social context metadata can also be used to boost gait recognition. Recent research demonstrated that information pertaining to daily routines of the users such as being at a given location during a given time (i.e. being in the lecture room during class hours) and specific conditions related to the location (i.e. attending a conference) provide valuable context metadata that can be used to improve the recognition performance. This information can be treated as social behavioural gait-based biometrics, and enhance user authentication [23, 24]. Similarly, social behavioural information can be extracted from a collaborative environment “where user behaviour related to the manner of communication, situational responses, temporal patterns of user activity, preference of spatial location, and interaction among group members and project organizer can be utilized as metadata” [8]. Let us now take a closer look at gesture and activity recognition research that emerged recently.

3

parametric values are tracked over time, which is used as the gait signature representation. However, constructing the model, fitting it on the captured gait data, and estimating the parametric values are computationally expensive, which makes model-based gait recognition approaches timeconsuming. The commonly used models are based on the two spatio-temporal parameters: cadence and stride length [5]. A 3D temporal motion model to represent synchronized video sequences was proposed by [28] and then enhanced by [6]. These methods were representing individual body parts with the goal of mimicking their movements. Contrary, model-free approaches tried to come up with the general representation of gait motion through using the silhouette obtained from the video sequences [19]. One of the earlier methods appeared in 2000 was based velocity moment features that capture object and motion in image sequences [19]. Around the same time, researchers [2] proposed the motion energy image (MEI) representation of a human movement, stored as a static vector image where each point is a function of the motion attributes at the corresponding spatial location of a sequence image. A method that uses EigenGait for dimensionality reduction of pairwise correlation of silhouettes was developed in 2002 [5]. Both the motion energy image (MEI) and the EigenGait based representation of human movement preceded the development of the popular appearance-based gait recognition method, namely the gait energy image (GEI) [15]. This method combined the principal component analysis (PCA) and the multiple discriminant analysis (MDI) to even further reduce the feature dimensionality, while maintaining a high class degree of distinctiveness of features. The release of the low-cost Kinect sensor has opened new research horizons in the area of a real-time motion analysis, and resulted in renewed interest in gait, gesture and activity recognition using Kinect. In addition to different data streams that can be obtained from Kinect, one of the highly useful features is that it can also construct a 3D virtual skeleton from a human body and track it in real-time [17]. This feature makes some of the time-consuming pre-processing tasks, such as a background segmentation, or a silhouette computation, obsolete. As a result, many recent gait recognition methods utilize the computationally-inexpensive real-time depth sensing and skeleton tracking data in the gait recognition research [8-10]. B. Vision-Based Action Recognition

A. Biometric Gait Recognition Depending on the type of features being used, video-based gait recognition methods found in literature can be divided into two categories: model-based approaches and model-free approaches [14,15]. In the model-based approaches, movements of different body parts are modeled precisely based on a set of parameters [19]. The variations in the

Vision-based action recognition methods found in literature focus on two primary issues: finding appearance-based global or local features or spatio-temporal interest points, and modeling actions based on these spatio-temporal features [Yamato]. In a video-based action recognition, appearancebased motion templates are the most commonly-used features. One of the appearance-based features include optical flow measurements-based motion representation was


introduced in 2003 [3]. A highly interesting research on spatiotemporal interest points utilized Harris and Forstner operators to isolate important local variations in a space-time domain [12]. Here, researchers proposed to use the spatio-temporal bag-of-features for realistic action recognition from the movie clips [12]. Where single camera-based action recognition systems focus on extracting appearance-based motion representations and spatio-temporal interest points, multi-camera motion capture (MOCAP) systems construct virtual skeletons and extract 3D joint locations to yield effective model-based motion descriptors. However, such motion capture systems are often dependent on wearable markers and typically are expensive to deploy. One of the studies utilized synthetic poses extracted from multiple viewpoint MOCAP to construct a conditional random field motion representation [25]. Another work [10] introduced a view-independent action recognition system that utilizes a combination of trajectory and image-based similarity descriptors. Although the tracked skeletal joints obtained from the Kinect may be more noisy than the traditional MOCAP data, the computationally-inexpensive nature of the Kinect realtime skeleton tracking has contributed to its high popularity for both gait and action recognition. In 2012, researchers [20] proposed to use histograms of 3D joints as a convenient posture representations for 10 different actions recognition using Kinect. The set of posture-based visual words was trained using a Hidden Markov Model (HMM) on a database of 200 sequences for each of the different activities. In 2014, scientists [30] proposed a motion representation based on EigenJoints, which combines static posture and dynamic motion properties from frames selected based on accumulated motion energy method (AME) with a nonparametric Naive-Bayes Nearest-Neighbor (NBNN) classification method. A sparse representation based on classification of human action models, and utilizing the skeletal joint data classified by k-NN and support vector machine (SVM), was conducted in 2014 by [13]. Two novel work on action recognition from the KINECT were presented very recently at CASA 2016 [9,29]. A new action recognition method that uses joint-triplet motion image and local binary pattern for 3d action recognition and outperforms existing methods for KINECT action recognition was introduced in [9]. Another novel work looked at human emotion classification based on the professional actor recorded video sequences [29]. Finally, research on the integration of KINECT with other behavioral biometric modalities such as face and EEG) has recently emerged at the forefront of multi-modal biometrics [39].

V.

COGNITIVE SYSTEMS DESIGN

4

Cognitive systems are brain-inspired systems underpinned by cognitive informatics theories, cognitive computing technologies, and denotational mathematics. Cognitive Informatics (CI) is initiated by Y. Wang and his colleagues [32] and is defined as “a transdisciplinary enquiry of computer science, information sciences, cognitive science, and intelligence science that investigates into the internal information processing mechanisms and processes of the brain and natural intelligence, as well as their engineering applications in cognitive computing (CC) and cognitive robotics (CR)” [32-34]. In CI, Denotational Mathematics (DM) and the Layered Reference Model of the Brain (LRMB) [34] provide an integrated framework for modeling the brain and the mind. CI is a field of research that synergizes computer science, information science, brain science, neuroscience, intelligent computing, knowledge science, robotics, cognitive linguistics, philosophy, and engineering technologies. The basic studies in CI have led to Cognitive Computing as “a paradigm of intelligent computing platforms of cognitive methodologies and systems based on CI, which embodying computational intelligence by cognitive and autonomous systems mimicking the mechanisms of the brain [34]”. Cognitive knowledge learning as a central ability of cognitive systems cannot be implemented at the neural level. Instead, it needs cognitive knowledge bases and a cognitive learning engine to replicate human decision-making process and deep learning abilities. Knowledge learning is one of the most important categories for knowledge acquisition, which can enhance and surpass typical machine learning methods in its capability for data analytics, video and image processing, and interactive game playing through KINECT type console. This approach is focused on mimicking learning mechanism of the brain and can surpass the traditional machine learning approaches, embodied at the low neural level, to reach deep knowledge acquisition and thus support reinforced learning. Combining current KINECTbased methods for gait and activity recognition with the power of cognitive knowledge learning can assist in the development of the new generation biometric sensor based cognitive systems that can understand not only the gestures, but also the emotion status of a user, his health condition, recent injuries, and potentially his mental state. This can be useful in a variety of applications, from virtual game designs, to border security, from medical rehabilitation to new ways to educate elementary school students.


5

An important cognitive system is gait, gesture and action recognition based on a fundamentally different approach compared to traditional motion analysis techniques. The proposed framework revolves around the construction of a texture image representation based on joint relative movements, thus switching from spatio-temporal data analysis to a more familiar domain of spatial image analysis, by capturing the entirety of a motion sequence in a holistic spatial representation. The proposed method then utilizes a cognitive

learning engine, based on the convolutional neural network (CNN) to extract effective features from these spatial texturebased representations of skeletal motion. Figure 4 shows an overview of the proposed gait and action recognition method, which comprises multi-stage processing of the 3D full-body skeleton data obtained from the Kinect or any other MOCAP sensor. The detection of skeletal motion inside the environment being monitored triggers the computation of the joint relative angle (JRA) features for different joint-triplets.

Fig. 3. A flowchart representation of biometric template built from person images/videos/audios captured by sensors. The feature vector can be used for gait, gesture, activity and even emotion recognition through process of classification, training and matching against database stored templates (adopted from [35]).

Training Action Sequence Captured Using Kinect Training Database

Skeletal Motion and Gait Cycle Detection

CNN Features

Joint Relative Angle Computation

Convolutional Neural Network Feature Extraction Enrollment

JTMI Texture Image Construction

Support Vector Machine Classifier

Recognize Gait/Action

Skeletal Motion Sequence Unknown Action Sequence

Joint-Triplet Relevance Analysis

Convolutional Neural Network Feature Extraction

Joint-Triplet Relevance Analysis

JTMI Image Construction

Recognition

Fig. 4. A framework of the gait and action recognition system based on KINECT walking sequences (adopted from [35])


The main advantage of JRA-based feature representation is that it is invariant against view and scale changes. As a result, the proposed method does not require a person to perform an activity in a fixed fronto-normal or fronto-parallel view or always maintain a certain distance from the camera. Next, in order to assess the relevance of a particular JRA sequence in motion representation, a flatness measure is utilized to evaluate the corresponding joint-triplet based on their level of engagement in the particular action-specific motion. Only the most relevant joint-triplets are considered in the next step, which involves construction of the joint triplet motion image (JTMI) - a compact holistic texture representation of the overall skeletal motion. Lastly, these texture images are passed to a convolutional neural network to extract effective and discriminating image features, which are then learned using a support vector machine (SVM). The methodology based on this novel paradigm was discussed on a conceptual level in [36], with the list of future research directions and open problems identified in [35]. VI. CONCLUSIONS AND OPEN PROBLEMS

Biometric based devices, originally intended for large-scale applications in airports, at border controls, in disaster zones or refugee migration cites, are gaining more and more applications in commercial and consumer sectors, as standalone systems or as part of interconnected sensor networks. This article has discussed a set of novel applications of the Microsoft KINECT sensor from both consumer and research point of views. It presented an overview of the stateof-the-art methodology for gesture and activity recognition using KINECT sensor, and discusses applications of KINECT sensors in home monitoring, health care, biometric surveillance and emotion recognition. It also discussed advantages of cognitive architectures in combination with new texture based approach to gait and activity recognition. Open problems include further studies in the domain of context-based gesture recognition research, ensuring data privacy during communication between sensors. The challenges for understanding how emotional state of a person affects one’s behavior, habits and everyday activities, in gesture or action identification will be addressed. Further studies will pave the way to the development of a new generation of cognitive gesture and action recognition systems, demanded in intelligent automobiles, smart homes, medical care facilities, professional training devices, and smart consumer electronics.

6

ABOUT THE AUTHORS

Dr. Marina L. Gavrilova ([email protected]) is Professor of biometric security and pattern recognition at the University of Calgary, and Co-Director of the Biometric Technologies Laboratory and the SPARCS Laboratory. She published over 200 journal and conference papers, edited a number of special issues and wrote three books. She is a founding Editor-in-Chief of the Springer Transactions on Computational Sciences Journal, on editorial boards of the Visual Computer, Journal of Biometrics, Journal of Supercomputing, and other journals. Dr. Yingxu Wang ([email protected]) is Professor of cognitive informatics, brain science, software science, and denotational mathematics. He is the founding President of International Institute of Cognitive Informatics and Cognitive Computing (CICC), a Fellow of ICIC, a Fellow of WIF (UK), and a Senior Member of IEEE and ACM. He is the founder of the IEEE International Conference on Cognitive Informatics and Cognitive Computing. He is founding Editor-in-Chiefs of International Journal of Cognitive Informatics & Natural Intelligence; International Journal of Software Science & Computational Intelligence; Journal of Advanced Mathematics & Applications; Journal of Mathematical & Computational Methods, and Associate Editor of IEEE Trans on SMC: Systems. Dr. Padma P. Paul ([email protected]) has received his Ph.D. in Computer Science from the University of Calgary in 2015. He is currently a Post-Doctoral Fellow in Oxford Computational Neuroscience lab at the University of Oxford and a Fellow at the Brain Science Foundation, USA. His research interest includes Brain Big Data Mining, Big Data Analytics, Computational Neuroscience, Machine Learning and Pattern Recognition. He has co-authored over 70 publications. Faisal Ahmed ([email protected]) has received his BSc from the Islamic University of Technology, Bangladesh in 2010 and his MSc from the University of Calgary in 2016. His research interests include biometric data processing, video processing and multi-modal fusion. He has published over 30 journal and conference articles. He held position of lecturer and is currently working as a Data Scientist in Canada. ACKNOWLEDGEMENT

Authors acknowledge NSERC Discovery Grant RT731064, as well as MITACS Accelerate grant for partial support of this project.


REFERENCES [1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

A. Ball, D. Rye, F. Ramos, and M. Velonaki, “Unsupervised clustering of people from ’skeleton’ data,” ACM/IEEE International Conference on Human Robot Interaction, pp. 225–226, 2012. A. Bobick and J. Davis, “The recognition of human movement using temporal templates,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 3, pp. 257–267, 2001. A. Efros, A. Berg, G. Mori, and J. Malik, “Recognizing action at a distance,” IEEE International Conference on Computer Vision, vol. 2, pp. 726–733, 2003. S. Greene, H. Thapliyal, A. Caban-Holt, “A Survey of Affective Computing for Stress Detection: Evaluating technologies in stress detection for better health,” IEEE Consumer Electronics Magazine 5(4), pp. 4456, 2016. C. BenAbdelkader, R. Cutler, and L. Davis, “Stride and cadence as a biometric in automatic person identification and verification,” in IEEE International Conference on Automatic Face and Gesture Recognition, pp. 372–377, 2002. C. Yam, M. Nixon, and J. Carter, “Automated person recognition by walking and running via model-based approaches,” Pattern Recognition, vol. 37, pp. 1057– 1072, 2004. F. Ahmed and M. Gavrilova “Biometric-Based User Authentication and Activity Level Detection in a Collaborative Environment,” Transparency in Social Media, pp. 166-179, 2015. F. Ahmed, P. Paul, and M. Gavrilova, “DTW-based kernel and rank level fusion for 3d gait recognition using kinect,” The Visual Computer, vol. 31, no. 6-8, pp. 915–924, 2015. F. Ahmed, P. Paul, and M. Gavrilova, “Joint-triplet motion image and local binary pattern for 3d action recognition using Kinect,” Computer Animation and Social Agents, pp. 111–119, 2016. I. Junejo, E. Dexter, I. Laptev, and P. Perez, “Viewindependent action recognition from temporal selfsimilarities,” IEEE Transactions on Pattern Recognition and Machine Intelligence, vol. 33, no. 1, pp. 172–185, 2011. J. Lemley; S. Bazrafkan, P. Corcoran "Deep Learning for Consumer Devices and Services: Pushing the limits for machine learning, artificial intelligence, and computer vision”, IEEE Consumer Electronics Magazine, pp 48-56, 2017. I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld, “Learning realistic human actions from movies,” in IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8, 2008. I. Theodorakopoulos, D. Kastaniotis, G. Economou, and S. Fotopoulos, “Pose-based human action recognition via sparse representation in dissimilarity space,” VCIR, 25(1): 12–23, 2014. J. Han and B. Bhanu, “Individual recognition using gait energy image,” IEEE TPAMI, vol. 28, pp. 316–322, 2006.

7

[15]

[16]

[17]

[18]

[19]

[20]

[21]

[22]

[23]

[24]

[25]

[26]

[27]

[28]

[29]

J. Han and B. Bhanu, “Statistical feature fusion for gaitbased human recognition,” in IEEE Conf. on Computer Vision and Pattern Recognition, vol. 2, pp. 842–847, 2004. J. Preis, M. Kessel, C. Linnhoff-Popien, and M. Werner, “Gait recognition with Kinect,” in Workshop on Kinect in Pervasive Computing, 8 pages, 2012. J. Shotton “Real-time human pose recognition in parts from single depth image,” in IEEE Conf. on Computer Vision and Pattern Recognition, pp. 1297–1304, 2011. A. Munir “Safety Assessment and Design of Dependable Cybercars: For today and the future,” IEEE Consumer Electronics Magazine, pp 69-77, 2017. J. Wang, M. She, S. Nahavandi, and A. Kouzani, “A review of visionbased gait recognition methods for human identification,” in IEEE International Conference on Digital Image Computing: Techniques and Application, pp. 320– 327, 2010. L. Xia, C. Chen, and J. Aggarwal, “View invariant human action recognition using histograms of 3d joints,” in IEEE Conf. on Computer Vision and Pattern Recognition, pp. 20–27, 2012. M. Gavrilova and M. Monwar Multimodal Biometrics and Intelligent Image Processing for Security Systems, IGI, 2013. M. Gavrilova and R. Yampolskiy “Applying biometric principles to avatar recognition,” Transactions on Computational Sciences XII, pp.140-158, 2011. M. Sultana, P.P. Paul, and M. Gavrilova “Person Recognition from Social Behavior in Computer Mediated Social Context”, IEEE Transactions on Human Machine Systems, Early Access Print and Online, March 2017 M. Sultana, P.P. Paul, M. Gavrilova “Social Behavioral Information Fusion in Multimodal Biometrics”, IEEE Transactions SMC: Systems, Early Access Print ISSN: 21682216, Online ISSN: 2168-2232, April 2017 P. Natarajan and R. Nevatia, “View and scale invariant action recognition using multiview shape-flow models,” in IEEE Conf on Computer Vision and Pattern Recognition, 2008, pp. 1–8, 2008. A. Majumder; J. Goswami; S. Ghosh; R. Shrivastawa. S.P. Mohanty; B.K. Bhattacharyya "Pay-Cloak: A Biometric Back Cover for Smartphones: Facilitating secure contactless payments and identity virtualization at low cost to end users’, IEEE Consumer Electronics Magazine, pp. 78-88, 2017 P.P. Paul, M. Gavrilova and R. Alhajj “Decision Fusion for Multimodal Biometrics Using Social Network Analysis Systems,” IEEE Transactions on SMC: Systems, vol. 44, no 11, pp. 522 – 1533, 2014. R. Urtasun and P. Fua, “3d tracking for gait characterization and recognition,” Autom. Face and Gesture Rec., pp. 17–22, 2004. S. Senecal, L Cuel, A. Aristidon, N. Magnenat-Thalmann “Continuous body emotion recognition system during theater performances”, Computer Animation and Virtual Worlds, vol 27, issue 3-4, pp 311-321, 2016


[30]

[31]

[32]

[33]

[34]

[35]

X. Yang and Y. Tian, “Effective 3d action recognition using eigenjoints,” Visual Comm. and Image Repr. 25 (1):. 2–11, 2014. Y. Tian, Y. Wang, M. Gavrilova, and G. Ruhe “A formal knowledge representation system (FKRS) for the intelligent knowledge base of a cognitive learning engine”, IJCINI vol 3, pp 1-17,2011 Y. Wang, B. Widrow, L.A. Zadeh, N. Howard, S. Wood, V.C. Bhavsar, G. Budin, C. Chan, M.L. Gavrilova, D.F. Shell, “Cognitive Intelligence: Deep Learning, Thinking, and Reasoning with Brain-Inspired Systems”, IJCINI, 10(4), 121, 2016 Y. Wang, E.T. Rolls, N. Howard, V. Raskin, W. Kinsner, F. Murtagh, V. C. Bhavsar, S. Patel, D. Patel, and D. F. Shell, “Cognitive Informatics: From Information Revolution to Intelligence Revolution”, IJCINI, 7(2), 52-71, 2015. Y. Wang, S. Patel, and D. Patel, “A Layered Reference Model of the Brain (LRMB)”, IEEE Transactions on Systems, Man, and Cybernetics (Part C), 36(2), March, 124-133, 2006 M. Gavrilova, F. Ahmed, S. Azam, P. Paul, M. Sultana, W. Rahman, F. Zohra, “Emerging Trends in Security System Design using the Concept of Social Behavioral Biometrics“ Book Chapter 10, Vol. 691, pp. 229-252, Information Fusion for Cyber-Security Analytics, 2016

8