Segmenting Iris Images in the Visible Spectrum with ... - MSU CSE

Pattern Recognition Letters 57 (2015) 4–16

Contents lists available at ScienceDirect

Pattern Recognition Letters journal homepage: www.elsevier.com/locate/patrec

Segmenting iris images in the visible spectrum with applications in mobile biometrics ✩ Raghavender Reddy Jillela a,1 , Arun Ross b,∗ a b

Digital Signal Corporation, Chantilly, VA, USA Michigan State University, East Lansing, MI, USA

a r t i c l e

i n f o

Article history: Received 25 July 2014 Available online 31 October 2014 Keywords: Iris Segmentation Mobile Biometrics Visible

a b s t r a c t The widespread use of mobile devices with Internet connectivity has resulted in the storage and transmission of sensitive data. This has heightened the need to perform reliable user authentication on mobile devices in order to prevent an adversary from accessing such data. Biometrics, the science of recognizing individuals based on their biological and behavioral traits, has the potential to be leveraged for this purpose. In this work, we briefly discuss the suitability of using the iris texture for biometric recognition in mobile devices. One of the critical components of an iris recognition system is the segmentation module which separates the iris from other ocular attributes. Since current mobile devices acquire color images of an object, we conduct a literature review for performing automated iris segmentation in the visible spectrum. The goal is to convey the possibility of successfully incorporating iris recognition in mobile devices. © 2014 Elsevier B.V. All rights reserved.

1. Introduction The degree of electronic connectivity in our society has increased tremendously over the past few years. According to the global statistics report compiled by the International Telecommunications Union, 2.7 billion people (i.e., almost 40% of the world’s population) are currently connected through the Internet [1]. Such a staggering level of connectivity has been possible within just 25 years of the invention of the World Wide Web. An important factor driving these statistics is the growth of communication via mobile devices. Market research agencies indicate that 1.7 billion people around the world currently access the Internet via mobile devices [2]. In 2016, this number is projected to be close to 2.5 billion. As the advancements in communication technology become more accessible and affordable, mobile device connectivity is expected to rapidly increase. Examples of mobile devices include smartphones, tablet PCs, digital image and video recording devices, Personal Digital Assistants (PDAs), handheld and wearable computers, portable media, and game consoles (see Fig. 1). However, in this article we focus primarily on smartphones. This explosion in the use of mobile devices for conducting private communication, financial transactions and business negotiations has ✩

This paper has been recommended for acceptance by G. Sanniti di Baja. Corresponding author. Fax: +1-517-432-1061. E-mail addresses: [email protected] (R.R. Jillela), [email protected] (A. Ross). 1 This work was performed when Raghavender Jillela was with West Virginia University, WV, USA ∗

http://dx.doi.org/10.1016/j.patrec.2014.09.014 0167-8655/© 2014 Elsevier B.V. All rights reserved.

heightened the need for reliably authenticating users accessing these devices. The authentication process is expected to be fast, automated, and preferably carried out remotely. Recent literature has explored the possibility of utilizing biometrics to authenticate mobile users. Biometrics is the science of recognizing individuals based on their physical or behavioral traits [3,4]. These traits include face [5], fingerprints [6], iris [7–10], periocular region [11], gait [12,13], voice, etc. While biometric solutions have been incorporated into a number of border security and access control applications, their use in mobile devices has only recently gained attention. The term mobile biometrics is now being used to refer to the process of utilizing biometric traits for authenticating users of mobile devices.2 The mobility aspect of these devices refers to their high degree of portability and their ability to provide Internet connectivity on-the-move (e.g., cellular service, Wi-Fi, etc.). A vast majority of mobile devices are equipped with sophisticated audio, image, and video recording capabilities. Furthermore, due to the reduced size and cost of processing units and memory devices, additional sensors (e.g., accelerometer, fingerprint sensor, etc.) can be easily embedded into the mobile devices. This allows for the fast and reliable acquisition of various biometric traits for personal authentication. Examples of various biometric traits that can be acquired using mobile devices are shown in Fig. 2. The past two decades have witnessed a significant growth in biometric technology (i.e., sensors, devices, algorithms, applications).

2 In some instances, the term has also been used to refer to compact and portable standalone biometric devices (e.g., EyeLock Myris, Iritech IriShield, etc.).

R. R. Jillela, A. Ross / Pattern Recognition Letters 57 (2015) 4–16

5

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Fig. 1. Examples of mobile devices capable of capturing biometric data: (a) smartphones, (b) tablet computers, (c) digital cameras, (d) digital video recorders, (e) wearable activity trackers (e.g., Fitbit), (f) wearable head-mount displays (e.g., Google Glass), (g) 3D sensors (e.g., [14]), and (h) portable game consoles (e.g., XBOX Kinect).

4.

(a)

(b)

(c)

5.

6.

7.

(d)

(e)

(f)

Fig. 2. Examples of biometric traits that can be acquired using mobile devices: (a) face, (b) fingerprints, (c) iris, (d) handwriting, (e) voice, and (f) 3D structure, respectively.

However, a majority of this growth has been observed in the government (e.g. border crossing, law enforcement, national ID), forensics (e.g., criminal investigations, corpse identification), and commercial (e.g., access control) domains. In comparison, the usage of biometric technology in consumer electronics (e.g., mobile devices, computer login, etc.) has been limited. With the advent of mobile devices, this trend is likely to change in the near future. The Apple iPhone Touch ID and Google Nexus Face Unlock already utilize fingerprint and face recognition technologies, respectively, for personal authentication. Although still in the developmental stage, mobile biometric technology can potentially offer several advantages: 1. Portability: Owing to their compact footprint and high degree of portability, mobile devices are typically carried in person by their owners to different locations. 2. Low operational cost: The cost of performing biometric authentication on these devices is minimal due to decreasing size and increasing computational power of the processing units. 3. Ease of multi-biometric data acquisition: A number of mobile devices are equipped with sensors that can acquire multi-biometric data (e.g., high resolution cameras for face and iris recogni-

tion, voice recorders for speaker recognition, etc.). Fusion of biometric information can provide highly reliable authentication performance. Multi-factor authentication: Mobile devices lend themselves to multi-factor authentication since the biometric data can be used in conjunction with graphical passwords (through touch screen) and geospatial data (from GPS) to harden the authentication process. Market penetration: Due to the popularity and widespread use of mobile devices, they are better positioned to penetrate society thereby offering a broader customer base to the biometric industry. Data privacy: Since a user’s biometric template can be encrypted and retained within the mobile device, it is relatively difficult for an adversary to steal or modify the template of a user. Remote identification: Mobile biometrics are more suitable for user verification, rather than identification. This is because of the higher computational cost of identification (1:N matching) compared to verification (1:1 matching). However, in cases where identification is required, mobile devices can also securely transfer the biometric data to a remote server (e.g., a cloud platform) to perform matching.

The focus of this paper is on the iris biometric trait. In particular, we focus on the segmentation module of the iris processing pipeline. The remainder of this paper is organized as follows: Section 2 discusses the significance of iris in the context of mobile biometric recognition. The various steps involved in an iris recognition system are outlined in Sections 3–7. Section 8 lists some of the major challenges in mobile iris recognition. A review of existing iris segmentation techniques that are suitable for mobile iris recognition is provided in Section 9. Section 10 concludes the article. 2. Mobile biometric recognition A number of physical and behavioral traits have been studied as biometric indicators (e.g., face, fingerprints, iris, voice, gait, palm prints, ocular region, etc.). Amongst these traits, face, fingerprint, iris and voice have particularly received extensive coverage in the published literature. The potential of each of these four modalities in the context of mobile biometrics is reviewed below. 1. While a face image can be easily acquired using a mobile device, the degree of variability due to changes in facial pose and ambient illumination can severely impact the recognition performance.

6

R. R. Jillela, A. Ross / Pattern Recognition Letters 57 (2015) 4–16

Further, it is difficult to acquire a good quality face image in a nighttime environment thereby restricting its use in challenging environments. In spite of these limitations, face recognition has been implemented in Apple iPhone, Samsung Galaxy, and Google Nexus smartphones. 2. Fingerprint acquisition is typically contact-based where the fingerprint of a user has to touch the sensor. Environmental factors such as humidity and temperature, and fingerprint conditions such as dry skin or scars can impact the quality of the acquired image. Due to reduced sensor size on mobile devices and the availability of fewer minutiae points on the ensuing images, the matching accuracy of a fingerprint system can degrade. In spite of these limitations, Apple has incorporated a fingerprint sensing and matching scheme in the iPhone 5S smart phone. 3. The availability of a microphone in mobile devices facilitates speaker recognition [15,16]. However, the performance of a speaker recognition system can be impacted by ambient noise and change in microphone channel characteristics across devices. Further, if the system uses a text-dependent approach where a specific passphrase is uttered, then the user can unwittingly reveal her password to people in the vicinity. 4. Iris recognition is usually based on near infrared (NIR) lighting and sensors. Smart phone cameras, on the other hand, typically operate in the visible (VIS) spectrum and, therefore, color iris images acquired using these cameras may not lend themselves to classical iris processing algorithms. This is because the texture of dark-colored irides are not easily discernible in the VIS spectrum. However, recent literature has explored the possibility of performing iris recognition using color images [17]. Several aftermarket accessories and products have been developed to facilitate reliable iris recognition on smartphones. An example of such a product is AOptix Stratus, a wraparound sleeve that facilitates NIR iris recognition on the iPhone. In general, the problem of iris recognition is understudied in the context of smartphones. Other biometric traits such as gait [18] and touch-based gestures [19–22] have also been used in a smartphone environment. Table 1 provides a list of traits that are suitable for mobile biometrics along with the factors that impede their recognition performance. The focus of this work, however, is on mobile iris recognition. 3. Iris recognition The word iris refers to the annular region within the eye, located between the cornea and the lens [23]. The main purpose of the iris is to regulate the amount of light that enters the eye, by controlling the size of the pupil. The anterior surface of the iris is divided into two regions: the pupillary, and the ciliary zone. The pupillary zone is the inner region, that is located closer to the pupil. The pupillary boundary separates pupil from the iris. The ciliary zone is the outer region, that comprises the rest of the iris. These two regions are separated by the collarette. The sclera and iris are separated by limbic boundary. The average radius of the human iris is approximately 6 mm, with an average thickness of about 0.5 mm. A sample image showing the external structure of the iris is provided in Fig. 3. The iris begins to form during the third month of gestation and develops a distinctive structure by the eighth month [24]. The Table 1 Traits suitable for biometrics in smartphones and the corresponding factors that negatively impact their performance. Biometric trait

Performance impeding factors

Face Fingerprints Voice Iris Gait

Change in pose, illumination, expression Additional sensor requirement, partial image Noise in environment, lack of privacy Pose variations, occlusion, color images Pace, footwear, surface

Fig. 3. Morphology of the iris stroma when observed in the visible spectrum.

distinctiveness is imparted by the fibrous and cellular structures such as ligaments, furrows, crypts, rings, frills, corona, collarette [25], and sometimes moles, freckles, nevi, and other macro-features [26]. The distinctiveness of the iris across individuals is due to the complexity of its texture. The first recorded usage of information related to the eye for personal identification purposes is due to Alphonse Bertillon. In his seminal work [27,28], Bertillon described the usage of eye color patterns to distinguish criminals. Subsequent research in medical literature investigated the stability and uniqueness of the iris texture [23,29–31]. Flom and Safir’s patent first described the concept of an automatic iris identification system [32]. Later, Daugman’s landmark research helped establish iris as a highly reliable biometric [25]. An automatic iris recognition system works by extracting and comparing the intricate textural information on the surface of the iris. These operations can be divided into four major steps as follows: 1. Image acquisition: The input ocular3 images (or videos) are typically acquired using a sensor of adequate resolution to capture the iris texture. The usefulness of an acquired iris image mainly depends on its quality and the spatial extent of the iris present in the captured image. Both these factors can be automatically monitored during the image acquisition stage to obtain an iris image that can be successfully processed. Most iris recognition systems require a considerable level of user cooperation. However, iris-on-the-move and iris-at-a-distance systems are becoming increasingly feasible with advances in lens optics and image processing algorithms [33– 36]. 2. Iris segmentation: Depending on the field of view of the sensor, the acquired images can potentially include other regions of the eye (e.g., the sclera, eyelashes, eyelid, eyebrow, surrounding skin regions, etc.). Therefore, the annular region between the pupillary and limbic boundaries has to be localized to perform feature extraction. Furthermore, anatomical features that usually occlude the iris texture (e.g., eyelashes and eyelids) have to be excluded from consideration. This process of automatically locating the iris boundaries and excluding the noisy regions is called as iris segmentation. A number of segmentation algorithms have been proposed in the literature [37]. This step is often considered critical, as incorrect segmentation can negatively impact iris recognition performance [38]. 3. Normalization: The size of the iris can vary significantly across images due to pupil dilation and contraction, resolution of the sensor used and the imaging distance [39]. To address such variations in size, the segmented iris is usually unwrapped and mapped to a normalized coordinate system. This normalization operation is performed by representing the segmented iris as a rectangular image, the rows of which correspond to concentric regions in the original iris image. A widely popular technique for iris normalization, Daugman’s rubber sheet model, re-maps every point in the segmented iris region to a pair of polar coordinates [40]. While 3 Most iris systems capture both the iris and the surrounding ocular region that includes eyelids, eyelashes and periocular skin texture.

R. R. Jillela, A. Ross / Pattern Recognition Letters 57 (2015) 4–16

Fig. 4. A block diagram of an iris recognition system.

most iris recognition methods rely on this normalization routine, there are certain approaches that do not require iris normalization. Such approaches are referred to as segmentation-free approaches [41]. 4. Feature extraction and matching: Feature extraction refers to the process of encoding the discriminatory information obtained from the segmented (or normalized) iris, as a feature vector. In the matching stage, the feature vector obtained from a probe image is compared with the other feature vectors in the gallery to perform recognition. Various techniques have been proposed in the literature to perform feature extraction and matching [39]. The focus of all such techniques is to reduce computational time and complexity, while improving recognition performance. The above listed steps are illustrated in Fig. 4. 4. Iris image acquisition The device used for acquiring iris images (for enrollment or authentication purposes) is typically referred to as an iris sensor (also sometimes as iris reader, iris camera, or iris scanner). There are several types of iris sensors with different optical and imaging characteristics (e.g., image resolution, focal length, and depth of field of the sensor, wavelength and configuration of the illuminating source, optics and polarizing filters used, stand-off distance4 etc.). The choice of an iris sensor is mainly dependent on the application in which the iris recognition system is deployed. Various types of iris imaging principles have been proposed in the literature [34,42–46]. Based on their portability and the convenience offered to the users, iris sensors can be divided into the following categories: 1. Fixed, or wall-mounted sensors: These sensors are widely used, and require considerable amount of user cooperation. Users are required to position themselves and/or gaze at a predefined location to acquire a good quality image. The typical standoff distance for this type of sensor varies from 5 to 24 inches. Owing to their moderate size, these sensors could be integrated with their processing units to function as a standalone iris recognition system. Examples of this type of sensors include LG IrisAccess, Panasonic BM-ET, and Oki IrisPass-M series, as shown in Fig. 5(a), (b), and (c), respectively. 2. Portable, or hand-held sensors: Mainly used in 1:1 user verification scenarios, the hand-held sensors typically offer a high degree of portability. The typical standoff distance for these sensor varies from 3 to 12 inches. Examples of such sensors include Iridian Authenticam, EyeLock Myris, and the Oki IrisPass-H series. 4

Distance between the user and the sensor.

7

Images acquired using these sensors may have to be processed separately on a different device or platform. Alternatively, there are a few portable sensors that can perform both image acquisition and recognition within the device itself, and are designed to be functional in rugged situations. However, such sensors require cooperation between two individuals (subject and the operator) in order to acquire a good quality iris image. An operator is required to align the sensor with the subject’s eye. Examples of this type of sensors include Datastrip Easy Verify and Retica MobileEyes. Sample images of portable sensors are shown in Fig. 5(d), (e), and (f). 3. Portals, or walk-through sensors: These sensors are designed for applications involving rapid iris recognition for a large volume of users (e.g., airports). Images of the iris are captured when the user passes through a portal-like structure. The illuminants are usually mounted on the walls of the portal. These systems are suitable for iris recognition in both covert and overt situations. The standoff distance for these sensors varies between 2 and 9 ft. Examples of portal sensors include Hoyos HBox, Sarnoff Iris on the Move, and AOptix InSight system, as shown in Fig. 5(g), (h), and (i), respectively. 5. Iris imaging wavelengths The performance of an iris recognition system can significantly vary depending on the wavelengths (i.e., spectral band) used to acquire the image [47]. Typically, iris images are acquired using sensors that operate in the near-infrared (NIR) spectrum. The wavelength of the illumination sources range between 700 and 900 nm. However, the possibility of performing iris recognition in the visible (VIS) [46,48–51] and Short Wave InfraRed (SWIR) [52] spectra has also been studied. More recently, Ives et al. [53] discussed the performance of iris recognition over a broad range of wavelengths between 405 and 1070 nm. The reasons why iris recognition systems typically operate in the NIR spectrum are listed below. 1. The effect of melanin, a color inducing compound, is negligible at longer wavelengths above 700 nm. Therefore, operating in the NIR spectrum ensures that the acquired image reveals information related to the texture of the iris, rather than its pigmentation. This property in turn contributes to the high recognition performance of iris imaged in the NIR spectrum. 2. The nuances of the iris texture of dark-colored irides are better observed in the NIR spectrum since longer wavelengths penetrate the stroma of the iris and reveal the intricate texture of the iris. Besides the NIR spectrum, images acquired under the visible (VIS) spectrum have also been used for iris recognition [17]. In most studies, it has been observed that NIR iris images outperform VIS iris images from a matching accuracy perspective. This is because the richness of iris texture are not always well captured in a VIS image. Iris recognition under VIS spectrum, however, is an equally important research area. This is because of the following factors: 1. Recognition at large distances: The current iris sensor and illuminator technology makes it difficult to acquire iris images from a distance using NIR sensors. Most surveillance cameras operate in the visible wavelength and can be easily adapted to acquire iris information. 2. User convenience: NIR sensors are typically expensive compared to VIS sensors. 3. Classification: Iris images acquired in the visible spectrum reveal details related to its pigmentation. Such information when combined with iris macro-features (e.g., moles, freckles, nevi, etc.) can help in iris classification [26]. 4. Periocular information: Depending on their field of view, iris imaging systems typically capture some periocular information

8

R. R. Jillela, A. Ross / Pattern Recognition Letters 57 (2015) 4–16

(a)

(b)

(d)

(g)

(c)

(e)

(h)

(f)

(i)

Fig. 5. Examples of various types of iris sensors. Top row: Wall-mounted sensors (a) LG IrisAccess, (b) Panasonic BM-ET, and (c) Oki IrisPass-M series. Middle row: Portable sensors (d) Oki IrisPass-H series (e) DataStrip Easy Verify, and (f) Eyelock Myris. Bottom row: Portable sensors (g) Hoyos HBox, (h) Sarnoff Iris on the Move, (i) AOptix Insight.

(from the surrounding regions of the iris). It has been observed that the periocular information from VIS images provides better recognition performance than that from NIR images [54]. 5. Suitability for mobile iris recognition: Since most smartphones deliver images in the VIS spectrum, the subject of iris recognition in the VIS spectral band has gained traction. In addition to the imaging wavelength, the quality of an iris image is critical for high recognition performance. Sample iris images acquired under NIR and VIS wavelength illumination are provided in Fig. 6.

Fig. 6. Corresponding pairs of iris images acquired in the NIR and VIS spectral bands. Image taken from [47].

6. Iris segmentation Iris segmentation refers to the process of automatically locating the annular region encompassed between the pupillary and limbic boundaries of the iris in a given image. The segmentation stage is considered to be of critical importance in the iris recognition process. This is because errors in segmentation can significantly lower the recognition performance [55]. The major steps involved in the iris segmentation process are listed below. 1. Specular reflection removal: Specular reflections are small regions in an iris image characterized by pixels of high intensity values. Such reflections are typically caused by improper focusing of the light source. Iris segmentation is rendered challenging if specular reflections are present on (or even close to) the iris boundaries. Additionally, specular reflections that lie within the iris boundaries act as noise, thereby lowering the recognition performance. Therefore, most iris segmentation approaches first focus on handling specular reflections. Owing to their high intensity values, specular reflections can be typically detected by simple thresholding of the image, and mitigated by multiple image processing routines (e.g., connected component analysis, region filling, etc.) 2. Pupillary and limbus boundary detection: The crux of an iris segmentation algorithm is to detect the pupillary and limbus boundaries of the iris. Most segmentation algorithms are modeled by assuming these boundaries to be circular. Such assumption holds true in a majority of cases, but may not be applicable in iris images corresponding to a deviated gaze. The order of boundary detection (limbus first, or pupillary first) does not carry any significance with

R. R. Jillela, A. Ross / Pattern Recognition Letters 57 (2015) 4–16

9

Table 2 List of a few iris segmentation approaches.

Fig. 7. Various steps involved in the iris segmentation process.

regards to the segmentation performance. However, most NIR iris segmentation approaches aim to detect the pupillary boundary first. This is because the pupil in NIR images exhibits a strong intensity variation across its boundary and can be easily detected. This region can then be used as a starting point for the outward propagated search of the limbus boundary. On the other hand, most VIS iris segmentation approaches aim to approximate the pupillary boundary first. This is because of (a) the strong edge response between the iris and sclera, and (b) weak edge response between the iris and pupil in dark colored irides. The region within the limbus boundary is then used for an inward propagated search for the pupillary boundary. 3. Eyelid detection: Under normal conditions, a minor portion of the human eye is typically occluded on the top and the bottom, by the upper and the lower eyelid, respectively. Such occlusions reduce the number of iris pixels present in an image. Owing to their physical appearance, the eyelid boundaries are typically modeled as parabolic arcs. The regions falling outside the arc but within the limbus boundary are then excluded as noise. 4. Eyelash removal: The occlusions caused by eyelashes are typically very minimal. However, eyelash occlusions render iris segmentation challenging as they can cause uneven interruptions of the limbus boundary. Eyelash removal was initially proposed as a post processing operation after iris unwrapping [56]. Most of the newer approaches, however, attempt to exclude eyelash occlusions during the segmentation stage. Fig. 7 illustrates the iris segmentation process based on the above listed steps. A summary of the most popular iris segmentation algorithms is provided in Table 2. 7. Brief review of NIR iris segmentation One of the earliest works on iris segmentation was performed by John Daugman [57]. An integro-differential operator was used to detect the iris boundaries that are approximated as perfect circles. Given an iris image I (x, y), it is first convolved with an image smoothing function (e.g., a Gaussian filter). The smoothening step helps in (a) attenuating the effect of noise in the image (e.g., sensor noise), and (b) eliminating undesired weak edges within the iris, while retaining the desired strong edges (e.g., iris boundaries, eyelid boundaries, etc.). An integro-differential operator (IDO) is then used to search for the maximum value of a normalized integral along circular contours of varying radii and center coordinates. The search process using an IDO over the image domain (x, y), can be mathematically expressed as:

  max(r, x0 , y0 ) Gσ (r) ∗

∂ ∂r



r,x0 ,y0

 I (x, y)  ds , 2π r

(1)

Reference

Imaging wavelength

Segmentation technique

[57] [42] [58] [59] [60] [61] [62] [63] [64] [65] [66] [67] [10] [48] [68] [69] [70] [71]

NIR NIR NIR NIR NIR NIR NIR NIR NIR NIR NIR NIR NIR VIS VIS VIS VIS VIS

[72] [73] [74]

VIS VIS VIS

[75] [76] [77]

VIS VIS VIS

[78] [79]

VIS VIS

Integro-differential operator Hough transform Hough transform Hough transform Pulling and pushing elastic model Active shape models Geodesic active contours Hough transform Variational level sets Graph cuts Texture statistics Iterative directional ray approximation Fourier approximation Color component analysis Zernike moments Edge presence and sclera intensity measurement Boundary regularization Clustering, semantic rules, and integro-differential operator Knowledge based (successive decision rules) Color and shape descriptor fusion Multiscale sparse representation of local Radon transform Mathematical morphology Watershed algorithm K-means clustering of gray-level co-occurrence histograms Adaboost and color segmentation Iterative refinement approach

represents the radial Gaussian with a center r0 and standard deviation (scale) σ , which is used for image smoothing. The symbol ∗ denotes the convolution operation and r represents the radius of the circular arc ds, centered at the location (x0 , y0 ). The division by a factor of 2π r normalizes the circular integral with respect to its perimeter. In other words, the IDO behaves as a circular edge detector that searches iteratively for the maximum response of a contour path defined by the parameters (x0 , y0 , r). Depending on the values of the radii considered, the optimal parameters of the IDO are treated as either the pupillary or limbus boundaries. Once both the iris boundaries are detected, the boundaries of the eyelids can be detected by changing the integration path of the operator from circular to arcuate. Another widely popular approach for performing iris segmentation uses Hough transforms [42]. Similar to IDO, this technique also approximates iris boundaries as perfect circles and the eyelid boundaries as ellipses. A wide number of iris segmentation approaches improve on the idea of Hough transform [58,59,63,80]. The approximation of the iris boundaries as perfect circles can be accepted when an iris image is acquired under near-ideal conditions from a cooperative subject.5 In an image acquired under non-ideal conditions, the limbus boundary may not be completely circular (due to the occlusions, gaze deviations, etc.). Some of the prominent approaches that can perform segmentation of non-circular irides use Geodesic Active Contours [62], variational level sets [64], Fourierbased approximations [57], Active Shape Models (ASM) [61], graph cuts [65], iterative directional ray based segmentation [67], and texture statistics [66]. Some of the well known segmentation free approaches include the usage of SIFT features [41,81]. A fairly detailed survey of iris segmentation approaches is presented in [37,82,83]. Research on improving iris segmentation includes techniques for eyelash detection and removal [9,56,80], segmentation error prediction [55], and modifications to existing open-source segmentation implementations [84,85].

where (r−r0 )2 1 −( ) Gσ (r) = √ exp 2σ 2 2πσ

(2)

5 However, it must be noted that the iris boundaries are not circular for many subjects even under ideal conditions.

10

R. R. Jillela, A. Ross / Pattern Recognition Letters 57 (2015) 4–16

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

Fig. 8. VIS iris images acquired using mobile devices, showcasing factors that render iris segmentation very challenging. (a) and (b) Improper illumination, (c) and (d) specular reflections, (e) and (f) deviated gaze, (g) eyelid occlusion, (h) eyelash occlusion, (i) out of focus blur (compare with image in (j) that contains no blur), (k) and (l) dark colored irides, (m) and (n) eyeglasses, (o) and (p) improper sensor positioning. Images taken from [86] database. The images have been scaled down without altering their aspect ratio.

8. Mobile iris recognition

• •

Mobile iris recognition depends on using the camera provided within a user’s mobile device, without requiring additional sensors. Typical sensor-to-eye distances under such conditions can range between 5 and 25 in. An evaluation of various mobile devices based on their ability to perform iris recognition is still pending in the biometrics literature. Owing to their image resolution and processing capabilities, current mobile devices can be expected to provide a fair verification performance under cooperative conditions. One of the major difference between mobile and conventional iris recognition systems is the image acquisition spectra. Mobile iris recognition is perceived to be performed mainly using iris images acquired under the visible (VIS), instead of the near-infrared (NIR), spectrum of light. Such difference necessitates several modifications to the conventional iris recognition process. The most significant of these is the iris segmentation and feature extraction stages. In this work, the various aspects of VIS iris segmentation are discussed. 8.1. Challenges

• • • • • •

Variations in illumination Specular reflections Deviated gaze Eyelid and eyelash occlusions Out of focus blur (or, motion blur) Dark colored irides Eyeglasses and contact lenses Improper sensor (camera) positioning

Sample VIS iris images acquired using mobile devices that highlight each of the above listed factors are shown in Fig. 8. 9. VIS iris segmentation Research in VIS iris segmentation has been gaining attention in the past few years, due to an increasing interest in iris recognition at a distance. In this regard, the Phase I of the Noisy Iris Challenge Evaluation (NICE-I) [17] competition provided a platform for rigorous research in VIS iris segmentation. Several other approaches were proposed in the following years. In this section, some of the well known VIS iris segmentation techniques are reviewed with a focus on limbus and pupillary boundary detection. 9.1. Color component analysis approach – Proenca

Iris segmentation in VIS images acquired using mobile devices (even with cooperative subjects) is a very challenging task. Some of the factors that contribute to this challenge are:

Proenca’s research can be considered as one of the earliest in VIS iris segmentation [48,87]. The goal of this technique is to

R. R. Jillela, A. Ross / Pattern Recognition Letters 57 (2015) 4–16

11

Fig. 10. Left to right: Sclera detection output, sc← (x, y) map, saturation component, and the chrominance component, respectively. Image taken from [87]. Fig. 9. Left to right: The hue, chrominance, and red chroma components of a VIS iris image, respectively. Image taken from [87].

automatically segment VIS iris images acquired from distances of 4 to 8 m under varying illumination. The sclera region is first detected, based on the observation that it is the well distinguishable even under varying illumination. The neighborhood condition, the fact that sclera mandatorily lies next to the iris, is invoked to reliably detect the iris region. A color channel analysis within the iris region of the image is then performed to determine the iris boundaries. Consider a VIS iris image I of size M × N. A multi-dimensional feature vector is first constructed for each pixel at location (x, y), where 1 ≤ x ≤ M and 1 ≤ y ≤ N. The feature vector is generated using the following expression:



μ,σ

x, y, hr



(x, y), uμr ,σ (x, y), crrμ,σ (x, y) ,

(3)

where h(), cb(), and cr() denote the hue, chrominance, and red chroma components of the image at that pixel location. Sample images showing the color components of a VIS iris image are shown in Fig. 9. The choice of the three color components is based on an empirical observation that they maximize the contrast between the sclera and all other regions within the image. The subscript r denotes the radius of a circle centered at the pixel. Parameters μ, and σ , denote the mean, and the standard deviation, respectively, of the set of pixels which fall within the considered circular regions. A 20 dimensional vector is then generated by considering three different values for r (i.e., r = 1, 3, and 7). Once the feature vectors for all the image pixels are calculated, a neural network classifier is used to classify the sclera and non-sclera regions. In the following step, the sclera map is used to generate a new feature, proportion of sclera, p(x, y), at each pixel location. This feature helps in measuring the proportion of pixels that belong to the sclera in a given direction d, with respect to the pixel at location (x, y). It is suggested that a pixel which lies within iris boundaries, when frontally imaged, will have similar number of sclera pixels on both sides. On the other hand, if the iris is off-angled, then the number of pixels of sclera on one side will be more than the number of pixels on the other side. This information, determined using p(x, y), is used to highlight the iris regions within the confines of the sclera. The proportion of sclera features along four directions are computed using the following equations:

Fig. 11. Top row (left to right): Given VIS iris image, saturation component, and normalized blue component, respectively. Bottom row (left to right): Difference of chroma red (cr) and chroma blue (cb) components, difference of RGB, and mean component of RGB, respectively. Image taken from [68].

where s() and cb() denote saturation and blue chrominance at the considered location (x, y). Sample images corresponding to the considered features are shown in Fig. 10. Once again, the choice of color components is based on empirical evaluation. A multi-layered perceptron feed-forward neural network with one hidden layer is used again to classify the iris and non-iris regions. The output of this stage produces a roughly highlighted iris region bound by the pupillary and limbus boundaries. The highest possible contours of the limbus and pupillary boundaries are then determined using shape parametrization techniques. 9.2. Zernike moments approach – Tan and Kumar Tan and Kumar proposed a segmentation approach [68] that is similar to that of Proenca [48], in which the iris is localized by first determining the sclera regions. Given an iris image I, the sclera feature vector at a given location (x, y) is computed as:



 μσ μσ μσ x, y, Sr , nbr , dcr−cb(r), dRGB (x1 , x2 ), μRGB (x, y) ,

(9)

p↑ (x, y) =

μ(sc((x − 1, 1), (x, y))),

(6)

where S, nb, dcr−cb , dRGB and μc denote the saturation, normalized blue, difference of chroma red (cr) and chroma blue (cb), and the mean value of the R, G, B channels, at (x, y) respectively. The term r denotes the radius of the region of interest (ROI) circle centered at (x, y). The superscripts μ and σ indicate the mean and the standard deviation of the considered features within the ROI. The saturation and normalized blue values (i.e., S and nb) are considered after subtracting their mean values within the ROI. The terms dcr−cb , dRGB , and μc are computed as follows:

p↓ (x, y) =

μ(sc((x − 1, y), (x, h))).

(7)

dcr−cb (x, y) = cr(x, y) − cb(x, y),

(10)

dRGB (x, y) = 2IR (x, y) − IG (x, y) − IB (x, y),

(11)

p← (x, y) =

μ(sc((1, y − 1), (x, y))),

(4)

p→ (x, y) =

μ(sc((x, y − 1), (w, y))),

(5)

The notation used for the directions are ↑ for north, ↓ for south, ← for east and → for west. The term sc() denotes the regions of the image cropped from the detected sclera, delimited by the top-left and bottom-right corner coordinates. w and h denote the width and height, respectively (i.e., w = M and h = N). The proportion of sclera values, pixel locations, local image saturation, and blue chrominance are then used to form a new feature vector represented as:



 μ,σ μ,σ x, y, s0,3,7 (x, y), cb0,3,7 (x, y), p←,→,↑,↓ (x, y)

(8)

μRGB (x, y) =

1 N



Ic (x, y).

(12)

c{R,G,B}

The term N represents the total number of color channels in the given image (N = 3), and acts as a normalizing factor. Sample images corresponding to various features used in this method are shown in Fig. 11. The final feature vectors are then provided as input to an SVM classifier which distinguishes the sclera from non-sclera regions. The iris

12

R. R. Jillela, A. Ross / Pattern Recognition Letters 57 (2015) 4–16

feature vectors are computed using the localized Zernike moments and sclera information (determined in the previous stage) as follows: r (I), pw,e,n,s (x, y)} , {x, y, IR (x, y), Zmn

(13)

where IR represents the red channel of the input image. The term Z represents a function that computes the Zernike moments of I within a circular ROI of radius r, centered at (x, y), given as:

Zmn =

m + 1 

π

x

f (x, y)[Vmn (x, y)] ∗ dxdy.

(14)

y

The subscripts m and n denote the order and angular dependence of the Zernike moments. The term f (x, y) denotes the ROI under consideration and Vmn is the Zernike polynomial. The terms pw,e,n,s (x, y) denote the proportion of sclera in the west (w), east (e), north (n), and south (s) directions, respectively. The values of pw,e,n,s (x, y) can be computed using Eqs. (4), (5), (6), (7), respectively. A different SVM classifier is then used to classify image pixels into iris and non-iris regions. The edge maps obtained from this stage are refined to accurately determine the limbus and pupillary boundaries. 9.3. Multi-stage refinement approach – Sankowski et al. The iris segmentation routine proposed by Sankowski et al. performs VIS iris segmentation using a multi-stage approach [69]. The various stages involved are: specular reflection removal, iris boundary detection, and eyelid detection. Given an iris image I, the specular reflections are detected by an image-specific thresholding approach. To this end, the RGB image is converted to a gray scale image, Ig , based on the YIQ (luminance phase quadrature) model. The image specific threshold used to highlight specular reflections (TSR ) is given as:

Tsr = μIg + k · (M − μIg ),

(15)

where k ∈ (0, 1), and μIg represents the average value of all the pixel intensities within the image Ig . The term M denotes the average value of a fixed number of pixels with highest intensities within the image. This fixed number is specific to the image considered and is set to 0.04% of the total number of pixels in the image. All pixel locations with image intensities above the threshold are considered as specular reflections. The pixel intensity at specular reflection location k, I (k), is replaced in the original RGB image using an interpolation process given by:

Ic (k) =

4

i=1

4

wi Ic (i)

i=1

wi

(16)

where, c denotes the channel considered (R, G, or B), i denotes the specific neighbor within a four-pixel neighborhood. The term wi represents the weight associated with the neighbor and is computed as the inverse of the distance between k and i. The limbus and pupillary boundaries are then detected using an IDO see Eq. (1)). For dark colored irides, the pupillary boundary can be more challenging to detect than the limbus boundary. To this end, the pupillary boundary detection is enhanced by invoking the IDO process (a) only within the limbic boundary region, and (b) only using the R channel of the image (based on empirical observations). The lower eyelid, modeled as a circular arc, is detected using multiple cues. To this end, the iris image I is first smoothed using a Gaussian filter to filter noise. A Sobel edge detector is then applied on I to generate an edge image E. The goodness of fit measure of a given arc within E is computed by the edge presence and the sclera intensity criteria. Let the number of arcs within the edge image be denoted by N. Let the number of pixels belonging to an arc n (n = 1, 2, . . . N) be denoted by M(n). The edge presence criterion of a given edge n, g1 (n), is computed using the following equation:

gl1 (n) =

M(n) 1  E(x(n, i), y(n, i)), M(n) i=1

(17)

Fig. 12. Sampling process used to determine the eyelid boundaries. The red colored rectangles represent regions included in the sclera. The white colored rectangles represent regions from the eyelid area. Image taken from [69]. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

where (x(n, i), y(n, i)) denote the coordinates of the ith pixel of the nth edge. Let (xend (n, j), yend (n, j)) denote the coordinates of the jth pixel from the ends of the nth arc. Let J(n) denote the number of pixels belonging to ends of the nth arc. The sclera intensity criterion, gl2 (n) is computed using the equation:

gl2 (n) =

J(n)

1  Ig (xend (n, j), yend (n, j) − yoff set ), J(n)

(18)

j=1

where Ig denotes the gray scale image obtained using the YIQ model. The term yoff set denotes the distance from arc end pixels to the pixels in which the sclera intensity is sampled. The two goodness of fit criteria are combined by a product rule as: glower (n) = gl1 (n)·gl2 (n). The edge n corresponding to the largest value of glower (n) is considered as the lower eyelid boundary. Unlike the lower eyelid, the upper eyelid is modeled as a line segment and not an arc. It is noticed that the upper eyelid often can be occluded by eyelashes, thereby hindering the localization process. Therefore, a separate algorithm is used for upper eyelid detection. Based on the output obtained from the limbus boundary, small rectangular ROIs from the sclera and eyelid regions of the image are considered. Let these regions be of uniform size of W × H, and be denoted by Ssclera and Seyelid , respectively. Three different measures, gu1 , gu2 , and gu3 , are computed using the following equations:

gu1 = 1 + gu2 =

gu3

μscleraR + μscleraG + μscleraB

(19)

3 1 μ

(20)

+μ

1 + eyelidG 2 eyelidB

1 + GB if GB ≥ 0 = 1 otherwise

(21)

where

GB =

μscleraG − μeyelidG + μscleraB − μeyelidB 2

,

(22)

and the terms μscleraR , μscleraG , and μscleraB denote the mean values of the pixel intensities of R, G, and B channels of Ssclera , respectively. Similarly, the terms μeyelidR , μeyelidG , and μeyelidB denote the mean values of the pixel intensities of R, G, and B channels of the image Seyelid , respectively. The three measures are then combined using the product rule as gupper = gu1 ·gu2 ·gu3 . To determine the upper eyelid boundaries, several pairs of Ssclera and Seyelid ROIs are first sampled around the upper region of the limbus boundary. The sampling process is illustrated in Fig. 12. The goodness of fit measure for the upper

R. R. Jillela, A. Ross / Pattern Recognition Letters 57 (2015) 4–16

eyelid, gupper is computed for every considered pair of Ssclera and Seyelid . Points that lie on the line segments connecting the pairs of ROIs with largest gupper are considered to form the upper eyelid boundary. 9.4. Boundary regularization approach – Labati and Scotti

(23)

where  represents the angle image, computed using the following equation:

(x, y, x0 , y0 ) = tan−1 [(y − y0 )/(x − x0 )].

(24)

The radial strip image corresponding to the pupillary boundary (xp , yp ) is computed using the equation:

⎧ R(x, y) |xp ,yp , if rp (1 − α) ≤ . . . ⎪ ⎪ ⎨  Rp (x, y) = . . . ((x − xp )2 + (y − yp )2 ) ≤ rp (1 + α); ⎪ ⎪ ⎩ 0, otherwise .

(25)

Similarly, the radial strip image corresponding to the limbus boundary (xl , yl ) is computed using the equation:

⎧ R(x, y) |xl ,yl , if rl (1 − α) ≤ . . . ⎪ ⎨  Rl (x, y) = . . . ((x − xl )2 + (y − yl )2 ) ≤ rl (1 + α); ⎪ ⎩ 0, otherwise .

first approximation of the limbus and pupillary boundaries, bp (θ ) and bl (θ ), are computed from Rp (ρ , θ ) and Rl (ρ , θ ), as:

bp (θ ) =

max

rp (1−α)≤ρ ≤rp (1+α)

Rp (ρ , θ ),

(27)

and

The approach proposed by Labati and Scotti performs iris segmentation on RGB images by first converting them to gray scale [70]. The segmentation process is divided into multiple stages. Given an iris image I, an IDO is used to determine the rough locations of the pupillary and limbus boundaries [25]. Let the output of this step be denoted by {xp , yp , rp } and {xl , yl , rl }, where (xp , yp ) and (xl , yl ) represent the pupillary and limbus boundary centers, and rp and rl are their corresponding radii, respectively. In the next stage, radial strips of fixed sizes around both the boundaries are extracted from the gradient image of I. These strips can be perceived as regions encompassed between concentric circles of specific radii around the limbus and pupillary boundaries. A Sobel edge filter is convolved with the iris image I. Let the magnitude and phase of the convolution output be denoted by G(x, y) and θ (x, y), respectively. A gradient image, R(x, y), with respect to a reference point (x0 , y0 ), denoted by R(x, y) |x0 ,y0 , can then be computed as:

R(x, y) |x0 ,y0 = G(x, y) cos(θ (x, y) − (x, y, x0 , y0 )),

13

(26)

The value of α is determined empirically based on a training dataset. The radial strip images, Rp (x, y) and Rl (x, y), in Cartesian coordinates are then unwrapped using Daugman’s approach [25] into pseudopolar coordinates, yielding Rp (ρ , θ ) and Rl (ρ , θ ), respectively. Sample radial strip images extracted from a gradient image are shown in Fig. 13. The next step of the algorithm attempts to determine the accurate locations of the limbus and pupillary boundaries from the corresponding unwrapped Rl (ρ , θ ) and Rp (ρ , θ ) images, respectively. To this end, the local maxima along the columns (ρ axis) of the images are computed. This is based on the observation that an iris boundary exhibits high values within the localized radial gradient image. The

Fig. 13. Left: Given VIS iris image. Middle: Corresponding gradient image obtained by Sobel edge filtering. Right top and bottom: Radial strip images corresponding to the pupillary and limbic boundaries, respectively. Image taken from [70].

bl (θ ) =

max

rl (1−α)≤ρ ≤rl (1+α)

Rl (ρ , θ ),

(28)

respectively. It is possible that the obtained boundary approximations suffer from abrupt variations in their shape and continuity due to occlusions from eyelashes or eyelids. This is mitigated by applying boundary regularization based on empirical observations. A reliable estimate of pupillary boundary, Bθ , can be obtained from its corresponding approximation bp (θ ) as follows: 1. Starting from an angle θi , all the values of bp (θ ) that satisfy the inequality | bp (θi ) − bp (θj ) |< t1 are collected. The term t1 denotes an empirical threshold, (i < j), and i = 1 : 1 : N where N is the angular resolution of the analysis and is set to 360/N degrees. This step helps in identifying the continuous segments. 2. To ensure polar continuity, two segments are merged if the first point of the first segment and the last point of the last segment satisfy previous condition. 3. If (j − i) > t2 , the segment is considered to be continuous, and the values {bp (θi ), bp (θi=1 ), . . . , bp (θj )} are stacked in the vector B. Otherwise, the segment is considered to be discontinues and (j − i) zeros are stacked in B. 4. The above steps are repeated until all the available points within the approximated boundary are processed. The same procedure is applied on the approximate limbus boundary, bl (θ ), to determine its corresponding reliable estimate, Bl (θ ). Both Bp (θ ) and Bl (θ ) can exhibit sharp variations in their contours (potentially caused by eyelashes, eyelids, etc.). To this end, boundary smoothing is performed using the Fourier coefficients method suggested by Daugman [10]. In this method, a given iris boundary B(θ ) is expressed in terms of the set of coefficients corresponding to the Fourier series as:

Ck =

N−1 

B(θ ) exp(−j(2π /N)kθ ).

(29)

θ =0

To obtain the smoothing effect, the boundary is expressed using only the first M (instead of N) Fourier coefficients as: M−1 1  Ck exp(−j(2π /N)kθ ), bˆ (θ ) = N

(30)

i=0

where M < N. The choice of M regulates the smoothness of the boundary and is determined based on empirical evaluations. A sample iris segmentation output obtained by the regularization process is shown in Fig. 14.

Fig. 14. Left: Regularization output obtained by the segmentation approach. Right: Segmentation output overlaid on the gray scale version of the input VIS iris image. Image taken from [70].

14

R. R. Jillela, A. Ross / Pattern Recognition Letters 57 (2015) 4–16

9.5. Clustering and semantic rules approach: Tan et al. The approach proposed by Tan et al. performs iris segmentation using a two stage approach [71]. In the first stage, a clustering based scheme is used for coarse iris boundary localization. In the next stage, the accurate boundaries are determined using a variation of the integro-differential operator. This method combines some of the concepts introduced by the same authors in another work proposed for NIR iris segmentation under challenging conditions [88]. Given an RGB iris image, the red channel is first separated for further processing. This is based on the observation that the red channel is closer to the NIR wavelength. Let the extracted red channel image be denoted by I. To avoid the impact of specular reflections on iris segmentation, an image specific reflection detection and removal scheme is first invoked. A specular reflection threshold, TSR is computed as the average value of the top 5% of the brightest intensity pixels in the iris image. A binary reflection map R(x, y) of I (x, y) is generated using TSR threshold. All the pixel locations with value 1, called as reflection points, are considered to be specular reflections. These regions are filled using a bilinear interpolation method. For a given reflection lef t right top point P(0xo ,yo ), four envelope points {P(x ,yo ), P(xr ,yo ), P(xo ,yt ), P(down xo ,yd )} are l computed using the following equations:



xl = max x : x

L−1 

xr = max x : x

•

•

R(x+i,yo ) = 0, R(x+L,yo ) = 1, x < xo ,

L−1 

(31) •

 R(x−i,yo ) = 0, R(x−L,yo ) = 1, x > xo ,

yt = max y : y

L−1 

(32)

 R(xo ,y+i) = 0, R(xo ,y+L) = 1, y < yo ,

•

(33)

i=0

and



yd = max y : y

L−1 

(34)

i=0

The term L denotes the separation (or neighborhood) between a reflection point and its envelope points. The bilinear interpolation scheme used to fill the value of a reflection point coordinate in the image I is expressed as:

I (P )(xr − xo ) + I (P )(xo − xl ) 2(xr − xl ) l

+

r

I (P t )(yd − yo ) + I (P d )(yo − yt ) . 2(yd − yt )

| I (P ) − μR |

σR

,

(36)

where I (P ) is the intensity of the un-clustered point. Step 3: An un-clustered point P is assigned to a region R if it satisfies both the following conditions: (a) D (P, R) is below a fixed threshold, TP2R , and (b) there exists a valid eight-neighbor connection path that connects P to R. Step 4: The steps (1)–(3)are repeated until all regions are handled, and all un-clustered points are assigned to a region.

A challenging problem after clustering the image is to separate the iris region from the other regions. To address this issue, the following semantic priors are taken into account:

 R(xo ,y−i) = 0, R(xo ,y+L) = 1, y > yo .

Step 1: The mean and standard deviation of the pixel intensities of each candidate region R, μR and σR , are computed. Step 2: For a randomly chosen candidate region R, the distance of each un-clustered point, P, to the region is computed using the point-to-region distance equation given by:

D (P, R) =

i=0



I (P o ) =

the iris pixels is relatively lower than those of the skin region pixels. Accordingly, the coarse cluster initialization is performed by assigning the top 30% and 20% pixels to the skin and the iris regions, respectively. To cluster the remaining un-clustered points, an iterative process is carried that consists of the following steps:



i=0



Fig. 16. Left: Clustering initialization stage. Middle: Clustering result after the first two steps of the iteration process. Right: Semantic labeling output. Image taken from [71].

(35)

A sample image showing the output of the specular removal process is provided in Fig. 15. After the specular reflections are removed, a clustering algorithm is used to classify the pixels into different regions based on structure of various parts (e.g., iris, eyebrows, surrounding skin, etc.). Based on empirical observations, the authors indicate that the intensities of

Fig. 15. Left: Red channel of the input VIS image. Right: Output image generated by the specular reflection removal process. Image taken from [71].

•

• •

An iris region, unlike other regions, has a “-o-” like shape with its center thicker than its ends. The eyebrow region has a stripe like appearance. Eye glasses, if present, appear approximately rectangular.

Using the above semantic labeling rules, a coarse estimation of the iris is obtained. Images obtained after the clustering and semantic labeling processes are shown in Fig. 16. Within the iris region, the limbus and pupillary boundaries are detected using a variation of the classical IDO. 10. Summary and research challenges The process of segmenting the iris plays a critical role in the performance of an iris recognition system. A wide number of iris segmentation approaches have been proposed in the biometric literature. However, a majority of these approaches have been focused on images acquired in the near-infrared spectrum. Iris segmentation using color images has been a relatively less explored research area. With an increasing interest in mobile device based iris recognition (viz., smartphones), color iris recognition has been gaining significant traction. In this regard, this paper focuses on the segmentation aspect of color iris images. An overview of several color iris segmentation approaches presented in the literature was presented. The research challenges involved in color iris segmentation are: 1. Reliable imaging of good quality iris images from a distance. 2. Efficient localization of limbus and pupillary boundaries, even for dark colored irides. 3. Accurate estimation of eyelid, eyelash, and specular reflection locations.

R. R. Jillela, A. Ross / Pattern Recognition Letters 57 (2015) 4–16

It is necessary to perform an exhaustive evaluation of iris segmentation and recognition schemes based on color iris images acquired using mobile devices. Fusion of iris with conjunctival vasculature [89,90] and periocular information [91] also be considered to improve recognition performance in color images. References [1] ITU, International Telecommunications Union, ICT Facts and Figures – Global Statistics Report, 2013, (Last accessed in July 2014 from http://www.itu.int/en/ ITU-D/Statistics/Documents/facts/ICTFactsFigures2013-e.pdf). [2] INMOBI, Global Mobile Media Consumption Whitepaper, 2013, (Last accessed in July 2014 from http://info.inmobi.com/rs/inmobi/images/Global_Mobile_Media_ Consumption_Wave_2_Whitepaper.pdf). [3] A.K. Jain, A. Ross, S. Prabhakar, An introduction to biometric recognition, IEEE Trans. Circuits Syst. Video Technol. 14 (1) (2004) 4–20. [4] A. Jain, A. Ross, K. Nandakumar, Introduction to Biometrics, Springer, 2011. [5] S. Li, A. Jain, Handbook of Face Recognition, Springer, 2005. [6] D. Maltoni, D. Maio, A. Jain, S. Prabhakar, Handbook of Fingerprint Recognition, Springer, 2009. [7] M. Burge, K. Bowyer (Eds.), Handbook of Iris Recognition, Springer, 2013. [8] A. Ross, Iris recognition: the path forward, Computer 43 (2) (2010) 30–35. [9] J. Daugman, The importance of being random: statistical principles of iris recognition, Pattern Recognit. 36 (2) (2003) 279–291. [10] J. Daugman, New methods in iris recognition, IEEE Trans. Syst. Man Cybern. Part B: Cybern. 37 (5) (2007) 1167–1175. [11] U. Park, R. Jillela, A. Ross, A.K. Jain, Periocular biometrics in the visible spectrum, IEEE Trans. Inf. Forensics Security 6 (1) (2011) 96–106. [12] M. Nixon, T. Tan, R. Chellappa, Human Identification Based on Gait, Springer, 2005. [13] B. DeCann, A. Ross, Gait Curves for Human Recognition, Backpack Detection, and Silhouette Correction in a Nighttime Environment, Proc. of SPIE Conference on Biometric Technology for Human Identification VII, (Orlando, USA), 2010. [14] STRUCTURE, 3D Structure Sensor for Mobile Devices, 2014, (Last accessed in July 2014 from http://structure.io/). [15] SPEECHPRO, Bi-modal Authentication Solution (Voice+Face) Voicekey Onepass, 2014, (Last accessed in July 2014 http://speechpro-usa.com/product/voice_ authentication/voicekey-onepass). [16] K. Lee, B. Ma, L. H., Speaker verification makes its debut in smartphone, IEEE Signal Process. Soc. SLTC Newslett. (2013). [17] H. Proença, L.A. Alexandre, The NICE I: noisy iris challenge evaluation, in: First IEEE International Conference on Biometrics: Theory, Applications, and Systems, IEEE, 2007, pp. 1–4. [18] F. Juefei-Xu, C. Bhagavatula, A. Jaech, U. Prasad, M. Savvides, Gait-ID on the move: pace independent human identification using cell-phone accelerometer dynamics, in: 2012 IEEE Fifth International Conference on Biometrics: Theory, Applications and Systems (BTAS), IEEE, 2012, pp. 8–15. [19] N. Sae-Bae, K. Ahmed, K. Isbister, N. Memon, Biometric-rich gestures: a novel approach to authentication on multi-touch devices, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 2012, pp. 977–986. [20] S. Karatzouni, N. Clarke, Keystroke Analysis for Thumb-based Keyboards on Mobile Devices, in: H. Venter, M. Eloff, L. Labuschagne, J. Eloff, R. von Solms (Eds.), New Approaches for Security, Privacy and Trust in Complex Environments, volume 232 of IFIP International Federation for Information Processing, Springer, USA, 2007, pp. 253–263. [21] M. Frank, R. Biedert, E. Ma, I. Martinovic, D. Song, Touchalytics: on the applicability of touchscreen input as a behavioral biometric for continuous authentication, IEEE Trans. Inf. Forensics Security 8 (1) (2013) 136–148. [22] A. De Luca, A. Hang, F. Brudy, C. Lindner, H. Hussmann, Touch me once and i know it’s you!: implicit authentication based on touch screen patterns, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 2012, pp. 987–996. [23] F.H. Adler, Physiology of the Eye: Clinical Application, The C.V. Mosby Company, London, 1965. [24] P.C. Kronfeld, Gross anatomy and embryology of the eye, in: H. Davson (Ed.), The Eye, Academic Press, London, 1962. [25] J.G. Daugman, High confidence visual recognition of persons by a test of statistical independence, IEEE Trans. Pattern Anal. Mach. Intell. 15 (1993) 1148–1161. [26] M. Sam Sunder, A. Ross, Iris image retrieval based on macro-features, in: Proceedings of the 20th International Conference on Pattern Recognition, IEEE, 2010, pp. 1318–1321. [27] A. Bertillon, La couleur de l’iris, Rev. Sci. 36 (3) (1885) 65–73. [28] A. Bertillon, Alphonse Bertillon’s instructions for taking descriptions for the identification of criminals, and others, by the means of anthropometric indications, in: Identifiction Anthropomtrique, Instructions Signaltiques, Translated from French by Muller, G., American Bertillon Prison Bureau, Chicago, IL, 1889. [29] I. Mann, The Development of the Human Eye, Cambridge University Press, for the British Journal of Ophthalmology, 1928, p. 306. [30] H. Rohen, The construction of the iris in humans and some mammals, Gegenbaur Morphol. J. 91 (1951) 140–181. [31] H.P. Wasserman, Ethnic Pigmentation: Historical, Physiological, and Chemical Aspects, 1974 American Elsevier, New York, . [32] L. Flom, A. Safir, U. S. Patent No. 4 641 349, 1987, (U. S. Government Printing Office, Washington, D. C.).

15

[33] C. Fancourt, L. Bogoni, K. Hanna, Y. Guo, R. Wildes, N. Takahashi, U. Jain, Iris recognition at a distance, in: Proceedings of the 5th International Conference on Audio- and Video-Based Biometric Person Authentication, AVBPA’05, SpringerVerlag, 2005, pp. 1–13. [34] J.R. Matey, O. Naroditsky, K. Hanna, R. Kolczynski, D.J. LoIacono, S. Mangru, M. Tinker, T.M. Zappia, W.Y. Zhao, Iris on the move: acquisition of images for iris recognition in less constrained environments, Proc. IEEE 94 (11) (2006) 1936–1947. [35] J. De Villar, R. Ives, J. Matey, Design and implementation of a long range iris recognition system, in: 2010 Conference Record of the Forty Fourth Asilomar Conference on Signals, Systems and Computers (ASILOMAR), 2010, pp. 1770– 1773. [36] D. Ackerman, Opt. Iris Imag. Syst. (2013) 367–393. [37] R. Jillela, A. Ross, Methods for iris segmentation, in: K. Bowyer, M. Burge (Eds.), Handbook of Iris Recognition, Springer, 2012. [38] J. Zuo, N. Kalka, N. Schmid, A robust Iris Segmentation Procedure for Unconstrained Subject Presentation, in: 2006 Biometrics Symposium: Special Session on Research at the Biometric Consortium Conference, 2006, pp. 1–6. [39] K.W. Bowyer, K. Hollingsworth, P.J. Flynn, Image understanding for iris biometrics: a survey, Comp. Vis. Image Und. 110 (2) (2008) 281–307. [40] J. Daugman, Probing the uniqueness and randomness of IrisCodes: results from 200 billion iris pair comparisons, Proc. IEEE 94 (11) (2006) 1927–1935. [41] C. Belcher, Y. Du, Region-based SIFT approach to iris recognition, Opt. Lasers Eng. 47 (1) (2009) 139–147. [42] R. Wildes, Iris recognition: an emerging biometric technology, Proc. IEEE 85 (9) (1997) 1348–1363. [43] M. Negin, T. Chmielewski, M. Salganicoff, U. von Seelen, P. Venetainer, G. Zhang, An iris biometric system for public and personal use, Computer 33 (2) (2000) 70–75. [44] K. Smith, V. Pauca, A. Ross, T. Torgersen, M. King, Extended evaluation of simulated wavefront coding technology in iris recognition, in: IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS), 2007, pp. 1–7. [45] R. Plemmons, M. Horvath, E. Leonhardt, V. Pauca, S. Prasad, S. Robinson, H. Setty, T. Torgersen, J. van der Gracht, E. Dowski, R. Narayanswamy, P. Silveira, Computational Imaging Systems for Iris Recognition, in: Proceedings of the SPIE, vol. 5559, 2004, pp. 346–357. [46] S. Tankasala, R. Pasula, V. Gottemukkula, S. Saripalle, V. Nalamati, A. Ross, R. Derakhshani, A video based hyper-focal imaging method for iris recognition in the visible spectrum, IEEE International Conference on Technologies for Homeland Security, 2012. [47] C. Boyce, A. Ross, M. Monaco, L. Hornak, X. Li, Multispectral iris analysis: a preliminary study, in: Computer Vision and Pattern Recognition Workshop on Biometrics, 2006. [48] H. Proença, Iris recognition: on the segmentation of degraded images acquired in the visible wavelength, IEEE Trans. Pattern Anal. Mach. Intell. 32 (8) (2010) 1502–1516. [49] G. Santos, M. Bernardo, H. Proenca, P. Fiadeiro, Iris recognition: preliminary assessment about the discriminating capacity of visible wavelength data, in: IEEE International Symposium on Multimedia (ISM), 2010, pp. 324–329. [50] E. Krichen, M. Mellakh, S. Garcia-Salicetti, B. Dorizzi, Iris identification using wavelet packets, in: Proceedings of the 17th International Conference on Pattern Recognition, vol. 4, 2004, pp. 335–338. [51] M. Vatsa, R. Singh, A. Ross, A. Noore, Quality based fusion for multichannel iris recognition, in: 20th International Conference on Pattern Recognition (ICPR), 2010, pp. 1314–1317. [52] A. Ross, R. Pasula, L. Hornak, Exploring multispectral iris recognition beyond 900 nm, in: IEEE 3rd International Conference on Biometrics: Theory, Applications, and Systems (BTAS), 2009, pp. 1–8. [53] R. Ives, H. Ngo, S. Winchell, J. Matey, Preliminary evaluation of multispectral iris imagery, in: IET Conference on Image Processing, 2012, pp. 1–5. [54] K. Hollingsworth, S. Darnell, P. Miller, D. Woodard, K. Bowyer, P. Flynn, Human and machine performance on periocular biometrics under near-infrared light and visible light, IEEE Trans. Inf. Forensics Security 7 (2) (2012) 588–601. [55] N. Kalka, N. Bartlow, B. Cukic, An automated method for predicting iris segmentation failures, in: IEEE 3rd International Conference on Biometrics: Theory, Applications, and Systems, IEEE, 2009, pp. 1–8. [56] D. Zhang, D. Monro, S. Rakshit, Eyelash removal method for human iris recognition, in: IEEE International Conference on Image Processing, IEEE, 2006, pp. 285–288. [57] J. Daugman, How iris recognition works, IEEE Trans. Circuits Syst. Video Technol. 14 (1) (2004) 21–30. [58] L. Ma, Y. Wang, T. Tan, Iris recognition using circular symmetric filters, in: Proceedings of the 16th IEEE International Conference on Pattern Recognition, vol. 2, 2002, pp. 414–417. [59] J. Huang, Y. Wang, T. Tan, J. Cui, A new iris segmentation method for recognition, in: Proceedings of the 17th IEEE International Conference on Pattern Recognition, vol. 3, 2004, pp. 554–557. [60] Z. He, Z. Sun, T. Tan, X. Qiu, C. Zhong, W. Dong, Boosting ordinal features for accurate and fast iris recognition, in: IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2008, pp. 1–8. [61] A. Abhyankar, S. Schuckers, Active shape models for effective iris segmentation, in: Proceedings of SPIE, vol. 6202, 2006, pp. 166–175. [62] S. Shah, A. Ross, Iris segmentation using geodesic active contours, IEEE Trans. Inf. Forensics Security 4 (4) (2009) 824–836. [63] J. Thornton, M. Savvides, B. Vijayakumar, Robust iris recognition using advanced correlation techniques, Image Anal. Recognit. 3656 (2005) 1098–1105.

16

R. R. Jillela, A. Ross / Pattern Recognition Letters 57 (2015) 4–16

[64] K. Roy, P. Bhattacharya, C. Suen, Iris segmentation using variational level set method, Opt. Lasers Eng. 49 (4) (2011) 578–588. [65] S. Pundlik, D. Woodard, S. Birchfield, Iris segmentation in non-ideal images using graph cuts, Image Vision Comput. 28 (12) (2010) 1671–1681. [66] V. Boddeti, B. Kumar, K. Ramkumar, Improved iris segmentation based on local texture statistics, in: 2011 Conference Record of the Forty Fifth Asilomar Conference on Signals, Systems and Computers (ASILOMAR), 2011, pp. 2147–2151. [67] X. Hu, V. Pauca, R. Plemmons, Iterative Directional Ray based Iris Segmentation for Challenging Periocular Images, in: Z. Sun, J. Lai, X. Chen, T. Tan (Eds.), Biometric Recognition, volume 7098 of Lecture Notes in Computer Science, Springer Berlin Heidelberg, 2011, pp. 91–99. [68] C. Tan, A. Kumar, Automated Segmentation of Iris Images using Visible Wavelength Face Images, in: 2011 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2011, pp. 9–14. [69] W. Sankowski, K. Grabowski, M. Napieralska, M. Zubert, A. Napieralski, Reliable algorithm for iris segmentation in eye image, Image Vis. Comput. 28 (2) (2010) 231–237. [70] R. Labati, F. Scotti, Noisy iris segmentation with boundary regularization and reflections removal, Image Vis. Comput. 28 (2) (2010) 270–277. [71] T. Tan, Z. He, Z. Sun, Efficient and robust segmentation of noisy iris images for non-cooperative iris recognition, Image Vis. Comput. 28 (2) (2010) 223–230. [72] P. Almeida, A knowledge-based approach to the iris segmentation problem, Image Vis. Comput. 28 (2) (2010) 238–245. [73] H. Proenca, G. Santos, Fusing color and shape descriptors in the recognition of degraded iris images acquired at visible wavelengths, Comp. Vis. Image Und. 116 (2) (2012) 167–178. [74] A. Raffei, H. Asmuni, R. Hassan, R. Othman, Feature extraction for different distances of visible reflection iris using multiscale sparse representation of local radon transform, Pattern Recognit. 46 (10) (2013) 2622–2633. [75] M. Oroz, E. Faure, J. Angulo, Robust iris segmentation on uncalibrated noisy images using mathematical morphology, Image Vis. Comput. 28 (2) (2010) 278– 284. [76] M. Frucci, M. Nappi, D. Riccio, Watershed based iris segmentation, in: J. CarrascoOchoa, J. Martinez-Trinidad, J. Rodreguez, G. Sanniti di Baja (Eds.), Pattern Recognition, volume 7914 of Lecture Notes in Computer Science, Springer Berlin, Heidelberg, 2013, pp. 204–212. [77] P. Li, X. Liu, L. Xiao, Q. Song, Robust and accurate iris segmentation in very noisy iris images, Image Vis. Comput. 28 (2) (2010) 246–253.

[78] D. Jeong, J. Hwang, B. Kang, K. Park, C. Won, D. Park, J. Kim, A new iris segmentation method for non-ideal iris images, Image Vis. Comput. 28 (2) (2010) 254–260. [79] A. Uhl, P. Wild, Multi-stage visible wavelength and near infrared iris segmentation framework, in: A. Campilho, M. Kamel (Eds.), Image Analysis and Recognition, volume 7325 of Lecture Notes in Computer Science, Springer, 2012, pp. 1–10. [80] L. Ma, T. Tan, Y. Wang, D. Zhang, Efficient iris recognition by characterizing key local variations, IEEE Trans. Image Process. 13 (6) (2004) 739–750. [81] F. Alonso-Fernandez, P. Tome-Gonzalez, V. Ruiz-Albacete, J. Ortega-Garcia, Iris recognition based on SIFT features, in: International Conference on Biometrics, Identity and Security (BIdS), 2009, pp. 1–8. [82] R. Jillela, A. Ross, V. Boddeti, B. VijayaKumar, X. Hu, R. Plemmons, P. Pauca, An evaluation of iris segmentation algorithms in challenging periocular images, in: K. Bowyer, M. Burge (Eds.), Handbook of Iris Recognition, Springer, 2012. [83] R. Labati, A. Genovese, V. Piuri, F. Scotti, Iris Segmentation: State of the Art and Innovative Methods, volume 37 of Intelligent Systems Reference Library, Springer, 2012. [84] L. Masek, P. Kovesi, MATLAB Source Code for a Biometric Identification System Based on Iris Patterns, Bachelor Thesis, University of Western Australia, 2003. [85] X. Liu, K. Bowyer, P. Flynn, Experiments with an improved iris segmentation algorithm, in: Fourth IEEE Workshop on Automatic Identification Advanced Technologies, IEEE, 2005, pp. 118–123. [86] MICHE, Mobile Iris CHallenge Evaluation (MICHE), Biometric and Image Processing Lab, University of Salerno, 2012. [87] H. Proença, Iris recognition: a method to segment visible wavelength iris images acquired on-the-move and at-a-distance, in: Advances in Visual Computing, Springer, 2008, pp. 731–742. [88] Z. He, T. Tan, Z. Sun, X. Qiu, Toward accurate and fast iris segmentation for iris biometrics, IEEE Trans. Pattern Anal. Mach. Intell. 31 (9) (2009) 1670–1684. [89] V. Gottemukkula, S. Saripalle, S. Tankasala, R. Derakhshani, R. Pasula, A. Ross, Fusing iris and conjunctival vasculature: ocular biometrics in the visible spectrum, in: IEEE Conference on Technologies for Homeland Security (HST), 2012, pp. 150– 155. [90] S. Crihalmeanu, A. Ross, Multispectral scleral patterns for ocular biometric recognition, Pattern Recognit. Lett. 33 (14) (2012) 1860–1869. [91] D. Woodard, S. Pundlik, P. Miller, R. Jillela, A. Ross, On the fusion of periocular and iris biometrics in non-ideal imagery, in: 20th International Conference on Pattern Recognition, IEEE, 2010, pp. 201–204.