The Optical Barcode Detection and Recognition Method Based ... - MDPI

applied sciences Article

The Optical Barcode Detection and Recognition Method Based on Visible Light Communication Using Machine Learning Jingyi Li

and Weipeng Guan *

School of Automation Science and Engineering, South China University of Technology, Guangzhou 510640, China; [email protected] * Correspondence: [email protected] or [email protected]; Tel.: +86-156-0223-8930 Received: 1 October 2018; Accepted: 13 November 2018; Published: 29 November 2018







Abstract: Visible light communication (VLC) has developed rapidly in recent years. VLC has the advantages of high confidentiality, low cost, etc. It could be an effective way to connect online to offline (O2O). In this paper, an RGB-LED-ID detection and recognition method based on VLC using machine learning is proposed. Different from traditional encoding and decoding VLC, we develop a new VLC system with a form of modulation and recognition. We create different features for different LEDs to make it an Optical Barcode (OBC) based on a Complementary Metal-Oxide-Semiconductor (CMOS) senor and a pulse-width modulation (PWM) method. The features are extracted using image processing and then support vector machine (SVM) and artificial neural networks (ANN) are introduced into the scheme, which are employed as a classifier. The experimental results show that the proposed method can provide a huge number of unique LED-IDs with a high LED-ID recognition rate and its performance in dark and distant conditions is significantly better than traditional Quick Response (QR) codes. This is the first time the VLC is used in the field of Internet of Things (IoT) and it is an innovative application of RGB-LED to create features. Furthermore, with the development of camera technology, the number of unique LED-IDs and the maximum identifiable distance would increase. Therefore, this scheme can be used as an effective complement to QR codes in the future. Keywords: visible light communication (VLC); machine learning; CMOS image sensor; image processing; RGB-LED

1. Introduction The Internet of Things (IoT) is an important part of the new generation of information technology. As the name suggests, the IoT technology aims to build a set of networks in which each object is connected. The IoT is widely used in network convergence through intelligent sensing, recognition technology, and ubiquitous computing and other communication sensing technologies [1]. The most widely used form of IoT is the quick response code (QR code) of different objects [2]. However, when the QR code is outdoors, especially in dark and distant conditions, the QR code scanning is difficult to implement. This limits the wider application of 2-D code access technology. Therefore, the market is in urgent need of access technology that can meet the specific environmental needs. The visible light communication (VLC) becomes the preferred technology. A lot of research in the field of VLC has been done in recent years [3–7]. By modulating the frequency of light-emitting diodes (LEDs) up to several hundred MHz, which is undetectable to the human eyes [8,9], this is the basis for using VLC as access technology. Furthermore, there are a lot of advantages of VLC. First, the visible part of the electromagnetic spectrum is unlicensed and unregulated, and LEDs are more energy efficient than other radio transmitters [10]. Second, the line of

Appl. Sci. 2018, 8, 2425; doi:10.3390/app8122425

www.mdpi.com/journal/applsci

Appl. Sci. 2018, 8, 2425

2 of 21

sight (LOS) property ensures secrecy in the process of information transmission [11]. Lastly, it is well known that LEDs have been widely used, so VLC is easy to popularize. It is attractive to make the VLC as a supplement to traditional QR codes due to its advantages. In Ref. [12], a security system that is built using a barcode-based VLC system between smart phones is proposed. The results show that the system achieves a high level of security. However, the transmitter of the system uses LEDs in a smartphone, which leads to uneven light output, and the error rate increases due to the “blooming” effect of image sensors caused by uneven light output. In Ref. [13], Chen et al. use LED panel lights to reduce the possibility of overexposure and avoid the blooming effect of the camera. This method achieves good performance in the symbol error rate (SER) test. However, the effective transmission distance is too short (about 4 cm), and when the distance is large, the data packet may be partially lost. In fact, VLC can be divided into two modes: one is photodiode-based (PD-based) and the other is image-sensor based [14–16]. A PD-based VLC system works by receiving the light intensity of different LEDs. It is low cost but it is not stable. With the interference of the background light or the reflection of the wall, performance would deteriorate. In the VLC system based on an image sensor, the LED projection of the image is taken as the foreground and the ambient light as the background. The image processing method makes it easy to extract the foreground and reduce the interference of ambient light. Therefore, it is different from the method based on PD since ambient light does not directly affect the captured images of the LED projection. In addition, with the popularization of smart phones, VLC based on an image sensor can be conveniently applied. Smartphones generally have cameras with a CMOS sensor but no PD. Therefore, comparing two modes of the VLC system, the image-sensors are more suitable to supplement QR code scanning technology. In our previous work [17], we proposed an LED-ID detection and identification method based on image-sensors-based VLC for indoor visible light positioning (VLP). However, only 1035 unique IDs were implemented under the same brightness condition. This was far from meeting the requirements of the supplementary technique for QR code. Therefore, in this paper, we propose Optical Bar Code (OBC), which is an entirely new VLC system that is different from the traditional 0–1 digital encoding and decoding. We adopted the method of image feature extraction and innovatively used machine learning to build classifiers. We used RGB-LED as the transmitter and modulated each LED with a variable frequency, duty-ratio, and phase difference of three input signals of red (R), green (G), and blue (B) diodes using pulse-width modulation (PWM) method so that different LEDs would have six different features: the number of the R, G, and B image’s bright stripes (3); the area of the LED image; the skewing of the R, G, and B images’ stripes (1); and the ratio of the bright stripe’s width to both the bright and dark stripes’ width (duty-ratio of the bright stripes). Once the LED image is captured by a CMOS image sensor, an image processing method is used to extract the features of RGB-LED-ID. In order to use the extracted features to identify the LED-ID, a linear support vector machine (SVM) and a Back Propagation (BP) neural network are used. By off-line training for the classifiers and online recognition of LED-ID, the system proposed can detect and identify LED-ID well. As the experimental result shows, the proposed method could label roughly 1.05 × 107 unique LED-ID with more than 95% recognition. The maximum distance could reach to 6 m, which is greater than the QR code and scheme in Ref. [13]. It can meet that requirement of a supplementary technique for QR codes. Besides, with the upgrading of camera technology, the number of the recognizable LEDs would also increase and LEDs with larger luminous areas can significantly increase the maximum distance. As an LED-to-camera communication technology, this is the first application of VLC in the field of IoT. The OBC we proposed can be used in many scenes. In the field of mobile payment, the security of payment can be increased. Furthermore, it is convenient to use in dark and distant conditions, reducing the queuing time at the time of payment. In the outdoor advertising field, the OBC can also be used. Limited by the size of the billboard, the product information displayed on the billboard is simple. If the OBC is used, people interested in the product can scan the billboard through the smartphone to get the details of the product. These are just two examples of OBC applications, but many more are available.

Appl. Sci. 2018, 8, x FOR PEER REVIEW

3 of 21

smartphone to get the details of the product. These are just two examples of OBC applications, but many more available. Appl. Sci. 2018, are 8, 2425 3 of 21 The remainder of this paper is organized as follows: The background is described in Section 2. The system principle is in Section 3. Experiments and analysis are shown in Section 4. Finally, in The5,remainder of this paper is organized as follows: The background is described in Section 2. Section the conclusion is provided. The system principle is in Section 3. Experiments and analysis are shown in Section 4. Finally, in Background Section 5, the conclusion is provided. 2. 2. Background 2.1. Using a CMOS Sensor in VLC 2.1. Using a CMOS Sensor in VLC 2.1.1. Rolling Shutter Mechanism 2.1.1.For Rolling Shutter Mechanism a Charge Coupled Device (CCD) sensor, the whole sensor is exposed simultaneously, and the data of all pixels are recorded simultaneously. This sensor mechanism is called the global and shutter For a Charge Coupled Device (CCD) sensor, the whole is exposed simultaneously, the mechanism of CCD is illustrated in Figure 1a. On the other a CMOS sensor, data of all pixels are sensors recordedwhich simultaneously. This mechanism is called the hand, globalfor shutter mechanism each rowsensors in the which CMOSissensor is activated whichhand, means the data readout and of CCD illustrated in Figuresequentially, 1a. On the other forthat a CMOS sensor, each row exposure are performed row by row. When the exposure of the row is over, the read out for a row of in the CMOS sensor is activated sequentially, which means that the data readout and exposure are data is given immediately. Finally, scanlines captured different times arefor combined form performed row by row. When the all exposure of the row isatover, the read out a row oftodata is images. This is the mechanism of the CMOS sensor we learned of from Refs. [18–20], which is called given immediately. Finally, all scanlines captured at different times are combined to form images. the shutter mechanism. The rolling mechanism CMOS sensors is illustrated in Thisrolling is the mechanism of the CMOS sensor shutter we learned of fromofRefs. [18–20], which is called the Figure 1b. Themechanism. rolling shutter mechanism the CMOSofsensor, and closes during rolling shutter The rolling shutterofmechanism CMOS which sensorsopens is illustrated in Figure 1b. exposure to LED light, will lead to CMOS image sensor captures on the bright stripes and The rolling shutter mechanism of the CMOS sensor, which opens and closes during exposure to dark LED stripes. light, will lead to CMOS image sensor captures on the bright stripes and dark stripes. Row

Row

Frame Rate

Frame Rate

Read-out time

Exposure time

Exposure time

Time

Time

(a)

Read-out time

(b)

Figure 1. 1. Sketch Sketchmaps maps global shutter of CCD the CCD sensor, and (b) rolling of the Figure of of thethe (a) (a) global shutter of the sensor, and (b) rolling shuttershutter of the CMOS CMOS sensor. sensor.

2.1.2. Camera Requirement 2.1.2. Camera Requirement The camera has many performance indicators that affect the images they capture. When a camera Theascamera hasinmany performance that affectcharacteristics the images they capture. time When a is used a receiver the VLC system, theindicators two most important are exposure and camera is used as a receiver in the VLC system, the two most important characteristics are exposure International Organization for Standardization (ISO). time and International Organization for Standardization (ISO). • Exposure Time • Exposure Time The exposure time is a time horizon that the camera shutter is open for to allow light into the The exposure time is a time horizon that the camera shutter is open for to allow light into the photodiode matrix. The exposure duration determines how long a pixel collects photons. Pixels in the photodiode matrix. The exposure duration determines how long a pixel collects photons. Pixels in light irradiation accumulate charge until saturation. the light irradiation accumulate charge until saturation. LED is not only used as a transmitter, but also in many cases as a light source in the indoor environment, so the intensity of the LED is very high and the charge of the pixels is saturated in a short time. When the pixel charge is saturated, the charge will spill over to the nearest pixel if it continues

Appl. Sci. 2018, 8, 2425

4 of 21

to irradiate, resulting in the accumulation and saturation of the charge in the near pixel. Therefore, if the exposure time is too long, the width of the bright stripe will increase, and the reception will be affected. Especially when we use RGB-LED, this effect is more obvious. Incorrect doping of color will greatly increase the error of detection and recognition.

•

ISO

In digital cameras, ISO indicates the speed of photosensitivity of the CCD or CMOS photosensitive elements. The higher the ISO value, the stronger the photosensitive ability of the photosensitive component. The higher the ISO, the fewer photons that a pixel needs to reach saturation. This means that the probability of the saturation pixel in the same exposure time increases, resulting in the width of the bright stripe in the captured image not normally increasing such that recognition rate will be reduced. 2.2. Avoiding Flicker and Keeping the Light White Human eyes perceive objects around them by projecting images onto the retina. The brain needs a certain amount of time to process the received pictures, and our visual system cannot respond to the stimulus of the changes in front of the eyes, causing a delay in the observation of brightness or color changes. If intermittent stimuli appear below a certain frequency, our visual system can perceive changes (the effect is called flicker). Above the frequency at which the flicker effect ceases is called the flicker fusion threshold [21]. The flicker frequency of our LEDs are above 100 Hz, greater than the threshold, so the naked eye cannot see the flicker. The most common and universal light in our lives is white light. The LEDs we are discussing here are used for both communication and general lighting in many cases. In general lighting, people generally have difficulty in accepting colored lights. Therefore, it is necessary to discuss the maintenance of white light in color modulation. In the process of perceiving objects around us, our eyes go through a process called the sum of time. In the process, the eye accumulates photons until it is saturated. This period is called the tc : ζ = I ∗ t(t ≤ tc )

(1)

I is the intensity of the stimulus (LED project’s color stimulus in our case), and tc is the critical duration. With the increase of t, once the threshold is reached, additional stimulus light will not be perceived by the visual system. The perception of color by the human eye is the average of the stimulus within the critical duration. According to Bloch’s law, the color Ψ is

R ψ=

IR (t)dt +

R

IG (t)dt + t

R

IB (t)dt

(2)

where IR (t), IG (t), and IB (t) are the intensity functions of red, green, and blue light, respectively. For RGB-LED, if the three colors are emitted at the same ratio, the human eye will perceive the white light in the critical duration for the sum of the time. This is illustrated in Figure 2.

Appl. Sci. 2018, 8, 2425

5 of 21

Appl. Sci. 2018, 8, x FOR PEER REVIEW

5 of 21

Transmitter sends out equal amount of red, green and blue light

…

RGB-LED t

Critical duration

Average of accumulated color within one critical duration

Human eye

White light

t Figure 2. Average color perceived by the eye in a critical duration. Figure 2. Average color perceived by the eye in a critical duration.

3. System Principle 3. System Principle 3.1. Method Overview 3.1. Method Overview We provide a brief overview of the system first. In the offline training phase, at the transmitter, provide a brief overview of thetosystem the green, offlineand training Pulse We Width Modulation (PWM)is used controlfirst. threeInred, blue phase, diodes at ofthe thetransmitter, RGB-LED. Pulse Width Modulation (PWM)is used to control three red, green, and blue diodes of the RGB-LED. Then, the light emitted by the red, green, and blue diodes is mixed in RGB-LED light bulbs to become Then,light. the light emitted by the red, green, and blue is mixed in RGB-LED lightisbulbs to become white At the receiver, smartphone cameras candiodes be used. The LED project image captured with white light. At the receiver, smartphone cameras can be used. The LED project image is captured stripes by the CMOS sensor because of the rolling shutter mechanism. Due to special modulation, with stripes by the stripe CMOS sensor because of features, the rolling shutter mechanism. Due to special different LED project images have different so we can call them the LED-IDs. Then, modulation, LED project stripe imagesfeatures have different features, so weand can extracting. call them the we use imagedifferent processing technology to extract to facilitate counting InLEDthe IDs. Then, we use image processing technology to extract features to facilitate counting process of image processing, the color LED projection is separated into three channels: R, G, andand B. extracting. In the process of image processing, the color LED projection is separated into three Next, according to our selected features, feature data will be extracted through image processing channels: R, G, and B.edge Next, accordingetc. to our selected features, feature data feature will be extracted through such as binarization, detection, Then, according to the acquired data, a machine image processing such as binarization, edge detection, etc. Then, according to the acquired feature learning algorithm is applied to build the classifier. At last, exploring the boundary and completing data, a machineoflearning algorithm the construction the LED-ID libraryisisapplied done. to build the classifier. At last, exploring the boundary andIn completing the construction of the LED-ID library done. the online training phase: First, when setting upisthe system, select the appropriate ID from In the online training phase: First, when setting up the system, select appropriate ID from the LED-ID library according to the required number and modulate the LEDthe with the parameters of thepairs LED-ID library according topage the required and modulate the LED with parameters ID in the library. The web or othernumber information corresponding to the IDthe is stored in theof ID pairs in the library. The web page or other information corresponding to the ID is stored from in the cloud server. Once the user’s achievement is recognized, the cloud content can be downloaded cloud server.The Once the user’sdiagram achievement recognized, the in cloud content can is bemore, downloaded from the network. schematic of theismethod is given Figure 3. What in practical the network.the Thereceiving schematic diagram of the method is given camera, in Figurebut 3. also Whatanison-board more, in camera, practical applications, end is not necessarily a cellphone aapplications, robot camera,the etc.receiving end is not necessarily a cellphone camera, but also an on-board camera, a robot camera, etc. In the next part of this section, we will select six representative RGB-LED-IDs to illustrate signal modulation, image processing, classifier establishment, and the online identification of our system.

Appl. Sci. 2018, 8, 2425

6 of 21

Appl. Sci. 2018, 8, x FOR PEER REVIEW

6 of 21

Assigning

Camera RGB-LED

Hybrid optical

PWM Control Capture

Capture Identification

Get Features Machine Learning Image process

Separate

Pre-download classifier

Access

Provide more information

Building a LED-ID Library Storage Cloud server

Online recognition

Offline training

Figure 3. 3. System Figure System overview. overview.

3.2.InLED the Modulation next part of this section, we will select six representative RGB-LED-IDs to illustrate signal modulation, processing, (PWM) classifier the online identification of our system. Pulse image width modulation is establishment, a very effectiveand technique to control analog circuits by the digital transmission of microprocessors. It is widely used in many fields from measurement and 3.2. LED Modulation communication to power control and transformation. For RGB-LEDs, three pulse signals generated to technique control theto red, green,analog and blue diodes Pulse width modulation (PWM) is aare very effective control circuits byofthe the RGB-LEDs, respectively, using PWM.ItThen, the three will fields produce a cumulative color. digital transmission of microprocessors. is widely useddiodes in many from measurement and According to the rolling control shutter and mechanism of a CMOS sensor, the number of bright stripes on the communication to power transformation. LED could be changed changing modulation frequency thegreen, LED, and duty-ratio Forproject RGB-LEDs, three pulsebysignals arethe generated to control theof red, andthe blue diodes of of bright stripes could be changed by changing the duty-ratio of the pulse. In addition, by controlling the RGB-LEDs, respectively, using PWM. Then, the three diodes will produce a cumulative color. the start time ofrolling the three pulsesmechanism that input red, and blue diodes in a scheme, the stripes three PWM According to the shutter of agreen, CMOS sensor, the number of bright on the signals have a phase difference. At the receiver, the captured LED images are separated into three R, LED project could be changed by changing the modulation frequency of the LED, and the duty-ratio G, and B images, and the stripes of three different colors in the same exposure period are different. of bright stripes could be changed by changing the duty-ratio of the pulse. In addition, by controlling The images of different LEDs modulated by PWM with different frequency, duty-ratio, and the start time of the three pulses that input red, green, and blue diodes in a scheme, the three PWM phase difference are shown in Figure 4 and Figure 5. Furthermore, their modulated schemes at the signals have a phase difference. At the receiver, the captured LED images are separated into three R, G, emitter are shown in Figure 6. and B images, and the stripes of three different colors in the same exposure period are different. In Figure 6, the phase difference and duty-ratio of R, G, and B signals in (a) and (b) are the same The images of different LEDs modulated by PWM with different frequency, duty-ratio, and phase but the frequencies are different. In (c) and (d), the three signals have the same frequency and phase difference arebut shown 4 and 5. Furthermore, their the emitter difference, they in areFigures different in duty-ratio. Examples (e) modulated and (f) haveschemes the sameatfrequency andare shown in Figure duty-ratio, but 6. the phase difference is different. The naked eye cannot clearly distinguish the specific In Figure 6, the difference and duty-ratio R, G, and B signals (a) andthrough (b) are the same differences of thesephase color images in some cases, so weofextract the features of in LED-IDs image butprocessing the frequencies are different. In (c) and (d), the three signals have the same frequency and phase for the convenience of the show, which is shown in Figure 4. difference, but they are different in duty-ratio. Examples (e) and (f) have the same frequency and duty-ratio, but the phase difference is different. The naked eye cannot clearly distinguish the specific differences of these color images in some cases, so we extract the features of LED-IDs through image processing for the convenience of the show, which is shown in Figure 4. As for (f) and (g), they are different from others in that the frequency of the R, G, and B signals are different from each other in a scheme that is shown in Figure 5.

Appl. Sci. 2018, 8, x FOR PEER REVIEW Appl. Sci. 2018, 8, 2425

7 of 21 7 of 21

Appl. Sci. 2018, 8, x FOR PEER REVIEW

7 of 21

LED1 LED1 LED2 LED2 LED3 LED3 LED4 LED4

Figure 4. Different LED images captured using a CMOS sensor (1).

As for (f) and (g), they are different from others in that the frequency of the R, G, and B signals Figure 4. Different LED that images are different from each other in a scheme is shown inusing Figure 5. captured a CMOS sensor (1). As for (f) and (g), they are different from others in that the frequency of the R, G, and B signals are different from each other in a scheme that is shown in Figure 5.

LED5 LED5 LED6 LED6 Figure Figure 5. 5. Different Different LED LED images images captured captured using using aa CMOS CMOS sensor sensor (2). (2).

Figure 5. Different LED images captured using a CMOS sensor (2).

Appl. Sci. 2018, 8, 2425 Appl. Sci. 2018, 8, x FOR PEER REVIEW

T1

8 of 21 8 of 21

LED1

LED2

T2

R

R

G

G

B

B (b)

(a)

LED1

LED3

R

R

G

G

B

B (c)

(d)

LED4

LED1

R

R

G

G

B

B (f)

(e)

LED5

LED6

R

R

G

G

B

B (g)

(h)

LED1~LED6 Figure 6. The modulated scheme. (a)~(h) (a)–(h) corresponds to the modulation sequence of LED1~LED6.

3.3. RGB-LED-ID Feature Extraction and Selection As the distance between the LED and camera increases, the area of LED projection on the CMOS 3.3. RGB-LED-ID Feature and Selection sensor will decrease. For Extraction the same LED project, the width of the stripe will not change due to the fixed scanning frequency of the sensor. As the of the LED projection onLED the CMOS sensor decreases, As the distance between the LED andarea camera increases, the area of projection on the CMOS the number of stripes will 7). The the relative position of the beto easily sensor will decrease. For decrease the same(Figure LED project, width of the relations stripe will not stripes changecan due the obtained after decomposing according theof three of R, G, on andthe B, and it does not fixed scanning frequency ofthe theimage sensor. As the to area the channels LED projection CMOS sensor change with distance. decreases, the number of stripes will decrease (Figure 7). The relative position relations of the stripes Therefore, in this after paper,decomposing we choose six the area of theR,number of can be easily obtained thefeatures: image according tothe theLED threeprojection, channels of G, and B, eachitcolor’s stripes, duty-ratio of the bright stripe, and the defined skewing coefficients. The steps of and does not change with distance. the RGB-LED-ID feature extraction and selection are given in Figure 8.

Appl. Sci. 2018, 8, x FOR PEER REVIEW

9 of 21

Appl. Sci. 2018, 8, 2425 Appl. Sci. 2018, 8, x FOR PEER REVIEW

9 of 21

9 of 21

Figure 7. The number of stripes varies with distance.

Therefore, in this paper, we choose six features: the area of the LED projection, the number of each color’s stripes, duty-ratio of the bright stripe, and the defined skewing coefficients. The steps of Figure7.7.The The number of stripes varies distance. the RGB-LED-ID feature Figure extraction and selection are given inwith Figure 8. number of stripes varies with distance. Therefore, in this paper, we choose six features: Original the area of the LED projection, the number of image each color’s stripes, duty-ratio of the bright stripe, and the defined skewing coefficients. The steps of the RGB-LED-ID feature extraction and selection are given in Figure 8. Gray scaleOriginal processingimage Binarization processing Gray scale

RGB separation

processing Closing operation Binarization processing Edge detection Closing

RGB separation

operation Image Edge segmentation detection Get the area of the LED Projection

Get the skewing Get the duty-ratio coefficient Image of bright stripes segmentation

Get stripe number

Figure8.8.Processing Processing of of LED LED features Figure features extraction. extraction. Get the area of the Get the skewing Get the duty-ratio Get stripe 3.3.1. Step 1: 1: Edge Detection Segmentation 3.3.1. Step Edge Detectionand andLED LEDcoefficient Segmentation of bright stripes LED Projection number The CMOS sensor may capture an image containing multiple LEDs or an LEDfar far away from The CMOS sensor may capture an image containing multiple LEDs or an LED away from thethe Figure 8. Processing of LED features extraction. receiver, need segmentout outthe theregion region of of interest that onon forfor receiver, so so wewe need totosegment that the theLED LEDproject projectisisjust justcentered centered further feature extraction. To achieve this goal, the whole image is transformed into a gray image further feature extraction. To achieve this goal, the whole image is transformed into a gray image first. 3.3.1. Step 1: Edge Detection and LED Segmentation first. After that, the threshold-based method is used to get binary images. Then, to make the LED After that, the threshold-based method is used to get binary images. Then, to make the LED project The CMOS sensor capture an imagea containing multiple LEDsoperation orused, an LED awaydetection fromedge the is project region form a may connected morphologically-closed isfarused, and region form a connected domain, a domain, morphologically-closed operation is and edge receiver, so we need to segment out the region of interest that the LED project is just centered on for detection done using a Canny edge of the edge of the is done using aisCanny operator to getoperator the edgeto ofget thethe LED. Next, theLED. edge Next, of thethe LED is used to LED intercept further feature extraction. To achieve this goal, the whole image is transformed into a gray image used to intercept the region of interest in the original image. Lastly, the image is separated according the region of interest in the original image. Lastly, the image is separated according to the R, G, and B first. threshold-based method used to get binary images. as Then, to make the LED to theAfter R, G,that, and Bthe three channels. At this point,ispreprocessing is completed, shown in Figure 9. three channels. At this point, preprocessing is completed, as shown in Figure 9. project region form a connected domain, a morphologically-closed operation is used, and edge detection is done using a Canny operator to get the edge of the LED. Next, the edge of the LED is used to intercept the region of interest in the original image. Lastly, the image is separated according to the R, G, and B three channels. At this point, preprocessing is completed, as shown in Figure 9.

Appl. Sci. 2018, 8, 2425 Appl. Sci. 2018, 8, x FOR PEER REVIEW Appl. Sci. 2018, 8, x FOR PEER REVIEW

10 of 21 10 of 21 10 of 21

Figure preprocessing. Figure 9. 9. Image Image preprocessing. Figure 9. Image preprocessing.

3.3.2. Step 2: Get the Area of the LED Project 3.3.2. Step 2: Get the Area of the LED Project 3.3.2. Step 2: Get the Area of the LED Project In In thethe preprocessing, Therefore,ititisiseasy easytotoget get the area preprocessing,we weget getthe theedge edge of of LED LED project. project. Therefore, the area of of thethe In the preprocessing, we get the edge of LED project. Therefore, it is easy to get the area of the LED project. AsAs shown inin Figure on the the edge edge is LED project. shown Figure10, 10,assuming assumingthat thatthe thecoordinates coordinatesof ofthe the leftmost leftmost point point on LED Asthe shown in Figure 10, assuming that the coordinates of the leftmost point on the edge e ,isye(𝑥 rightmost coordinate is x , y , then the diameter of the LED projection is: ( x1project. ), and ( ) 2 , 𝑦), and the rightmost coordinate is (𝑥 , 𝑦), then the diameter of the LED projection is: is (𝑥 , 𝑦), and the rightmost coordinate is (𝑥 , 𝑦), then the diameter of the LED projection is: (3) (3) 𝑑 =𝑥 −𝑥 (3) 𝑑 = 𝑥d =−x𝑥2 − x1 Thus, the area of the LED projection can be calculated using: Thus, thethe areaarea of the LEDLED projection can be using:using: Thus, of the projection cancalculated be calculated  𝑑 𝑠 =𝑑 𝜋 d 2 𝑠=𝜋 s=π 2 2 2

(4)

(4)

(4)

dd

x1

x1

x2

x2

Figure 10. The diameter of LED project. Figure 10.diameter The diameter LED project. Figure 10. The of LEDofproject.

3.3.3. Step Get theDuty-Ratio Duty-Ratioof ofthe the Bright Bright Stripe Stripe Counting 3.3.3. Step 3: 3:Get Counting 3.3.3. Step 3: Get the the Duty-Ratio of the Bright Stripe Counting In a binary image,each each pixelvalue value is is 11 or or 0. Set Set up and detection vector a binary image, up aa vertical vertical anddownward downward detection In In a binary image, each pixelpixel value is 1 or 0. Set0.up a vertical and downward detection vector vector through the image center. If 1 is detected, the pixel is in the bright stripe. Conversely, if 0 is detected, through image center. If detected, 1 is detected, the pixel is in the bright Conversely, if 0 is detected, through thethe image center. If 1 is the pixel is in the bright stripe.stripe. Conversely, if 0 is detected, the pixel is in the dark fringe. Furthermore, record the number of 1 and 0 detected during the test. It the pixel is in the dark fringe. Furthermore, record the number of 1 and 0 detected during theIttest. It is the pixel is in the dark fringe. Furthermore, record the number of 1 and 0 detected during the test.

Appl. Sci. 2018, 8, 2425

11 of 21

Appl. Sci. 2018, 8, x FOR PEER REVIEW

11 of 21

easy to to getget thethe number of R, B image stripes in this The number of R, G, B and image is easy number ofG, R,and G, and B image stripes in process. this process. The number of and R, G, B stripes is three features in our case, which is shown in Figure 11. image stripes is three features in our case, which is shown in Figure 11.

Figure Figure 11. 11. Get Get the the duty-ratio duty-ratio of of the the bright bright stripes stripes and and the the number number of of stripes. stripes.

3.3.4. Step Step 4: 4: Get Get the the Skewing Skewing 3.3.4. The three three images system, which is The images obtained obtainedfrom fromthe theseparation separationare areplaced placedininthe thesame samecoordinate coordinate system, which shown in in Figure 12.12. SetSet a vertical andand downward vector passing through the image center. In the is shown Figure a vertical downward vector passing through the image center. Infirst the image (R), the ordinate of the center line of the ith stripe that intersects this vector is d . In the second image 1 vector is 𝑑 . In the first image (R), the ordinate of the center line of the 𝑖th stripe that intersects this (G), the image ordinate of the the ordinate center line thecenter last stripe that this vector is d2 inthis (0, dvector 1 ). In the second (G), ofof the line of theintersects last stripe that intersects is 𝑑third in image thethird ordinate of the line of the ith stripe vector d3 in (0, d2this ). Order: (0, 𝑑 ).(B), In the image (B),center the ordinate of the centerthat lineintersects of the 𝑖ththis stripe thatisintersects vector

is 𝑑 in (0, 𝑑 ). Order: λi =

d1 − d2 d1 − d3

(5)

𝜆 = (5) In order to reduce the error, again, take the vector vertical downward. In the first image (R), the ordinate center of again, the jthtake stripe intersects vector is In d30 .the Infirst the image second(R), image In order of to the reduce theline error, thethat vector verticalthis downward. the 0 0 (G), the ordinate of the center line of the first stripe that intersects this vector is d in (0, d ). In the third ordinate of the center line of the 𝑗th stripe that intersects this vector is 𝑑 . In the second 2 3 image (G), image (B), theofordinate of the of stripe the first stripe that intersects this is vector is (0, d10 in d20 the ). Order: the ordinate the center linecenter of theline first that intersects this vector 𝑑 in 𝑑 (0, ). In third image (B), the ordinate of the center line of the first stripe that intersects this vector is 𝑑 in (0, 𝑑 ). d30 − d20 Order: (6) λj = 0 d3 − d10 The skewing coefficient is defined as:

𝜆 =

(6)

m n The skewing coefficient is definedkas:= ∑i=2 λi + ∑ j=2 λ j (7) λ m+n ∑ 𝜆 +∑ 𝜆 (7) 𝑘 = In general, m, n > 5 in the above equation, but 𝑚 +if𝑛the number of stripes is small due to a low frequency or a long m, above n and i,equation, j can be reduced simultaneously. In general, m, ndistance, > 5 in the but if the number of stripes is small due to a low Moreover, for LED5 and m, LED6, thei,number image simultaneously. stripes separated by the receiver is different frequency or a long distance, n and j can be of reduced due to the different frequencies of the channels at the transmitter. periodic chaos Moreover, for LED5 and LED6, thethree number of image stripes separatedInbythis thecase, receiver is different exists the phase difference of is extracted. Therefore, the system composed of a transmitter like due towhen the different frequencies the three channels at in the transmitter. In this case, periodic chaos LED5 or LED6, only five effective features except skewing can be extracted. exists when the phase difference is extracted. Therefore, in the system composed of a transmitter like above only is the process of feature However, is a situation that needs to be LED5The or LED6, five effective featuresextraction. except skewing can bethere extracted. considered: when theprocess naturaloflight or other non-system lightthere in the is very The above is the feature extraction. However, is environment a situation that needsstrong, to be the contrast between the bright and dark stripes of the LED project in the image captured by the sensor considered: when the natural light or other non-system light in the environment is very strong, the will be reduced. This problem can be solved based on our previous work [16]. An algorithm named contrast between the bright and dark stripes of the LED project in the image captured by the sensor contrast limited adaptive histogram (CLAHE) was proposed Ref. It can enhance will be reduced. This problem can beequalization solved based on our previous work in [16]. An[16]. algorithm named the contrast of the image itself when there is interference from external light. contrast limited adaptive histogram equalization (CLAHE) was proposed in Ref. [16]. It can enhance

the contrast of the image itself when there is interference from external light.

Appl. Sci. 2018, 8, 2425

12 of 21

Appl. Sci. 2018, 8, x FOR PEER REVIEW

12 of 21

Figure12. 12.Calculate Calculate the Figure theskewing. skewing.

3.4. LED Recognition 3.4. LED Recognition Machine Learning Machine Learning We extracted features, thefeature featurespace space is is six-dimensional space; thatthat is, all set data We extracted six six features, soso the six-dimensional space; is,test all test set data are in a six-dimensional space (data from a system similar to LED5 and LED6 is five-dimensional). are in a six-dimensional space (data from a system similar to LED5 and LED6 is five-dimensional). Furthermore, in our case, for two different LEDs, data is linearly separable. To achieve the recognition Furthermore, in our case, for two different LEDs, data is linearly separable. To achieve the recognition of RGB-LED-IDs, in the machine learning field, there are two typical linear classifiers that can be used of RGB-LED-IDs, in thesupport machine learning field, thereand areatwo linearnetwork classifiers that can be used in this case: a linear vector machine (SVM) backtypical propagation (BP-network). in this case: a linear support vector machine (SVM) and a back propagation network Select training samples and test samples: 150 groups of LED-ID images taken (BP-network). at different Select training samples where and test 150 groups LED-ID imagesand taken different distances distances were selected, 120samples: groups were used asof training samples, theatother 30 groups usedwhere as test samples. Partwere of theused inputasdata and classification arethe shown in 30 Table 1. were were selected, 120 groups training samples, and other groups were used as test samples. Part of the input data and classification are shown in Table 1. • Support Vector Machine

•

Support Machine SVMVector is based on the concept of decision planes that define decision boundaries. A decision plane is one that separates between a set of objects having different class memberships. For the linearly

SVM is based on the concept of decision planes that define decision boundaries. A decision separable samples, the optimal classification hyperplane can separate the instances into two plane is one that separates between a set of objects having different class memberships. For the categories [22]. We briefly introduce the mathematical principles of SVM as follows: For a two-class linearly classification hyperplane separate instances case,separable let the 𝑥 , isamples, = 1,2, …, n,the (n =optimal 4 in our case) be the feature vectors of can the training setthe x. The general into two categories [22]. We briefly introduce the mathematical principles of SVM as follows: For a two-class SVM classification can be described as a mathematical optimization problem: 1 the feature vectors of the training set x. The general case, let the xi , i = 1,2, . . . , n, (n = 4 in our case) be arg min ‖𝑤‖ , 2 (8) SVM classification can be described as a mathematical optimization problem: s. t. 𝑦 (𝑤 𝑥 + 𝑏) ≥ 1 1 classifies Once more, the goal is to find a hyperplane all the training vectors correctly: k w k2 argminthat w,b 2

T =𝑤 𝑥+𝑏 s.t. y𝑓(𝑥) i w xi + b ≥ 1

(9)




(8)

where w determines the direction of hyperplane, which has the same dimension of x, 𝑤

Once more, goal is of to w, find thatexact classifies allof the vectors correctly: represents thethe transpose anda bhyperplane determines the position thetraining hyperplane in space. Such a hyperplane is not unique, therefore we define the geometrical margin as:

f (x) = wT x + b

𝑧=

(9)

|𝑓(𝑥)|

(10)

𝑤

where w determines the direction of hyperplane, which has the same dimension of x, w T represents which represents the geometrical distance from the samples to the hyperplane. To get the the transpose of w, and b determines the exact position of the hyperplane in space. Such a hyperplane optimal hyperplane, we aim to maximize Z. Since the value of |𝑓(𝑥)| can be changed by scaling w is not unique, therefore we define the geometrical margin as: and b, when we make it equal to 1, the solution we have to find is to search the maximum

.

‖ ‖

With the help of a Laplace operator, the problem is: | f ( x )|

z=

𝑓(𝑥) = ∑

𝑦 𝛼w⟨𝑥, 𝑥 ⟩ + 𝑏

(10) (11)

which represents the geometrical distance from the samples to the hyperplane. To get the optimal hyperplane, we aim to maximize Z. Since the value of | f ( x )| can be changed by scaling w and b, when we make it equal to 1, the solution we have to find is to search the maximum kw1 k .

Appl. Sci. 2018, 8, 2425

13 of 21

With the help of a Laplace operator, the problem is: Appl. Sci. 2018, 8, x FOR PEER REVIEW

f (x) =

n

∑i=1 yi αi hx, xi i + b

(11)

13 of 21

where yi is the corresponding class indicator of xi (+1 for w1 , −1 for w2 ), αi is the Lagrange multiplier, and h x, xi i represents the vector inner product. A schematic diagram of the SVM classification is shown in Figure 13. Table 1. Part of the input data and classification. Data Table 1. PartInput of the input data and classification. Area (Pixel2) 290,333 1,089,884 Area (Pixel2) 2,949,832 290,333 281,563 1,089,884 1,205,794 2,949,832 3,048,741 269,702 281,563 1,075,131 1,205,794 3,056,484 3,048,741 298,024 269,702 1,270,761 1,075,131 3,218,487 3,056,484 307,778 298,024 950,331 1,270,761 3,166,775 3,218,487 266,033 1,027,878 307,778 2,853,222 950,331

3,166,775

Num R Num G Num B Du-Ra (%) Skewing 2 2 2 0.50000 0.357542 Input Data 5 4 4 0.50086 0.352153 Num R Num G Num B Du-Ra (%) Skewing 7 7 7 0.50184 0.347941 2 2 2 0.50000 0.357542 4 4 4 0.50056 0.348947 5 4 0.50086 0.352153 9 10 4 8 0.50149 0.349574 7 7 0.50184 0.347941 18 19 7 19 0.50045 0.358974 2 2 3 0.25005 0.350205 4 4 4 0.50056 0.348947 4 4 6 0.25086 0.354964 9 10 8 0.50149 0.349574 7 7 19 8 0.25464 0.358711 18 19 0.50045 0.358974 2 2 2 0.50058 0.684783 2 2 3 0.25005 0.350205 4 4 5 0.50264 0.676787 4 4 6 0.25086 0.354964 7 7 7 0.50484 0.673067 7 7 8 0.25464 0.358711 9 4 2 0.50154 0 2 2 2 0.50058 0.684783 16 8 4 0.50156 0 4 4 0.50264 0.676787 29 14 5 8 0.51899 0 7 7 7 0.50484 0.673067 7 4 3 0.50294 0 13 11 7 0.50154 0 9 4 2 0.50154 0 20 16 4 10 0.50000 16 8 0.50156 00

29

14

8

0.51899

Classification

Classification LED1

LED1 LED2

LED3 LED2

LED4

LED3

LED5

LED4 LED6

LED5

0

where 𝑦 266,033 is the corresponding class 4indicator of3 xi (+1 for w 1, −1 for w2), 𝛼 is the Lagrange multiplier, 7 0.50294 0 ⟩ represents the13vector inner and ⟨𝑥, 𝑥1,027,878 is 11 product. 7 A schematic 0.50154diagram of 0 the SVM classification LED6 20 16 10 0.50000 0 shown in2,853,222 Figure 13.

x2

Support Vector Class Two

Class One

Optimal Hyperplane

x1

Figure 13. An example of a linearly separable two-class problem with with two two possible possible linear linear classifiers. classifiers.

••

BP-Network BP-Network

Artificial Artificial neural neural networks networks (ANNs) (ANNs) also also known known as as neural neural networks networks (NNs), (NNs), which which are are based based on on the physiological research results regarding the brain, is a mathematical model of a simulation the physiological research results regarding the brain, is a mathematical model of a simulation ofofa abiological biologicalneural neural network process information, simulate certain mechanisms brain, network to to process information, to to simulate certain mechanisms of of thethe brain, to to achieve some aspect of function. Self-organizing competitive neural networks and BP neural achieve some aspect of function. Self-organizing competitive neural networks and BP neural networks networks are are usually usually used used for for classification classification and and pattern pattern recognition recognition [23]. [23]. In In our our case, case, the the BP BP neural neural network classifier is designed to classify the different LED-IDs. network classifier is designed to classify the different LED-IDs.

Initialize the network: A random initial value of connection weights and thresholds between the nodes of each layer is established between (−1, 1), which is usually generated by random functions. Given an input vector 𝑋 = (𝑥 , 𝑥 , … , 𝑥 ) and expected output vector 𝑃 = (𝑝 , 𝑝 , … , 𝑝 ) , for network learning, the number of neurons in the input layer and output layer is m and n

Appl. Sci. 2018, 8, 2425

14 of 21

Initialize the network: A random initial value of connection weights and thresholds between the nodes of each layer is established between (−1, 1), which is usually generated by random functions.  T k Given an input vector X k = x1k , x2k , . . . , xm and expected output vector Pk = ( p1 , p2 , . . . , pn ) T , for network learning, the number of neurons in the input layer and output layer is m and n respectively. The number of hidden neurons has great influence on the prediction accuracy of a BP neural network. At present, l = 2n + 1 is usually used to calculate the number of hidden layer nodes. Calculate the output of the hidden layer and output layer using:

Appl. Sci. 2018, 8, x FOR PEER REVIEW

yj = f




w x − θ ∑i ji i j = f net j

zl = f



∑i v ji yi − θl



(12) 14 of 21

= f (netl )

(13)

respectively. The number of hidden neurons has great influence on the prediction accuracy of a BP In the formula, y j is the output of hidden layer; w ji is the weights between input and hidden neural network. layer; xi isAtthe inputl =layer; θ jusually is the used threshold of hidden layer; zl is the output present, 2n + 1 is to calculate the number of hidden layer nodes. of output layer; v ji is the Calculate the of the hidden layerlayer; and output using: weights between theoutput hidden and output θl is layer the threshold of output layer; and f is excitation

function of S type, f ( x ) = 1−1e−x . 𝑦 = 𝑓 ∑ 𝑤 𝑥 − 𝜃 = 𝑓(𝑛𝑒𝑡 ) (12) The error of the output node is: 𝑧 = 𝑓 ∑ 𝑣 𝑦 − 𝜃 = 𝑓(𝑛𝑒𝑡 ) (13)  2 input and hidden In the formula, 𝑦 is the output 1 of hidden layer; 𝑤 is the weights between ( t − f ( v f w x − θ ) output layer; 𝑣 is layer; 𝑥 is the input layer; 𝜃 E is = the threshold of hidden layer; 𝑧 is the output of ji i ∑ ∑ ∑ l lj l j i 2 l

(14)

the weights between the hidden and output layer; 𝜃 is the threshold of output layer; and 𝑓 is . excitation functiontof S type, 𝑓(𝑥) = In the formula, l is the expected output value of the output node.

The error of the the output nodeand is: thresholds from the output layer, the error back propagation to By modifying weights modify the weights of each layer (𝑡 − 𝑓(∑ 𝑣network 𝑓(∑ 𝑤 𝑥 error, − 𝜃 )) as follows: 𝐸 =to∑minimize (14) In the formula, 𝑡 is the expected output value of the output node. w ji (k +from 1) =thewoutput η 0 δj0 xthe ji ( k ) +layer, i error back propagation to By modifying the weights and thresholds modify the weights of each layer to minimize network error, as follows: θ j (1) k+ 1) = θ (k) + η 0 δj0 = 𝑤 (𝑘) +j 𝜂 𝛿 𝑥 𝑤 (𝑘 + (15) 0 (𝑘 (𝑘) 𝜃 + 1) = 𝜃 + 𝜂 𝛿 In the formula, η is the learning coefficient, and δ is the node error. (16)

(15) (16)

j

In general, three layers the BPcoefficient, networkand can𝛿 beis done in error. any mode of classification; thus, we chose In the formula, 𝜂 is theof learning the node the three layers of the BPlayers network case.can There are in sixany types ofoffeature data, thus, so n we = 6. According to In general, three of thein BPour network be done mode classification; chose the three layers of the BP network in our case. There are six types of feature data, so n = 6. is structured, l = 2n + 1, the number of neurons in the hidden layer is 13. Therefore, a 6-13-6 BP network According to l = 2n + 1, the number of neurons in the hidden layer is 13. Therefore, a 6-13-6 BP network which is shown in Figure 14. is structured, which is shown in Figure 14.

Input layer

Hidden layer

Ouput layer

LED1

LED project Area

LED2

Strip number R

LED3

Strip number G

LED4

…

Strip number B Duty-ratio of bright stripe

LED5

Skewing Coefficient

LED6

6

13

6

Figure 14. BP network structure of LED-ID classification and recognition. Figure 14. BP network structure of LED-ID classification and recognition.

Appl. Sci. 2018, 8, x FOR PEER REVIEW

15 of 21

4. Experiment and Analysis

Appl. Sci. 2018, 8, 2425

15 of 21

In this section, we elaborate on the experiment and analysis of the framework proposed by us. The RGB-LED we used were normal RGB-LEDs (9W). We programmed them with a laptop and used 4. Analysis anExperiment STM32F407and (Xingyi Electronic Technology Co., Ltd., Guangzhou, China) to transmit signals. The threeInoutput ports of the STM32on respectively controlled the three diodes in the RGB-LED. In the this section, we elaborate the experiment and analysis of the framework proposed by us. receiver subsystem, Samsung’s Galaxy S8(Designed by Samsung, Seongnam, Korea; in The RGB-LED we usedawere normal RGB-LEDs (9W). We programmed them with a laptop andMade used an China, Huizhou Samsung Electronics Co., Ltd., Huizhou) camera was used to capture the image STM32F407 (Xingyi Electronic Technology Co., Ltd., Guangzhou, China) to transmit signals. The three data, and then the STM32 captured image wascontrolled processedthe by three usingdiodes the MATLAB software In (Developed by output ports of the respectively in the RGB-LED. the receiver MathWorks, Natick, MA, USA)S8(Designed in a computer with the proposed The relevant parameters subsystem, a Samsung’s Galaxy by Samsung, Seongnam,system. Korea; Made in China, Huizhou are shown in Table 2. Samsung Electronics Co., Ltd., Huizhou) camera was used to capture the image data, and then the captured image was processed by using the MATLAB software (Developed by MathWorks, Natick, 2. Parameter the Experiment. MA, USA) in a computer with theTable proposed system.ofThe relevant parameters are shown in Table 2.

Parameter Table 2. Parameter of the Experiment. The focal length (mm) Aperture Parameter Value Resolution The focal length (mm) 4.25 Exposure time (s) Aperture F1.7 Resolution 4032 × 2448 ISO Exposure time (s) 1/28,310 The diameter of the LED downlight (cm) ISO 100 The power of each LED (W) The diameter of the LED downlight (cm) 6 Current of each LEDLED (mA) The power of each (W) 9 Current of each LED (mA)

Value 4.25 F1.7 4032 × 2448 1/28,310 100 6 9 85

85

4.1. Frequency and the Bright Stripe’s Duty-Ratio Detection Error Analysis 4.1. Frequency the Bright of Stripe’s Duty-Ratio Analysis When theand modulation the frequency of Detection the LEDsError increases, the stripe will become dense at the sameWhen distance, which means the error of duty ratio of bright stripe large. If the dutydense ratio at of the the the modulation of the frequency of the LEDs increases, the is stripe will become strip can be correctly the number of the strip be accurately obtained. accurate same distance, whichextracted, means the error of duty ratio ofcan bright stripe is large. If theHowever, duty ratio of the identification of the number of bright stripes does not guarantee the exact duty cycle. Therefore, in strip can be correctly extracted, the number of the strip can be accurately obtained. However, accurate the case of aof duty-ratio of of 50%, we stripes swept does the frequency fromthe 0.1exact to 10 kHz in 500 Hz steps identification the number bright not guarantee duty cycle. Therefore, in and the evaluated the ability of the duty-ratio regarding bright stripe detection. case of a duty-ratio of 50%, we swept the frequency from 0.1 to 10 kHz in 500 Hz steps and evaluated On the other hand, the performance of different modulation frequencies varied greatly at the ability of the duty-ratio regarding bright stripe detection. different distances. In practical applications, distance should match the appropriate frequency as far On the other hand, the performance of different modulation frequencies varied greatly at different as possible. Therefore, it is necessary to explore the relationship between distance, frequency, and distances. In practical applications, distance should match the appropriate frequency as far as possible. recognition rate. Therefore, it is necessary to explore the relationship between distance, frequency, and recognition rate. Therelationship relationshipbetween betweenthe thetransmission transmissionfrequency frequencyand andthe theabsolute absolutevalue valueof ofthe thebright brightstripe’s stripe’s The duty-ratio recognition is shown in Figure 15. duty-ratio recognition is shown in Figure 15.

Figure 15. 15. The The recognition recognition rate rate at at different differentdistances distanceswith withdifferent differentfrequencies. frequencies. Figure

Appl. Sci. 2018, 8, 2425 Appl. Sci. 2018, 8, x FOR PEER REVIEW

16 of 21 16 of 21

Ascan canbebeseen seen from Figure 15, when the system in practice, the transmitter with a As from Figure 15, when the system is usedisinused practice, the transmitter with a relatively relatively low should frequency should bewhen preferred when the receiver is close to the transmitter, and the low frequency be preferred the receiver is close to the transmitter, and the transmitter transmitter with a relatively high frequency should be preferred when the receiver is far away from with a relatively high frequency should be preferred when the receiver is far away from the transmitter. the transmitter. it can be seen that 8frequency kHz is a high frequency criticalthan point; higher than Moreover, it canMoreover, be seen that 8 kHz is a high critical point; higher 8 kHz, and the8 kHz, and the recognition ratedramatically, has droppedand dramatically, system will fail. recognition rate has dropped the systemand willthe fail. 4.2. 4.2. Frequency Frequency Resolution Resolution and andLED-ID LED-IDRecognition RecognitionAnalysis Analysis At At the the same same distance, distance, with with the the increase increase of of frequency, frequency,the thenumber numberof ofstripes stripesincreases. increases. Increasing Increasing the the number number of of stripes stripes also also causes causes the thestripes stripesto tobecome becomedense. dense. Therefore, Therefore, the the accuracy accuracy of of feature feature extraction extractionwill willbe bereduced, reduced,which whichwill willaffect affectthe theclassifier’s classifier’straining trainingand andrecognition. recognition.This Thissection sectiontests tests the the impact impact of of different different frequency frequency resolution resolution on on the theLED-ID LED-ID recognition recognition rate, rate, and and the theexperimental experimental results resultsare areas asfollows. follows. When When the the minimum minimum frequency frequency resolution resolution was was smaller smaller than than aa certain certain critical critical value, value, both both of of the the recognition and SVM dropped sharply. We We can see Figure 16 that value recognitionrates ratesfor forANN ANN and SVM dropped sharply. canfrom see from Figure 16this thatcritical this critical is 50 Hz. value is 50 Hz.

Figure 16. 16. The The recognition recognition rate rate with with different differentfrequency frequencyresolution. resolution. Figure

4.3. 4.3. Skewing Skewing Resolution Resolution and and LED-ID LED-ID Recognition RecognitionAnalysis Analysis A Bayer filter, where thethe color information recorded by A modern moderncolor colorcamera camerasystem systemisisbased basedonon Bayer filter, where color information recorded the camera is estimated according to certain rulesrules [24],[24], so there is always some errorerror when we take the by the camera is estimated according to certain so there is always some when we take skewing of the When the phase difference of theof input signalsignal at the transmitter is too small, the skewing ofimages. the images. When the phase difference the input at the transmitter is too itsmall, will seriously affect the feature and recognition. This section the impact ofimpact different it will seriously affect the detection feature detection and recognition. Thistests section tests the of phase different the LED-ID For ease ofrate. understanding, of using different phaseresolution differenton resolution on recognition the LED-IDrate. recognition For ease of instead understanding, the skewing coefficient as the transverse we transverse use the phase difference of the transmitter signal as instead of using the skewing coefficientaxis, as the axis, we use the phase difference of the the transverse axis.asThe results given in Figure 17.are given in Figure 17. transmitter signal theexperimental transverse axis. Theare experimental results ItIt can can be be seen seen from from Figure Figure 17 17 that that the the recognition recognition rate rate of of ANN ANN and and SVM SVM decreased decreased greatly greatly when when ◦ .Because 360◦° = 18, there are A3 = 4698 different the phase difference resolution was less than 20 the phase difference resolution was less than 20°.Because 20◦ = there are A18 = 4698 different ° combinations of species caused by phase differences of the transmitter combinations of species caused by phase differences of the transmittersignal. signal. Through Throughexperiments experiments4.1–4.3, 4.1–4.3,and andthe theconclusion conclusionof ofexperiment experiment3.1~3.2 3.1~3.2 in in our our previous previous work workin in Ref. [17], which illustrates that the detection accuracy of the bright stripe’s duty-ratio would not affect Ref. [17], which illustrates that the detection accuracy of the bright stripe’s duty-ratio would not affect the the training trainingof of classifiers, classifiers,we wecan canknow knowthe the proposed proposedscheme schemeof ofLED LEDrecognition. recognition. With Withaamodulation modulation bandwidth of 0.1 to 8 kHz, frequency resolution of 60 Hz, and nine different duty-ratios bandwidth of 0.1 to 8 kHz, frequency resolution of 60 Hz, and nine different duty-ratios in in each each 3 8000−100 frequency, this could offer roughly × 9 × A = 6, 680, 556 unique LED-ID and systems with 18 × 9 × A = 6,680,556 unique LED-ID and systems with frequency, this could offer roughly 50 different transmission frequencies for three channels can offer roughly A3158 = 3,869,736 ( 800050−100 = 158) differentunique transmission frequencies three 1.05 channels offer roughly = 3,869,736 = different LED-IDs. The totalfor is about × 107can . This amount is farA greater than our( previous This amounttechnology is far greater than our 158) different unique LED-IDs. total isrequirement about 1.05 ×of10a7.complement work in Ref. [17]. It can meet theThe quantity of QR code, previous work in Ref. [17]. It can meet the quantity requirement of a complement technology of QR

Appl. Sci. 2018, 8, 2425

17 of 21

Appl.Sci. Sci.2018, 2018,8,8,xxFOR FORPEER PEERREVIEW REVIEW Appl.

17 of of 21 21 17

and with the development of camera technology, the proposed method would provide more channels code, and and with with the the development development of of camera camera technology, technology, the the proposed proposed method method would would provide provide more more code, for LED recognition. channelsfor forLED LEDrecognition. recognition. channels

Figure17. 17.The Therecognition recognitionrate ratewith withdifferent differentskewing skewingresolution. resolution. Figure 17. The recognition rate with different skewing resolution. Figure

4.4. 4.4.Comparison ComparisonExperiment Experimentof RecognitionDistance Distancewith withQR QRCode Code 4.4. Comparison Experiment ofofRecognition Recognition Distance with QR Code Compared long effective distance. Comparedwith withQR QRcodes, codes,one oneof ofthe theprominent prominentadvantages advantagesof ofOBC OBCis thelong longeffective effectivedistance. distance. Compared with QR codes, one of the prominent advantages of OBC isisthe the In this section, by increasing the distance between the receiver and transmitter, we compared In this this section, section, by by increasing increasing the the distance distance between between the the receiver receiver and and transmitter, transmitter, we we compared comparedthe the In the performance OBC andand QR QR codes whose area isarea the same as same the LED’s luminous The recognition performanceofof of OBC and QR codes codes whose area the same as the the LED’sarea. luminous area. The The performance OBC whose isis the as LED’s luminous area. rate data of QR code is obtained fromisisthe scanning function of QR code in many the recognition rate data of QR QR code code obtained from the scanning scanning function ofApplications QR code code in inin many recognition rate data of obtained from the function of QR many smartphone used in the experiment. Under the condition of no magnification, scanning within 5 s is Applications in in the the smartphone smartphone used used in in the the experiment. experiment. Under Under the the condition condition of of no no magnification, magnification, Applications considered a successful recognition. The result recognition. is shown as Figure 18. isisshown scanningwithin within considered successful recognition. Theresult result shownas asFigure Figure18. 18. scanning 55ssisisconsidered aasuccessful The As shown in the Figure 18, when the distance is greater than 3.5 m, the recognition rate of QRof code As shown in the Figure 18, when the distance is greater than 3.5 m, the recognition rate of QR As shown in the Figure 18, when the distance is greater than 3.5 m, the recognition rate QR drops sharply and almost theloses recognition ability, but the OBC high recognition code drops drops sharply sharply and loses almost loses the recognition recognition ability, butstill themaintains OBC still stillamaintains maintains high code and almost the ability, but the OBC aa high rate at 6 m. recognition rate at 6 m. recognition rate at 6 m.

Figure18. 18.Recognition Recognitiondistance distancecompare comparewith withQR QRcode. code. Figure 18. Recognition distance compare with QR code. Figure

4.5.Contrast ContrastExperiment Experimentwith withQR QRCode Codein inDark DarkCondition Condition 4.5. 4.5. Contrast Experiment with QR Code in Dark Condition Anotheradvantage advantageofof of OBC over QR codes is that that can work well in the the dark. dark. We still stillthe useQR the Another OBC over QR codes is is that it can work well in the dark. We We still use Another advantage OBC over QR codes itit can work well in use the QR code in the previous section. We tested and compared the performance of OBC and QR codes code in theinprevious section. We tested and compared the performance of OBCofand QRand codes QR code the previous section. We tested and compared the performance OBC QRunder codes under different different lighting lighting conditions, conditions, we we did did experiments experiments in in the the room room with with the the indoor indoor light light source source under switched off off completely, completely, and and recorded recorded the the recognition recognition rate rate at at different different times, times, which which are are shown shown in in switched Figure19. 19. Figure

Appl. Sci. 2018, 8, 2425

18 of 21

different lighting conditions, we did experiments in the room with the indoor light source switched off completely, and recorded the recognition rate at different times, which are shown in Figure 19.18 of 21 Appl. Sci. 2018, 8, x FOR PEER REVIEW

Appl. Sci. 2018, 8, x FOR PEER REVIEW

18 of 21

Figure 19. 19. Contrast Contrast experiment experiment with with QR QR code in in dark dark condition. QR code dark condition. condition. Figure

It can can can be be seen seen from from Figure Figure 19 19 that that the the dark dark condition condition completely completelybeats beatsthe theQR QRcode. code. It be seen from Figure 19 that the dark condition completely beats the QR code. Through experiments 4.4–4.5, we can conclude that the OBC is superior in distant and and dark dark dark Through experiments 4.4–4.5, we can conclude that the OBC is superior in distant conditions. It is feasible to use it as a complementary technique for QR code. conditions. complementary technique for QR code. conditions. It is feasible to use it as a complementary technique for QR code. Figure Figure 20 20 is is the the receiver receiver platform platform we we use. use. Figure 20 is the receiver platform we use.

Laptop Laptop

RGB-LED RGB-LED Cable Cable

STM32 STM32

Scale Scale

Figure 20. 20. LED LED transmitter transmitter platform. platform. Figure Figure 20. LED transmitter platform.

4.6. Comprehensive Comprehensive Analysis Analysis 4.6. To simulate simulate the the real real environment, environment, aa comprehensive comprehensive experiment experiment was was designed designed to to explore explore the the To environment, combined effects effects of above parameters. We randomly selected 20 LED-IDs that we used in the combined of above parameters. We randomly selected 20 LED-IDs that we used in the previous effects of above parameters. We randomly selected 20 LED-IDs that we used in the previous section to explore theeach effect of each each feature on on recognition recognition rate. The The feature data of these these section tosection exploreto the effect of feature on feature recognition rate. The feature datafeature of these LED-IDs is previous explore the effect of rate. data of LED-IDs is shown intotal Table 3. Agroups total of ofwere 120 groups groups were usedsamples, as training training samples, and the other other 30 shown in is Table 3. Ain of 3. 120 used aswere training andsamples, the otherand 30 groups were LED-IDs shown Table A total 120 used as the 30 groups were used as asfor testeach samples for each each class. class. used as were test samples class.for groups used test samples Table 3. 3. The The parameters parameters of of different different OBCs. OBCs. Table

Class Frequency Frequency (Hz) (Hz) Skewing Skewing Coefficient Coefficient (°) (°) Distance Distance (cm) (cm) Duty-Ratio Duty-Ratio (%) (%) Class 500 0.25 20 25 11 500 0.25 20 25 1000 0.25 40 25 22 1000 0.25 40 25 2000 0.25 60 25 33 2000 0.25 60 25

Appl. Sci. 2018, 8, 2425

19 of 21

Appl. Sci. 2018, 8, x FOR PEER REVIEWTable 3. The parameters of different OBCs.

Class

4 51 2 6 3 74 85 96 10 7 8 11 9 1210 1311 1412 13 1514 1615 1716 1817 18 19 19 2020

Skewing Coefficient (◦ )

Frequency (Hz)

3000 500 4000 1000 5000 2000 6000 3000 7000 4000 5000 8000 6000 1000 7000 1000 8000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000

0.25 0.25 0.35 0.25 0.35 0.25 0.35 0.25 0.35 0.35 0.35 0.35 0.35 0.35 0.35 0.5 0.35 0.5 0.35 0.5 0.5 0.5 0.75 0.5 0.75 0.75 0.75 0.75 0.75 0.75 10.75 1 1 1 11

Distance (cm)

80 20 100 40 120 60 140 80 160 100 120 180 140 200 160 20 180 20 200 20 20 20 20 20 20 20 100 20 100 100 100 100 100 100 100 100 100

19 of 21

Duty-Ratio (%)

25

25 50 25 50 25 50 25 50 50 50 50 50 50 50 50 50 50 50 50 50 50 50 50 75 50 75 75 75 75 75 75 75 75 75 75 75

The average training time was 2.304 s and the total image processing/classifying time was 9.47 The average training time was 2.304 s and the total image processing/classifying time was 9.47 ms. ms. The recognition rate of these LED-IDs is shown in Figure 21. It can be seen from Figure 21 that The recognition rate of these LED-IDs is shown in Figure 21. It can be seen from Figure 21 that the the recognition rate of most LED-IDs was 100%. recognition rate of most LED-IDs was 100%. 100%

Recognition rate

99% 98% 97% 96% 95% 94% 1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20

Class Figure 21. The result of the comprehensive test. Figure 21. The result of the comprehensive test.

This was undertaken in a laboratory scenario. In the future, the system can be applied to more extensive scenarios. Although the system needs pre-training, cancan specify the number of This was undertaken in a laboratory scenario. In the future,the theuser system be applied to more LED-IDs needed and then extracts them from LED-ID library wecan have built. the Thenumber cost is not very extensive scenarios. Although the system needsthe pre-training, the user specify of LEDhighneeded in this and way,then but it can achieve a fairly recognition rate andThe robustness. Atvery the same IDs extracts them from thesatisfactory LED-ID library we have built. cost is not high time, canbut seeit that the totalaprocessing/classifying time was when the system in in thiswe way, can achieve fairly satisfactory recognition ratevery andshort robustness. At the samewas time, use, reaching the millisecond level. Last, there is no need to pay too much attention to training time, we can see that the total processing/classifying time was very short when the system was in use, because users will use trained reaching the millisecond level.classifiers. Last, there is no need to pay too much attention to training time, What is more, ourtrained systemclassifiers. can be implemented in many large-scale scenarios, not just in the lab. because users will use First What of all, is inmore, the field fast payment, scanning a QRincode to pay is widely used now. it lab. also ourofsystem can be implemented many large-scale scenarios, notHowever, just in the has some and distant conditions, will fail. OBC a longer effective distance First of all,limitations. in the fieldIn ofdark fast payment, scanning a QRit code to pay is with widely used now. However, it also has some limitations. In dark and distant conditions, it will fail. OBC with a longer effective distance can reduce the queuing time for payments in some cases. Then, in the field of future autopiloting, we can use our system for traffic control. When the vehicle arrives at the designated node, the onboard camera captures the OBC and downloads the road information on the cloud server

Appl. Sci. 2018, 8, 2425

20 of 21

can reduce the queuing time for payments in some cases. Then, in the field of future autopiloting, we can use our system for traffic control. When the vehicle arrives at the designated node, the onboard camera captures the OBC and downloads the road information on the cloud server to perform the corresponding operation. In the outdoor advertising field, the system can also be used. Because the area of an advertising light board is limited, it cannot accommodate a lot of information, and it is limited to static information. By loading our OBC in the advertising light board, you can link to other web pages for more information. Additionally, in the field of indoor positioning, GPS tends to show weakness. Many scholars have proposed localization algorithms based on visible light communication in recent years [24–26]. However, these high-precision algorithms cannot be correctly labeled LED. It is rare to see the work of LED calibration in this field. Our system can solve this problem well. 5. Conclusions In this paper, an optical bar code (OBC) detection and recognition method based on visible light communication using machine learning is proposed as a complement to QR code technology. To provide a high robustness, widely used LED-ID identification scheme and a huge number of unique LED-ID, we translated the problem into an OBC recognition problem in the machine learning field using RGB-LEDs as the transmitters. A variable frequency, duty-ratio, and phase difference PWM method was employed to create different features for different LEDs. Therefore, in the feature space, the features of different LED-IDs are linearly separable. Once the proposed image processing method realized the feature extraction and selection of LED-IDs, an ANN or SVM could be used to identify LED-IDs. As the experiment result shows, the proposed scheme could offer roughly 1.05 × 107 unique LED-IDs with a greater than 95% LED-ID recognition rate. Moreover, the number can be more by using LEDs of different shapes and combining shape recognition. The effective working distance of the system could reach up to 6 m, which is significantly farther than QR code. Furthermore, with the upgrade of the camera technology, the proposed method would offer more unique LED-ID and the maximum distance between the transmitter and receiver would also increase. Therefore, the OBC can be applied as a complement to QR code technology. Author Contributions: Conceptualization, G.W.; Data curation, L.J.; Investigation, L.J.; Software, L.J.; Writing—original draft, L.J. Funding: This work supported by National Undergraduate Innovative and Entrepreneurial Training Program (No. 201810561195, No. 201810561218, No. 201810561219, No. 201810561217), Special Funds for the Cultivation of Guangdong College Students’ Scientific and Technological Innovation (“Climbing Program” Special Funds) (pdjh2017b0040, pdjha0028), Guangdong science and technology project (2017B010114001). Conflicts of Interest: The authors declare no conflict of interest.

References 1. 2.

3.

4. 5.

Castillo, A.; Thierer, A.D. Projecting the Growth and Economic Impact of the Internet of Things. Soc. Sci. Electron. Publ. 2015, 39, 40–46. [CrossRef] Marktscheffel, T.; Gottschlich, W.; Popp, W.; Werli, P.; Fink, S.D.; Bilzhause, A.; de Meer, H. QR code based mutual authentication protocol for Internet of Things. In Proceedings of the World of Wireless, Mobile and Multimedia Networks, Coimbra, Portugal, 21–24 June 2016; pp. 1–6. Tsiropoulou, E.E.; Gialagkolidis, I.; Vamvakas, P.; Papavassiliou, S. Resource allocation in visible light communication networks: NOMA vs OFDMA transmission techniques. In Proceedings of the International Conference on Ad-Hoc Networks and Wireless, Lille, France, 4–6 July 2016; pp. 32–46. Singhal, C.; De, S. (Eds.) Resource allocation in next-generation broadband wireless access networks; Springer: Cham, Germany, 2017. IGI Global, Bykhovsky, Dima, and Shlomi Arnon. Multiple access resource allocation in visible light communication systems. J. Lightw. Technol. 2014, 32, 1594–1600. [CrossRef]

Appl. Sci. 2018, 8, 2425

6. 7. 8. 9. 10. 11. 12. 13.

14.

15. 16. 17.

18. 19. 20. 21. 22.

23.

24. 25.

26.

21 of 21

Kim, W.-C.; Bae, C.-S.; Jeon, S.-Y.; Pyun, S.-Y.; Cho, D.-H. Efficient resource allocation for rapid link recovery and visibility in visible-light local area networks. IEEE Trans. Consum. Electron. 2010, 56, 524–531. [CrossRef] Jovicic, A.; Li, J.; Richardson, T. Visible light communication: Opportunities, challenges and the path to market. IEEE Commun. Mag. 2013, 51, 26–32. [CrossRef] Karunatilaka, D.; Zafar, F.; Kalavally, V.; Parthiban, R. LED Based Indoor Visible Light Communications: State of the Art. IEEE Commun. Surv. Tutor. 2015, 17, 1649–1678. [CrossRef] Zheng, D.; Chen, G.; Farrell, J.A. Joint measurement and trajectory recovery in visible light communication. IEEE Trans. Control Syst. Technol. 2017, 25, 247–261. [CrossRef] Fang, J.; Yang, Z.; Long, S.; Wu, Z.; Zhao, X.; Liang, F.; Jiang, Z.L.; Chen, Z. High-speed indoor navigation system based on visible light and mobile phone. IEEE Photon. J. 2017, 9, 8200711. [CrossRef] Zhang, B.; Ren, K.; Xing, G.; Fu, X.; Wang, C. SBVLC: Secure barcode-based visible light communication for smartphones. IEEE Trans. Mobile Comput. 2016, 15, 432–446. [CrossRef] Chen, H.-W.; Wen, S.S.; Liu, Y.; Fu, M.; Weng, Z.C.; Zhang, M. Optical camera communication for mobile payments using an LED panel light. Appl. Opt. 2018, 57, 5288–5294. [CrossRef] [PubMed] Cai, Y.; Guan, W.; Wu, Y.; Xie, C.; Chen, Y.; Fang, L. Indoor high precision three-dimensional positioning system based on visible light communication using particle swarm optimization. IEEE Photon. J. 2017, 9, 7908120. [CrossRef] Guan, W.; Wu, Y.; Xie, C.; Chen, H.; Cai, Y.; Chen, Y. High-precision approach to localization scheme of visible light communication based on artificial neural networks and modified genetic algorithms. Opt. Eng. 2017, 56, 106103. [CrossRef] Guan, W.; Wu, Y.; Xie, C.; Fang, L.; Liu, X.; Chen, Y. Performance analysis and enhancement for visible light communication using CMOS sensors. Opt. Commun. 2018, 410, 531–545. [CrossRef] Xie, C.; Guan, W.; Wu, X.; Fang, L.; Cai, Y. The LED-ID Detection and Recognition Method based on Visible Light Positioning using Proximity Method. IEEE Photon. J. 2018, 10, 1–16. [CrossRef] Danakis, C.; Afgani, M.; Povey, G.; Underwood, I.; Haas, H. Using a CMOS camera sensor for visible light communication. In Proceedings of the 2012 IEEE Globecom Workshops, Anaheim, CA, USA, 3–7 December 2012; pp. 1244–1248. Liu, Y. Decoding mobile-phone image sensor rolling shutter effect for visible light communications. Opt. Eng. 2016, 55, 016103. [CrossRef] Chow, C.W.; Shiu, R.J.; Liu, Y.C.; Liu, Y.; Yeh, C.H. Non-flickering 100 m RGB visible light communication transmission based on a CMOS image sensor. Opt. Express 2018, 26, 7079. [CrossRef] [PubMed] Landis, C. Determinants of the critical flicker-fusion threshold. Physiol. Rev. 1954, 34, 259–286. [CrossRef] [PubMed] Chen, P.; Yuan, L.; He, Y.; Luo, S. An improved SVM classifier based on double chains quantum genetic algorithm and its application in analogue circuit diagnosis. Neurocomputing 2016, 211, 202–211. [CrossRef] Liu, Y. The BP neural network classification method under Linex loss function and the application to face recognition. In Proceedings of the International Conference on Computer and Automation Engineering, Singapore, 26–28 February 2010; pp. 592–595. Hu, P.; Pathak, P.H.; Feng, X.; Fu, H.; Mohapatra, P. ColorBars: Increasing data rate of LED-to-camera communication using color shift keying. In Proceedings of the 11th ACM Conference on Emerging Networking Experiments and Technologies, Heidelberg, Germany, 1–4 December 2015; pp. 1–13. Neild, A.; Armstrong, J.; Wang, T.Q.; Armstrong, J. Position Accuracy of Time-of-Arrival Based Ranging Using Visible Light with Application in, Indoor Localization Systems. J. Lightw. Technol. 2013, 31, 3302–3308. Sertthin, C.; Tsuji, E.; Nakagawa, M.; Kuwano, S.; Watanabe, K. A Switching Estimated Receiver Position Scheme for Visible Light Based Indoor Positioning System. In Proceedings of the International Symposium on Wireless Pervasive Computing, Melbourne, VIC, Australia, 11–13 February 2009; pp. 1–5. Fu, M.; Zhu, W.; Le, Z.; Manko, D. Improved visible light communication positioning algorithm based on image sensor tilting at room corners. IET Commun. 2018, 12, 1201–1206. [CrossRef] © 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).