Detecting spatial defects in colored patterns using self ... - AIP Publishing

JOURNAL OF APPLIED PHYSICS 123, 215107 (2018)

Detecting spatial defects in colored patterns using self-oscillating gels Yan Fang,1,a) Victor V. Yashin,2,a) Samuel J. Dickerson,1,a) and Anna C. Balazs2,b) 1

Department of Electrical and Computer Engineering, University of Pittsburgh, 1238 Benedum Hall, Pittsburgh, Pennsylvania 15261, USA 2 Department of Chemical Engineering, University of Pittsburgh, 940 Benedum Hall, Pittsburgh, Pennsylvania 15261, USA

(Received 6 February 2018; accepted 18 April 2018; published online 6 June 2018) With the growing demand for wearable computers, there is a need for material systems that can perform computational tasks without relying on external electrical power. Using theory and simulation, we design a material system that “computes” by integrating the inherent behavior of self-oscillating gels undergoing the Belousov–Zhabotinsky (BZ) reaction and piezoelectric (PZ) plates. These “BZ-PZ” units are connected electrically to form a coupled oscillator network, which displays specific modes of synchronization. We exploit this attribute in employing multiple BZ-PZ networks to perform pattern matching on complex multi-dimensional data, such as colored images. By decomposing a colored image into sets of binary vectors, we use each BZ-PZ network, or “channel,” to store distinct information about the color and the shape of the image and perform the pattern matching operation. Our simulation results indicate that the multi-channel BZ-PZ device can detect subtle differences between the input and stored patterns, such as the color variation of one pixel or a small change in the shape of an object. To demonstrate a practical application, we utilize our system to process a colored Quick Response code and show its potential in cryptography and steganography. Published by AIP Publishing. https://doi.org/10.1063/1.5025052 I. INTRODUCTION

Due to the increasing demand for wearable computers and multi-functional, soft robots, there is a push to design flexible and stretchable materials that can perform functions such as sensing, actuation, and communication.1,2 One possible means of meeting this need is to design systems where the material and the computer are one and the same entity: in essence, to develop “materials that compute.” Ideally, these material systems should operate in a relatively self-sustained autonomous manner, without the need for external electrical power, which would limit the portability of the device. Furthermore, the mode of computing in these systems should harness the intrinsic properties of the materials and thus may be different from traditional computation based on Boolean logic. Notably, there have been a range of studies aimed at devising alternative approaches to performing computation. For example, researchers have harnessed various chemical reactions to create “chemical computers”3–7 and utilized metamaterials to enable optical computing.8 However, in chemical computers, the efficiency of information propagation is limited by the speed of diffusion of the reactive species. Researchers are also developing unconventional computing paradigms that use coupled oscillators to replace Boolean logic gates.9–13 These systems primarily rely on rigid solid-state devices and circuits, instead of the soft materials that are ideal for fabrics and soft robotics. Inspired by the previous efforts, we recently used theoretical and numerical modeling to design a hybrid material that harnesses the inherent properties of the materials to perform an a)

Electronic addresses: [email protected], [email protected], and dickerson@pitt. edu b) Author to whom correspondence should be addressed: [email protected] 0021-8979/2018/123(21)/215107/9/$30.00

important computational task: pattern recognition.14,15 The basic unit in this material system combines a self-oscillating polymer gel undergoing the Belousov–Zhabotinsky (BZ) reaction with an overlaying piezoelectric (PZ) bimorph cantilever (see Fig. 1). The BZ gels are unique because no external stimuli are needed to produce a periodic pulsation of the sample. Rather, the system exhibits autonomous chemo-mechanical transduction, converting the chemical energy from the reaction into the rhythmic motion of the polymer network.16 The PZ cantilever above the oscillating gel also exhibits exceptional material properties. In particular, when the oscillating gel deforms the cantilever, the PZ generates an electrical potential (voltage). Conversely, when a voltage is applied to a PZ layer, it bends. These distinctive material properties give rise to the coordinated behavior that emerges when the “BZ-PZ” units are inter-connected by electrical wires [as in Fig. 1(b)]. Here, the oscillating gel deflects the overlying PZ cantilever, which then generates a voltage that is transmitted through the wire to a connected PZ layer, which in turn bends and deforms the underlying gel. In this way, a given BZ-PZ unit senses the oscillations from the gel and converts this motion into an electrical signal, which then modifies the oscillations of a gel in a connected unit. The interconnected BZ-PZ units in Fig. 1(b) form a coupled oscillator network. The inter-conversion of chemical, mechanical, and electrical energy couples the oscillators, and hence, the oscillations of one gel affect the oscillations of all the others. Eventually, the oscillations of all the units become synchronized, and the system exhibits a stable dynamic state.14 As detailed below, this synchronization behavior enables the system to perform pattern recognition.15 The BZ-PZ network constitutes an ideal component in wearable or portable computing “fabrics.” The whole system

123, 215107-1

Published by AIP Publishing.

215107-2

Fang et al.

FIG. 1. Coupled oscillator network composed of three BZ-PZ units. (a) ðgÞ Swollen BZ gel (right) under applied compressive force Fi in comparison to the un-deformed BZ gel (left), depicted by green cubes. (b) Three BZ-PZ units connected with electrical wires in series. Periodic volumetric changes in the self-oscillating BZ gels cause rhythmic bending of the PZ plates. The orange and blue colors are used to distinguish the two parts of the bimorph PZ plate. The red and black solid lines depict the electric wires connected to the external and internal electrodes, respectively.

J. Appl. Phys. 123, 215107 (2018)

such as the Quick Response (QR) codes.21 The extra layers of information are encoded in the additional BZ-PZ networks (“channels”). In all the examples discussed below, it is the inherent properties of the materials and the synchronization of weakly coupled oscillators that enable the system to perform the computation operation of pattern recognition. When the coupling between two oscillators is weak, the amplitudes of oscillation are relatively constant, but these oscillators display a difference in the phase.22 Below, we first present our theoretical model describing the behavior of the coupled BZ-PZ oscillator network and the phase model used to simulate the network dynamics. We next describe how pattern matching is accomplished in a single BZ-PZ oscillator network and then analyze the method of processing colored patterns through multiple networks. II. BZ-PZ OSCILLATOR NETWORK

operates without an external electrical power supply. Once the reagents are consumed by the BZ reaction, a fresh solution of the reagents can be added to the gel, and hence, after “refueling,” the device can continue to operate in a selfsustained manner.17 Additionally, the gels’ chemo-mechanical oscillations can be modulated by external stimuli and sensory information (e.g., effects of pressure, light, and temperature) can be readily input into these systems.18–20 Moreover, relative to chemical computers, the production of electrical signals from the PZ cantilevers dramatically improves the efficiency of coupling between the basic units because there is no limitation on the spatial location of the units and topology of the network. In previous theoretical and numerical studies,15 we considered the behavior of BZ-PZ units that are connected in series to form a loop and demonstrated how to store a binary (“black and white”) image within this network. We then imposed a collection of input patterns onto different BZ-PZ networks, where each network stored a distinct pattern. We monitored the temporal evolution of the oscillators from a perturbed state to a final steady-state. The network encompassing the input pattern closest to the stored pattern exhibited the fastest convergence time to the stable synchronization behavior and could be identified as the “winner.” In this way, the networks of coupled BZ-PZ oscillators achieved pattern recognition. We demonstrated that the convergence time to the stable synchronization provides a robust measure of the degree of match (DoM) between the input and stored patterns. Herein, we extend the utility of BZ-PZ “materials that compute” by employing the BZ-PZ oscillator networks to store and recognize multi-attribute patterns. These images encode information not only in the shape of an object but also in a supplementary attribute, i.e., the color of each pixel. This is a significant advance because the device can now detect remarkably subtle differences between the input and stored patterns, such as a change in the color of just one pixel or a minute change in the shape of a feature within the image. Thus, the device could be used to discriminate between an original image and counterfeit likeness, which exhibits only small modifications from the original. Below, we show how this device could be used to distinguish subtle differences in two-dimensional, multi-attribute barcodes

Figure 1 shows an example of the BZ-PZ oscillator network that is composed of three BZ-PZ units connected in series by electrical wires. Each unit contains a cubic BZ gel that is 0.5 mm in length and an overlaying PZ bimorph plate, which is 1 mm in length and width and 20 lm in thickness. Driven by the BZ oscillating chemical reaction, each gel periodically swells and shrinks in volume and thus rhythmically deflects the overlaying PZ cantilever. The periodic deflection of the PZ plate generates an oscillating electrical voltage, which is transmitted to other units through the electrical wires. Due to the reverse PZ effect, the applied voltage deflects the PZ cantilever, which applies a mechanic force on the underlying gel and affects its chemo-mechanical oscillations. Hence, in a network of multiple BZ-PZ units, the chemo-mechanical oscillation of one BZ gel affects the oscillations of all other gels. A. Modeling the self-oscillating polymer gel

We assume that the gel is fabricated by crosslinking poly(N-isopropylacrylamide) (PNIPAAm) chains containing a grafted ruthenium (Ru) metal-ion catalyst. The gel is swollen in a solution of the BZ reagents, and the ongoing BZ reaction causes the periodic reduction and oxidation of the anchored Ru metal-ion. The solvent provides a more hydrating environment for the polymer chains when the catalyst is in the oxidized state (Ru3þ ), and hence, oxidation of the catalyst leads to swelling of the gel. On the other hand, reduction of the catalyst to the Ru2þ state makes the solvent less hydrating, and so, the gel contracts. As a result, the cyclic redox reaction drives the periodic changes in volume of the gel. The kinetics of the BZ reaction within the gel can be described by a modification of the Oregonator model for the reaction in solution.23 The original Oregonator model was formulated in terms of the dimensionless concentrations of the key reaction intermediate u (HBrO2 , the activator) and the oxidized metal-ion catalyst v (Ru3þ in the case considered here). The modified model24 accounts for the dependence of the BZ reaction rates on the volume fraction of polymer, /, assuming that the polymer affects the reaction as a chemically neutral diluent.

215107-3

Fang et al.

J. Appl. Phys. 123, 215107 (2018)

We take the gel sample to be sufficiently small that the diffusion of the dissolved activator, u, throughout the gel occurs faster than variations of u and v due to the BZ reaction. In this case, the reaction kinetics of the gel is given by the following equations:25,26 h i ð1  /Þ d=dt u ð1  /Þ1 ¼ FBZ ðu; v; /Þ; (1) / d=dt ðv /1 Þ ¼ eBZ GBZ ðu; v; /Þ; where FBZ and GBZ are the reaction rates that depend on the concentrations, u; v, and, /. In a system of n BZ-PZ units in series, the reaction kinetics within each gel is still given by Eq. (1), but the concentration variables are all labelled with the unit number i ðui ; vi ; /i Þ. Given that the gel is sufficiently small, the sample equilibrates in size faster than the time-scale for one cycle of the BZ oscillation. Consequently, the dimensions of the gel can be determined by balancing all the forces acting on the sample, namely, the elasticity of the polymer network, osmotic pressure of the polymer, and force exerted by the PZ cantilever. This balance yields the following equation to determine the size of the ith gel in the system: h i ðgÞ 1 þ ðh0 k? Þ2 Fi  / ð2 / Þ c0 ki k2 i 0 ? ¼ pFH ð/i Þ þ v vi /i :

(2)

The first term on the left hand side of Eq. (2) describes the elastic stress in the network. This term depends on the cross-link density of the gel, c0 , the degrees of swelling in the longitudinal and transverse directions, ki and k? , respectively [see Fig. 1(a)]. The volume fraction of the polymer is 2 calculated as /i ¼ /0 k1 i k? , where /0 is the polymer volume fraction in the un-deformed gel. We assumed that the gel deformation is uniaxial, and thus, k? is constant. The second term on the left hand side of Eq. (2) describes the compressive force exerted by the cantilever, ðgÞ Fi ,14,27 where h0 is the size of the un-deformed gel cube. Finally, the right-hand side of Eq. (2) is the osmotic pressure, which counteracts the elasticity of the polymer network and the compressive force. The osmotic pressure includes both the Flory-Huggins contribution, pFH , and the term proportion to v , which describes the hydrating effect of the oxidized catalyst. The Flory-Huggins contribution is equal to   pFH ¼  / þ logð1  /Þ þ vð/Þ /2 : (3) In Eq. (3), the parameter vð/Þ ¼ 0:338 þ 0:518 / describes interactions between PNIPAAm and water at 20  C.28 The strength of the hydrating effect of the oxidized catalyst is specified by the interaction parameter v .27 The specific value chosen for v and the other parameters characterizing the gel are given in Table I. We further note that in the simulations, the period of oscillation for a single BZ-PZ oscillator unit is approximately T0  1 min.14,15 (Although piezoelectric materials can operate in a frequency range of kHz to MHz, the frequency of the BZ-PZ oscillator is governed by the diffusion-reaction within the BZ gel; this gives rise to the lower operating frequency.)

TABLE I. Model parameters of the BZ gel. Parameter /0 c0 v h0 k? ¼ k

Value

Definition

0.16 4  104 0.105

Volume fraction of the polymer Crosslink density Interaction parameter for the hydrating effect of the oxidized catalyst Length of the un-deformed gel Gel’s degrees of swelling in the longitudinal and horizontal directions

0.5 mm 1.65

B. PZ cantilevers

The cantilevers in Fig. 1(b) are taken to be sufficiently thin (20 lm in thickness) that they can be bent by the relatively soft oscillating gels. Each cantilever consists of two identical layers of a polarized PZ material, with the internal and external surface electrodes connected in parallel. The PZ cantilever is assumed to be composed of polarized Lead-Zirconate-Titanate ceramic (PZT), one of the most commonly used piezoelectrics.29 We further assume that the piezoelectric was fabricated through advanced processing methods30,31 and thus exhibits a twofold increase in the piezoelectric constant relative to the typical PZT. All the PZ cantilevers are assumed to be attached to the underlying gels throughout the cycle of gel swelling and deswelling. As shown in Fig. 1, for an individual BZ-PZ unit, the chemo-mechanical oscillations in a BZ gel and the ðgÞ deflection of the PZ cantilever are related as Fi ¼ Fi and   ni ¼ ðki  k Þ h0 , where k h0 is the spatial offset between the gel and the cantilever and ni is always positive. C. Dynamic behavior of coupled BZ-PZ units

The frequency of the gel’s chemo-mechanical oscillations is on the order of 0.01 Hz, which is much lower than the eigenfrequency of a PZ cantilever (10 kHz). Hence, the behavior of the PZ cantilevers can be viewed as quasi-static. Therefore, the deflection of the PZ plate in unit i, ni , and electric charge on this plate, Qi , can be related to the force, Fi , applied to the cantilever and the electric potential difference (voltage), Ui , between the electrodes through the following equations: n ¼ m11 F þ e m12 U; Q ¼ e m12 F þ m22 U:

(4)

Here, m11 , m12 , and m22 are the coefficients determined by the properties of the piezoelectric material and the cantilever dimensions.14 The parameter ei is the force polarity; it specifies the polarity of the generated voltage, Ui , and depends on the relative direction between the polarization of the PZ materials and the bending force applied on the tip of the cantilever. If the polarization and force lie in the same direction, ei ¼ 1; otherwise, ei ¼ 1. For example, the first BZ-PZ unit with label ei in Fig. 1 has a positive force polarity (þ1) as we define the electrode with the red wire as the positive electrode of Ui .

215107-4

Fang et al.

J. Appl. Phys. 123, 215107 (2018)

Herein, we only discuss the cases where the BZ-PZ units are connected in series (as illustrated in Fig. 1) because this type of connection exhibits the synchronization modes that can be utilized to process information.14,15 For a serial circuit, Pn the sum of voltages across all n units is zero, i¼1 Ui ¼ 0, and the electric charges of all n units are equal, Qi ¼ Q. Using these relationships, together with Eq. (4), the bending force acting on cantilever i, i ¼ 1; 2; …; n, can be calculated as15 " # n X ð0Þ ð0Þ ð0Þ 1 (5) Fi ¼ Fi þ j Fi  ei n ej F j : j¼1 ð0Þ

Here, Fi ¼ m1 11 ni is Hooke’s law for a bending elastic plate, and j ¼ m212 ðm11 m22  m212 Þ1 is the coupling strength coefficient, which is small and depends only on the material properties of the cantilevers.14 Equation (5) indicates that the bending force acting on a given PZ cantilever is affected by the deflections of all the other cantilevers in the network. The cross-terms on the right-hand-side of Eq. (5) describe the effect of the pairwise interactions, with the sign dependent on the force polarities. These interactions are weak since j is small. Specifically, the strength of coupling is j  0:206 with the material parameters used in the study.14 For this coupling strength, the characteristic time (“cycle”) for the network is j1 T0  5 min, where T0  1 min is the period of oscillation for a single BZ-PZ unit (as we mentioned in Sec. II A). D. Phase model for BZ-PZ oscillators

Because the coupling between the oscillating BZ-PZ units is weak, we can use the phase dynamics model22 to describe the interactions between these oscillators in terms of the time-dependent deviation of the phase of each oscillator. The phase dynamics of the BZ-PZ oscillators connected in series (Fig. 1) is described by the following system of equations:14 j1 dui =dt ¼ Hð0Þ  n1

n X

ei ej Hðuj  ui Þ:

(6)

j¼1

Here, the function Hðuj  ui Þ is referred to as the “connection function,”22 which characterizes the rate of the phase shift of oscillator i due to the interaction with the oscillator j at a phase difference uj  ui . The phases of oscillation in Eq. (6) are normalized such that 0  ui  1, i ¼ 1; 2; …; n. The connection function HðhÞ is periodic at h 2 ½0 ; 1 . Figure 2 shows the numerically determined connection function HðhÞ.14 The plot reveals that the phase response of a BZ-PZ oscillator to an external action is quite complicated. Namely, the interaction between the oscillators can cause both positive and negative phase shifts depending on the phase difference. Due to the latter behavior, the BZ-PZ oscillator networks exhibits a variety of stable phase synchronization modes.15 In many oscillator-based computing paradigms, information is represented in terms of differences in the phase of oscillation within the oscillator network, and achieving a

FIG. 2. The connection function HðhÞ in the phase model of BZ-PZ oscillator networks. The function is periodic at h 2 ½0; 1 .

desired state of synchronization signifies the completion of some operation of computation.9,10 As we demonstrated previously, the synchronization behavior of the BZ-PZ networks could be utilized to perform such a computation operation as the recognition of binary, black, and white images.15 III. PATTERN MATCHING WITH THE BZ-PZ OSCILLATOR NETWORK

Using our computational models, we demonstrate how the BZ-PZ oscillator networks can perform “multidimensional” or multi-channel pattern matching and thus detect differences between a colored input pattern and a colored stored pattern. To illustrate this behavior, we first describe how a black and white image is stored in a single BZ-PZ oscillator network and discuss how the characteristics of the network enable the system to perform matching between input and stored images. Taking advantage of this behavior, we then model multiple BZ-PZ networks, where each network constitutes a “channel.” We use the different channels to process different aspects of an image. With the output from these multiple channels, minute variations between the input and original stored patterns can be detected. A. Single BZ-PZ network

The pattern matching functionality of the BZ-PZ network arises from the dependence of the mode of synchronization among the coupled oscillators on the polarities, ei , of the BZ-PZ units.15,27 In particular, we observed that in a serially connected network, the BZ-PZ units with the same force polarity synchronize in-phase, i.e., with no phase difference, u ¼ 0. BZ-PZ units that differ in their values of e exhibit the anti-phase synchronization with the phase difference of u  0:5. (Note that all phases here are normalized to vary between 0 and 1.) Hence, the phases of BZ-PZ units in a synchronized network are observed to always cluster into two distinct groups since the force polarity assumes one of the two values, þ1 or –1.

215107-5

Fang et al.

FIG. 3. Storing a binary image in one BZ-PZ oscillator network. A black and white square image of a “smiley face” that contains 100 pixels is stored in a network with 100 BZ-PZ units. The force polarity of each unit is set to þ1 for a white pixel and to 1 for a black pixel. The coloring in the BZPZ units is the same as in Fig. 1.

Taking advantage of this distinctive feature of serially connected BZ-PZ oscillators, a binary vector pattern can be stored by setting the polarity values, e’s, of BZ-PZ units. Figure 3 illustrates an example of storing a 100-pixel ð10 10Þ binary image pattern in a BZ-PZ oscillator network with 100 serially connected units. The assigned force polarities are based on the arrangement of the pixels in the image. As shown in Fig. 3, the white pixels are represented by e ¼ 1, while the black pixels are represented by e ¼ 1. A simple way to alter the force polarity in a unit is to swap the electrodes connecting the unit to the network. (To store a different pattern, the force polarities of the units must be set, according to the prescription in Fig. 3, to reflect the arrangement of pixels for that new image.) There are two stages in the pattern matching process once the image has been stored: initialization and synchronization. During the initialization stage, an input pattern is used to initialize the phase of each gel oscillator in the network. For example, in the cases given in Fig. 4, initial phases are set with u ¼ 0 for a white pixel and u ¼ 0:5 for a black pixel. In this way, we represent the information of the input pattern with the same rule we utilized for the pattern storage.

J. Appl. Phys. 123, 215107 (2018)

Because the dynamics of the BZ gels are chemo-, photo-, and mechano-responsive, the initial variations in phase among the units can be introduced by local applications of chemical stimulation, light, or pressure. These characteristics of BZ gels makes the BZ-PZ network a particularly responsive device, being sensitive to initial input from various stimuli. After the initialization, the system undergoes the synchronization stage. The phases of the coupled BZ-PZ units evolve into a stable synchronization mode.15 In particular, the input pattern converges to the stored pattern (which was set by the force polarities). In this context, “convergence” means that the oscillators representing the white pixels establish the in-phase synchronization among themselves and the anti-phase synchronization with the oscillators that represent the black pixels. The speed at which the input pattern converges to the stored pattern depends on the difference between the input and stored patterns. For images of the same size, this difference in patterns can be quantified by calculating the sum of the element-wise differences between two binary vectors (representing the input and stored patterns). This measure is referred to as the Hamming distance. The more similar the two images are (i.e., the smaller the Hamming distance), the shorter the convergence time. Hence, the convergence time can serve as a degree of match (DoM) that quantifies the differences between the input pattern and each stored pattern. Thus, a system consisting of multiple BZ-PZ oscillator networks can recognize patterns by detecting the shortest convergence time among the networks.15 We illustrate the above concept in Fig. 4, which shows examples of pattern matching performed by the BZ-PZ oscillator network that stores the “smiley face” in Fig. 3. The three different images, (a)–(c), are imposed on the network holding this stored pattern, i.e., the test input patterns are used to set the initial phases of the pixels, as discussed above. Note that Fig. 4(a) is identical to the stored pattern, while (b) and (c) are distorted relative to that image. The plots in the right of Fig. 4 show the phase differences between the first oscillator and all the other oscillators (in the given network) as a function of time. The blue line

FIG. 4. Pattern matching operation by the BZ-PZ oscillator network. We utilize the network and stored pattern shown in Fig. 3. Three test input patterns are used to initialize the phases of BZ-PZ units for pattern matching. (a) Original stored pattern. (b) and (c) Distorted patterns. In the three independent matching operations, the phase differences of the oscillations in the networks converge to the stored pattern in the course of synchronization. Their evolution processes are plotted on the right, and the corresponding convergence times are displayed. The phase difference shown on the y-axis is wrapped to ½0; 1 . The blue and red lines represent two groups of oscillators that converge to the phase difference around 0 and 0.5.

215107-6

Fang et al.

J. Appl. Phys. 123, 215107 (2018) TABLE II. Representation of the color. Channels Color

FIG. 5. Schematics of the four BZ-PZ networks used for recognition of colored patterns. After decomposition of the input image, the networks perform simultaneous matching operations in the corresponding information channels. The convergence time (measured in the number of cycles) is determined for each channel separately.

indicates a phase difference of u ¼ 0, while the red line indicates a phase difference of u ¼ 0:5. Plot (a) on the right side of Fig. 4 corresponds to the temporal evolution of the oscillators to the stable synchronization mode when image (a) on the left is imposed as the input onto the stored pattern. Similarly, plots (b) and (c) on the right show the convergence dynamics after the respective inputs (b) and (c) on the left are imposed on the network. The convergence times are measured in the number of cycles and displayed in the graphs. The plots in Fig. 4 reveal the distinct temporal evolution processes that occur when each of the different inputs is applied to the BZ-PZ network. While all the initial oscillations evolve into the final stable state, the time for convergence depends on the spatial features of the input patterns. When the input is identical to the stored pattern, the system converges essentially instantaneously. As the input image becomes more distorted (i.e., as the Hamming distance becomes larger), the convergence time increases. Thus, variations between the input and stored patterns can be detected by observing the time required for the synchronization of the oscillators in the network. B. Multi-channel BZ-PZ networks

A single BZ-PZ network can only perform pattern matching on black and white images (binary patterns). This limitation arises from storing patterns by setting binary values of the force polarities, 61, in the device. To store and process richer images, such as colored patterns, we employ several BZ-PZ networks simultaneously as depicted in Fig. 5, so that one pattern is processed using different “channels.” The patterns that we consider here contain up to eight colors, which are represented in terms of the red-green-blue (RGB) color scheme.32 As shown in Table II, these eight colors are encoded into four binary value channels; the red, green, and blue are the three fundamental color channels. The colors yellow, cyan, and magenta are generated as combinations of the basic colors in RGB. Note that black and white could also be represented by RGB channels as ð0; 0; 0Þ and ð1; 1; 1Þ, respectively. We, however, set these two colors as an independent channel to describe the shape of the pattern.

Name

Black/white

Red

Green

Blue

Black

0

0

0

0

White Red

1 0

0 1

0 0

0 0

Green

0

0

1

0

Blue

0

0

0

1

Yellow

0

1

1

0

Cyan

0

0

1

1

Magenta

0

1

0

1

In the original RGB model, any color can be obtained from the addition of different degrees of the RGB components. Because our color channel is binary and we assume that “0” and “1” represent the respective “none” and “full degree” of color, we are limited to eight colors in this case. Other colors can, however, be obtained with additional channels. For example, if the yellow, cyan, and magenta are defined as separate channels, more colors can be represented by through the combination of six channels, instead of three channels. To encode a specific color image (see Fig. 5), we decompose the pattern into four binary vectors; each binary vector is stored in a single channel, and there are four channels in total. In the black and white (B/W) channel, the pixel value is 1 only if the pixel is white. Namely, this channel marks the white background. For the remaining RGB channels, a value of “1” indicates that the corresponding color component is present in the pixel. For example, yellow is the combination of red and green; hence, it is coded as ð0; 1; 1; 0Þ. Each of the four channels in Fig. 5 is a network of serially coupled BZ-PZ oscillators. The first BZ-PZ network represents the B/W channel, and the second, third, and fourth networks represent the red, green, and blue channels, respectively. Figure 5 illustrates the decomposition of a stored pattern into these four networks. In Fig. 5, the stored pattern is a colored version of the “smiley face,” which now has two green “eyes” and a yellow “mouth.” The vector patterns in each of the different channels are displayed as binary, black, and white images. Figure 6 shows the comparable decomposition of two sample input patterns. (For convenience, we repeat the decomposition of the stored pattern as the first row of Fig. 6.) These different channels capture different visual features of the original color pattern. For example, the network associated with the B/W channel discriminates the color pixels from the background. The pixels representing the green “eyes” are captured in the decomposed pattern in the green channel, while the pixels representing the yellow “mouth” appear in both the red and green channels. The two examples of test input patterns shown in the second and third rows of Fig. 6 are deliberately designed to demonstrate the two types of variations on the stored pattern. The first test pattern shows a variation in color; one of the green “eyes” in the stored pattern is switched to red. The

215107-7

Fang et al.

FIG. 6. Pattern matching in color channels with multiple BZ-PZ networks. The first row shows the colored “smiley face” as the stored pattern and its decomposed binary pattern images in the B/W, red, green, and blue channels. The second and third rows present the two test input patterns with different types of image variations and their decomposed patterns in each channel. On the bottom of each row, the convergence time for each channel is shown.

location and the total number of colored pixels remain unchanged, and thus, the shape of the “smiley face” is the same as the stored pattern. These features are effectively reflected in the decomposed patterns (Fig. 6, second row). The decomposed pattern in the B/W channel still remains the same, but the “left eye” is moved from the green channel to the red channel, representing the color change. The second test pattern displays the same color scheme as the stored pattern, but the size of the left eye is enlarged. Namely, the shape of objects in the image is distorted. This change in the image is captured in the decomposed patterns of both the B/ W channel and the green channel. The pattern matching is performed simultaneously by the four BZ-PZ oscillator networks (i.e., the different channels). The networks contain the stored pattern according to the scheme in the top row of Fig. 6. As in the previous example, the input patterns are used to initialize the phase of the oscillators in the network, following the scheme in row 2 or 3 of Fig. 6. Once the BZ networks are initialized in this way, the decomposed binary images of the test pattern synchronize to the respective decomposed stored image. As expected, the convergence time is sensitive to any defect or the variation between the input and stored images in the corresponding channel. When the first input pattern is imposed on the network, the convergence time in both the B/ W and blue channels are 0 (see Fig. 6). The fact that the convergence time is 0 in the B/W channel indicates that the geometric shape of the input and stored patterns is the same. The convergence time of the green and the red channels, however, is non-zero, thus indicating a variation in the color between the input and stored patterns. In the case involving the second test pattern, the nonzero convergence time in the B/W channel indicates that shape of the input and stored patterns is no longer identical. Of the color channels, it is only the green one that exhibits a non-zero convergence time. Taken together, the information

J. Appl. Phys. 123, 215107 (2018)

reveals that the variation in the shape occurs in the green pixels. Hence, the measurements not only indicate a disparity in the shape of the test and stored patterns but also pinpoint the region where the disparity is localized. To gain further insights into the effect of defects on the convergence time in multiple channels, we perform another two tests on the same stored pattern, as illustrated in Fig. 7. In these two tests, a series of input patterns are generated by replacing the green pixels of the “right eye” one by one with blue pixels [Fig. 7(a)] or with white pixels [Fig. 7(b)], until all the pixels of the right eye are turned into blue [in (a)] or made to disappear [in (b)]. In the former case, we illustrate the effect of the modifying color, while in the latter case, we illustrate the effect of modifying the shape of the image. In Figs. 7(c)–7(e), we plot the convergence times in networks corresponding to the respective blue, green, and B/W channels. Consider the situation for case (a) where the green pixels are turned to blue. Here, the number of blue pixels

FIG. 7. Pattern matching with varying input patterns. (a) Nine color image patterns show the process of turning the green “left eye” into blue pixel by pixel. The first one is the stored pattern, and the rest eight patterns are the input patterns. (b) Similar test with the process that removes the “left eye” object. We plot convergence time in each channel as a function of the number of changed pixels. (c) Convergence time in the blue channel in test (a). (d) Convergence time in the B/W channel in test (b). (e) Convergence times in the green channel for both tests (a) and (b). Note that the green channel responds in exactly the same manner to tests (a) and (b), resulting in the same plot of the convergence time (e).

215107-8

Fang et al.

increases, while the number of green pixels decreases relative to the stored pattern. Notably, the convergence times increase in both the blue and green channels. This increase in convergence time is correlated with the increase in the number of “defective” pixels, which depends on the difference between the decomposed patterns of the input pattern and the stored pattern. Namely, the Hamming distance between input and stored patterns increases as the number of green bits is flipped to another color, and hence, the image becomes more “defective.” This result confirms our prior conclusion15 that the convergence time of a BZ-PZ oscillator network can serve as a distance metric between input and stored binary patterns. The above conclusion also applies to case (b), where the convergence times in the blue and the B/W channels are seen to increase. Again, the increase in convergence time is due to the accumulation of defects as the green pixels are replaced by white sites and the Hamming distance between the decomposed input and stored patterns increases. We now apply our system to an important practical application: the pattern matching of a Quick Response (QR) code. QR codes are the two-dimensional barcodes used to label a tremendous range of consumer goods. Read by a barcode scanner or cell phone, the label allows merchandise to be readily identified, tracked, and marketed.21 QR codes are also used to store sensitive information such as bank account numbers or credit card information and can be used to make automatic payments. Hence, being able to distinguish a legitimate QR code from a counterfeit image is of significant commercial importance. A typical QR code pattern consists of a matrix of black and white pixels. Square patches, or “anchors,” fixed at three corners of the image are used to locate and center the position of the pattern. The pixels in the remainder of the image are arranged in a specific order to encode the desired information. We first generate a 21  21 sized QR pattern that encodes the text phrase “Hello World.” (The message is encoded with the standard QR code algorithm.) We then convert this black and white image into a colored pattern by randomly converting all the black pixels into the colors in Table II to form a new pattern. Such a colored QR pattern can also be correctly decoded using a QR code scanner (if the grayscale threshold is set properly). Here, the encoded color pattern can be used to inscribe additional information.33 Similar to the previous tests, we store the color QR pattern in four BZ-PZ oscillator networks, as illustrated in Fig. 8. The network for each channel has the same number of BZ-PZ units as the number of pixels (21  21 ¼ 441). In the next step, we introduce defect pixels in the input patterns by randomly altering the colors in a number of pixels in the stored pattern. In this test, we add 1, 10, 20, 30, 40, and 50 defect pixels in sequence, and for each case, we repeat the simulation of pattern matching 100 times, with different randomized defects in the input patterns. Figure 8 shows the stored QR pattern and an example of an input pattern with 10 defect pixels. The results from the simulations are plotted as bar graphs that contain averaged, minimum, and maximum convergence times. The error bars for the minimum and maximum convergence times display a

J. Appl. Phys. 123, 215107 (2018)

FIG. 8. Pattern matching of the QR code in the multiple color channel. The black/white binary QR code pattern of the “Hello World” and randomly colored version as the stored pattern are shown in the black frame. An example of test input patterns with 10 defect pixels (marked with “x”) is placed on the top right. The bar graph on the bottom gives the averaged convergence time of the repeated simulation of matching operations. The error bars give the maximum and minimum. Each input pattern is labeled at the x-axis according to its number of defect pixels, with the three color bars corresponding to convergence time in RGB channels. Note that the B/W channel detects no channel in this test and its convergence time is 0, which is not shown.

relatively broad range for the cases with one “defective” pixel; this is due to the large variations associated with the random selection and color change of just one pixel. Changing the colors of pixels into yellow, cyan, or magenta involves two channels and thus brings larger variations in the convergence time in cases that involve just a few defect pixels. These variations in convergence time are reduced as the number of defect pixels is increased. Importantly, the simulation results shown in Fig. 8 indicate that the matching operation is very sensitive to the presence of defects, with even one defect producing an obvious delay in synchronization (above 30 oscillation cycles) as compared to perfect matching between the input and stored patterns. With respect to using the device for encryption or devising a security system, the test QR code information “Hello World” can be viewed as the “plain code,” while the codes in the RGB channels serve as a “password” (or a “watermark”) in different “layers.” In other words, the password is encrypted in multiple channels. In this case, any test pattern where a colored pixel has been flipped from the original colorized pattern (on the left in Fig. 8) would fail the verification test even though the test image carries the correct plain code for a QR scanner. Hence, this QR code test exhibits the potential of our system to be applied in cryptography or steganography. In the latter case, our system can hide an

215107-9

Fang et al.

encrypted message, where even if the encrypted message is deciphered (in this case, the black and white QR code), the hidden message (encoded in the colored pattern) is not seen. IV. CONCLUSIONS

Our overarching goal is to design a material system where the inherent properties of the different components enable the system to perform a computational task: pattern recognition. This task is enabled by combining the unique behavior of self-oscillating BZ gels and piezoelectric (PZ) films in a system that undergoes continuous oscillations without external electrical input. We also built on advances in computer science to utilize the synchronization of oscillators to perform computation. In essence, the properties of the material and the computing paradigm are ideally matched. A particularly challenging task for these “materials that compute” is to detect defects in an input pattern relative to a stored image that encompasses information in both the shape and the color of the pattern. In effect, the device must encode and recognize a significant amount of specific structural details. Here, we addressed this problem by employing a single BZ-PZ network as a distinct “channel” and utilizing multiple channels at once. By decomposing a colored image into sets of binary vectors, we used each channel to store distinct information about the color of the pattern and the shape of the image. By using multiple networks, we captured the spatial features of the entire image in this multi-channel system. Our simulations reveal that the proposed multi-channel BZ-PZ device can detect remarkably subtle differences in spatial features between the input and stored patterns. In particular, the device can detect a change in the color of just one pixel or a small change in the shape of an object in the image. We also applied our system to the task of recognizing a colored QR code and thereby showed its potential in cryptography or steganography. The multiple BZ-PZ oscillator networks recognize patterns by detecting the shortest convergence time among the networks.15 The convergence time is a measure of how long it takes for a set of oscillators to become phase-locked (i.e., synchronized to the final stable states). In the simulations, the latter condition is met if the phase differences among the units lie below a specified threshold value. In a physical realization of this system, peripheral circuitry would be necessary to detect the state of synchronization and measure the convergence time. Such synchronization detection circuits have been designed and implemented in previous studies of neuromorphic and oscillator-based computing.34–36 It is worth emphasizing that the BZ gels are sensitive to light, pressure, and chemical stimuli, and hence, the input pattern can be imposed onto the network through a number of physical or sensory means. In other words, the devices are responsive to external cues and thus can be used to detect environmental changes. By using multi-channel BZ-PZ devices, we can significantly expand the functionality of these “materials that compute” since we can encode and recognize patterns with a richer information content. Hence, the devices are practically suitable for pattern matching or recognition of

J. Appl. Phys. 123, 215107 (2018)

images that encrypt information at different “layers” or spatial levels. Thus, the device can prove to be particularly valuable for security applications that require detection of counterfeit information, encoding of hidden information, and verification of multi-attribute passwords. ACKNOWLEDGMENTS

We gratefully acknowledge the financial support from NSF [DMR-1344178]. 1

M. Amjadi, K. U. Kyung, I. Park, and M. Sitti, Adv. Funct. Mater. 26, 1678 (2016). 2 M. A. McEvoy and N. Correll, Science 347, 1261689 1 (2015). 3 N. G. Rambidi and D. Yakovenchuk, Phys. Rev. E 63, 026607 (2001). 4 J. Gorecki and J. N. Gorecka, Int. J. Unconv. Comput. 2, 321 (2006). 5 P. L. Gentili, V. Horvath, V. K. Vanag, and I. R. Epstein, Int. J. Unconv. Comput. 8, 177 (2012). 6 J. Gorecki et al., Philos. Trans. R. Soc., A 373, 20140219 (2015). 7 A. Adamatzky, PLoS One 11, 0168267 (2016). 8 A. Silva, F. Monticone, G. Castaldi, V. Galdi, A. Al u, and N. Engheta, Science 343, 160 (2014). 9 F. C. Hoppensteadt and E. M. Izhikevich, IEEE Trans. Circuits Syst., I 48, 133 (2001). 10 Y. Fang, C. N. Gnegy, T. Shibata, D. Dash, D. M. Chiarulli, and S. P. Levitan, IEEE J. Explor. Solid-State Comput. Devices Circuits 1, 94 (2015). 11 Y. Fang, V. V. Yashin, A. J. Seel, B. Jennings, R. Barnett, D. M. Chiarulli, and S. P. Levitan, in Proceedings of the 33th IEEE/ACM International Conference on Computer-Aided Design (2014), p. 86. 12 P. Chen et al., in Proceedings of the 35th IEEE/ACM International Conference on Computer-Aided Design (2016), p. 15. 13 D. Vodenicarevic et al., Sci. Rep. 7, 44772 (2017). 14 V. V. Yashin, S. P. Levitan, and A. C. Balazs, Sci. Rep. 5, 11577 (2015). 15 Y. Fang, V. V. Yashin, and A. C. Balazs, “Pattern recognition with materials that compute,” Sci. Adv. 2, e1601114 (2016). 16 R. Yoshida, T. Takahashi, T. Yamaguchi, and H. Ichijo, J. Am. Chem. Soc. 118, 5134 (1996). 17 R. Yoshida, Adv. Mater. 22, 3463 (2010). 18 P. Dayal, O. Kuksenok, and A. C. Balazs, Langmuir 25, 4298 (2009). 19 V. V. Yashin, O. Kuksenok, P. Dayal, and A. C. Balazs, Rep. Prog. Phys. 75, 066601 (2012). 20 V. V. Yashin, K. J. Van Vliet, and A. C. Balazs, Phys. Rev. E 79, 046214 (2009). 21 D. Wave, http://www.qrcode.com for QR code features (2010). 22 E. M. Izhikevich, Dynamical Systems in Neuroscience: The Geometry of Excitability and Bursting (MIT Press, Cambridge, 2007), Chap. 10. 23 J. J. Tyson and P. C. Fife, J. Chem. Phys. 73, 2224 (1980). 24 V. V. Yashin and A. C. Balazs, Macromolecules 39, 2024 (2006). 25 V. V. Yashin, S. Suzuki, R. Yoshida, and A. C. Balazs, J. Mater. Chem. 22, 13625 (2012). 26 V. V. Yashin, O. Kuksenok, and A. C. Balazs, J. Phys. Chem. B 114, 6316 (2010). 27 V. V. Yashin, S. P. Levitan, and A. C. Balazs, Chaos 25, 064302 (2015). 28 S. Hirotsu, J. Chem. Phys. 94, 3949 (1991). 29 A. Preumont, Mechatronics: Dynamics of Electromechanical and Piezoelectric Systems (Springer, Dortrecht, 2006). 30 S. H. Baek et al., Science 334, 958 (2011). 31 S. H. Baek, M. S. Rzchowski, and V. A. Aksyuk, MRS Bull. 37, 1022 (2012). 32 C. Poynton, Digital Video and HD: Algorithms and Interfaces (Elsevier, Amsterdam, 2012). 33 A. Grillo, A. Lentini, M. Querini, and G. F. Italiano, in Proceeding of 2010 International Multiconference on Computer Science and Information Technology (2010), p. 709. 34 D. Vodenicarevic, N. Locatelli, J. Grollier, and D. Querlioz, in Proceeding of 30th International Joint Conference on Neural Networks, 2015 (2016). 35 C. K. Volos et al., J. Eng. Sci. Technol. Rev. 8, 157 (2015). 36 T. Shibata et al., in Proceeding of 13th International Workshop on Cellular Nanoscale Networks and Their Applications (2012).