A bio-inspired multisensory stochastic integration ...

Neurocomputing 151 (2015) 11–33

Contents lists available at ScienceDirect

Neurocomputing journal homepage: www.elsevier.com/locate/neucom

A bio-inspired multisensory stochastic integration algorithm Alex Porras n, Rodolfo R. Llinas Department of Neuroscience and Physiology, NYU School of Medicine, New York, NY 10016, USA

art ic l e i nf o

a b s t r a c t

Article history: Received 24 April 2014 Accepted 23 June 2014 Available online 6 November 2014

The present paper describes a new stochastic multisensory integration system capable of combining a number of co-registered inputs, integrating different aspects of the external world, into a common premotor coordinate metric. In the present solution, the model uses a Stochastic Gradient Descent (SGD) algorithm to blend different sensory inputs into a single premotor intensionality vector. This is done isochronally, as the convergence time is independent of the number and type of parallel sensory inputs. This intensionality vector, generated based on “the sum over histories” [1], makes this implementation ideal to govern noncontinuous control systems. The rapid convergence of the SGD [2–7] is also used to compare with its biological equivalent in vertebrates -the superior tectum- to evaluate limits of convergence, precision and variability. The overall findings indicate that mathematical modeling is effective in addressing multisensory transformations resembling biological systems. & 2014 Elsevier B.V. All rights reserved.

Keywords: Bioinspired Multisensory Stochastic SPGD Intensionality vector Motor output

1. Introduction Multisensory integration has only recently started to move beyond the exclusivity of the cortical mantle and it now accepts that other structures play a more formative role than the telencephalic sensory areas [8–20]. Thus, the V1 cortical area in visuali impared patients, may be activated by auditory, Braille reading or other somatosensory clues [14–21]. In addition, intense activity with superadditive properties [10] can occur in the superior collicullus due to the multisensory activation [18,22]. These integration sites must be considered in addition to the association cortices widely known for multisensory convergence [23]. It would be unwise, therefore, given the complexity of the CNS to address these questions all at once. However, in vertebrate evolution the optic tectum (superior colliculus) is a multisensory combination site [24–34] receiving audition, vision, somatosensory and vestibular inputs, where the inputs from each sensor modality are topographically organized. However, as these maps do not encode shape or size, but only spatial continuity and connectedness they are, from a mathematical perspective, topological rather than topographically organized [24,30]. The optic tectum, basically addresses multisensory representation in the context of posture and motricity, but most importantly, the multisensory stimuli is integrated to organize responses that exceed the sum of the individual stimuli [10,14] – this is called superadditivity.

n

Corresponding author. E-mail address: [email protected] (A. Porras).

http://dx.doi.org/10.1016/j.neucom.2014.06.080 0925-2312/& 2014 Elsevier B.V. All rights reserved.

With regard to defining “real time multiple sensory driven motor control systems,” the issue has been hard to define [35] and most novel steps taken have been limited [36–41]. Common drawbacks relate to time response, output convergence, susceptibility to external noise, and most significantly, the proper “homogenization” of inputs from different sensors into a single intentionality vector. This is essential in interfacing with the motor plant. In addition, from a dynamic perspective response, time is a non-linear function relating to the magnitude of the error control required [42–46]. Thus, sensory-motor convergence speed is directly related to the number of recurrent cycles (discrete method) required for response stabilization following sensory input arrival time [4–7,47,48]. 1.1. Mathematical Model Following the issue of supervised, unsupervised and gradient algorithms, a more precise definition of convergence speed in large and/or multidimensional data sets is necessary. A prerequisite is the use of an open or “unsupervised” algorithm during sensory input acquisition, to be normalized for generating a compound image. Following image generation via geometrically co-registered inputs, the resultant n-dimensional matrix becomes the supervised parameter on which the center of mass [49] and the local density distributions are analyzed. The first part of the algorithm includes the normalization of the inputs. The basic initial assumption is that all the inputs are equivalent and that the multiplying factor ki for each, is 1. This multiplying factor could be a variable filter if its variation can be

12

A. Porras, R.R. Llinas / Neurocomputing 151 (2015) 11–33

Fig. 1. A) Video camera image of a non-processed three-dimensional object. B) Filtered video image of the 3D object. C) Sonar image of the 3D object D) Combination of images of a 3D object seen by the camera and sonar simultaneously.

associated with external factors or with the algorithm's cost functions. 0

x ði; jÞ ¼ kn Uxn ði; jÞ

ð1Þ

Such normalization procedure and the internal product for the factor ki when applied to all the inputs allow their combination into one single array[13,50–54] the base image. xm ði; jÞ ¼

x 0n1 ði; jÞ þ x 0n2 ði; jÞ þ… x 0nm ði; jÞ m

ð2Þ

As a first example of this procedure, the sonar and video input of a single rectangular object with a 3D projection (from the Materials and Methods section of this chapter) can be combined to observe the following results. The results in Fig. 1 correspond to the co-registered image space of the two input sensors (camera and sonar). The mathematical analysis requires that, the visual dimensions of the figure width and height be used as the limit values of the input vectors xvideo(width, height) and xsonar(width, height), as measured in pixels. The information received from the camera, after gray scale conversion and contrast increase filtering, is represented in each pixel as xvideo(i, j). See Fig. 1-B. As with the camera output, the sonar output generates a different representation of the same object, as is the case in the biological counterpart with visual and auditory senses and a common stimulus. The array from the sonar has the same dimensions as the one from the camera, xsonar(i, j). See Fig. 1-C. Thus far the unsupervised part of the algorithm generates the base image from the sensory inputs and will produce a normalized array of the same dimensions of the co-registered inputs of its summatory. xm ði; jÞ ¼

weight height k video xvideo ði; jÞ þ

∑

∑

i

j

ksonar xsonar ði; jÞ m

ð3Þ

Where, m ¼ k1 þ k2 þ … kn r n

ð4Þ

This assumes that both inputs are equivalent during the analysis and the constants ki have one (1) as a value. From the previous summation equations, the resultant base image is shown in Fig. 1-D. At this point, the normalized image X̄ m(i,j) has the information necessary to start a simple search to locate the object or objects, which generate the densities in the image. As a control, the center of mass is obtained, employing as a weight the 8 bit gray scale of each pixel. Therefore, massði; jÞ ¼ Pixelgrayscale ½xm ði; jÞ

ð5Þ

and in order to find the coordinates, Center mass ðiÞ ¼

height ∑width j ¼ 0 ∑i ¼ 0 jU massði; jÞ height ∑width j ¼ 0 ∑i ¼ 0 massði; jÞ

ð6Þ

and for j, Center mass ðjÞ ¼

height ∑width i ¼ 0 ∑j ¼ 0 iU massði; jÞ height ∑width i ¼ 0 ∑j ¼ 0 massði; jÞ

ð7Þ

The resultant base image with the center of mass located is represented on Fig. 2 (red circle). Following the unsupervised center of mass acquisition phase, the supervised algorithm determines pixel number location to determine the maximum area density site. The input array analyzed, the X̄ m(i,j), normalized sensor image, does not contain the expected solution to train a standard supervised algorithm. Instead, the system optimizes error determination based on the weight of all the gray scale pixels via a random sweep of the base image until converging into a maximum density area. An arbitrary size rectangle Pij of dimensions u and v is declared with origin coordinates given by a random generator algorithm,


13

The 28 value corresponds to the gray scale maximum value of each pixel. The resultant value is then used as the argument in the following equation, Sig Pij ffi Jumpt þ 1 ¼ Jumpt þ ρqffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 Sig 2 P ij

Fig. 2. Solution by the SGD Algorithm once the convergence point is reached. The thin green line denotes the pathway followed to convergence, the yellow circle indicate the convergence point, the red circle is the center of mass and the thick green line is the difference between the convergence and the center of mass. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article).

where area is: Area P ir jr ¼ ððir þ uÞ ir Þ U ð jr þ v jr Þ

ð8Þ

The total weight of the image is then calculated from the individual pixel summation of the base image ( Sum(X̄ m(i,j)). Once the initial weight is acquired, the rectangle Pij is subtracted from the base image and a new summation of pixels over the image is calculated. Weight Pir jr ¼ Weight ðAreaðxij Þ AreaðP i j ÞÞ r r

ð9Þ

From the difference between the previous weight and the new weight calculated without the randomly located rectangle, Pij, a loss function proportional to the image density, produces a gradient that can be used to locate a local maximum in the image. lossP ij ¼

1 ðWeight P ir jr ðtÞ Weight P ir jr ðt 1Þ Þ2 2

ð10Þ

Different from a typical SGD algorithm, where the weights vector multiplies all the inputs to change projection and reduce error function, the input sonar and camera images lack train algorithm output i.e. there is no weights vector to modify. Instead, jump magnitude and rectangular area determines direction, moving the base image until maximum loss function reaches a maximum and minimizes the loss function. The modified SGD algorithm proposed here could use the position of the object as a desired output, and use it to train the weights of future images. However, a fixed set of weights vector can not implement a comparison function defeating the possibility of finding a convergent solution that does not rely on memory or training to operate efficiently. However, convergence could be achieved by arbitrarily changing jump size, direction value and area of subtraction rectangle and assigning a random value for the direction. Two sigmoid functions based on the gradient of the error were used to produce next jump size and area values. Such sigmoid functions are commonly used in gradient algorithms due to their derivability and are a convenient option to test our algorithm [55,56]. Utilizing an arbitrary initial value for rectangular sides Pij in which u and v determine an equilateral square, the variable named ‘blockside’ is defined and it will be used as the normalized difference between the weights of the image: Sig Pij ¼

ðWeight Pir jr ðtÞ Weight Pir jr ðt 1Þ Þ blockside2 U 28

After the new jump value has been calculated, the now supervised algorithm, checks if the error value and jump size reach convergence. Using those two variables, the algorithm can define, with high precision, if the density area with in the rectangle Pij is a local maximum. If the convergence is not achieved, a new cycle of random direction displacement and weight image calculation is run and the cycle continues until convergence is achieved. The resultant image, once the convergence point is reached, is represented in Fig. 2 (Green pathway and yellow circle). Values for blocksize and learning rate were initially adjusted following trial and error, probing convergence times and final points determined as the areas where the density of whither pixels was larger compared to the size of the searching rectangle. Another factor to reach convergence was the convergence on the summatory of weights at the same rate of convergence of the blocksize and learning rate. According to other authors [55,56] an acceptable convergence point occurs when, log ρ Z t

2. Materials and Methods 2.1. General Description The system consist of two sensory inputs, video and sound detection (sonar), both sitting on a stationary platform 140 (cm) located in the center of a room away from walls and in the presence of 3D objects and images. The camera's visual range corresponds geometrically with the range detection area of the sonar, which is 114 86 (cm) (Fig. 3). The video camera is connected to a visual processor via USB and has a resolution of 640 480 pixels. The sonar is composed of two piezo electric transceivers of approximately 40 KHz, which are installed on a pan/tilt adapter with a radial mobility of 180 degrees in each direction. The information from the sonar is transferred into the processor through a dedicated acquisition board that transmits the distance values along with the visual information

ð11Þ

ð12Þ

ð14Þ

However, depending on algorithm closeness to the starting point, the resultant number of steps varied up to one order of magnitude. A final result image Fig. 2, in which a convergence point is compared against the center of mass [49], is generated once the SGD algorithm finishes indicating small differences when there was only one object detected, but very ambivalent when the number of objects was more than one (see results section).

where, 1 r Sig P ij r 1

ð13Þ

Fig. 3. Location of the Sonar, Camera and Visual Area.

14


from the camera. Both streams of information are acquired serially and analyzed in real time using Processings, a JAVA-based platform, which displays the two images and estimates the location of the objects based on correlation and stochastic gradient descent algorithms [2–7]. 2.2. Sonar – Theory And Specifications The basic principle of sonar, to generate a pulse of sound and respond to the eco generated when the sound reflects from an object in the sound path. By timing how long it takes for the echo to return, an accurate estimate can be made of the distance to the object. The sonar equipment used is SRF05 [57] (Fig. 4). An ultrasound pulse generated by the SRF05 is utilized as high frequencies perform better for short range, high precision needs. In addition as all the sonar requirements are built into the unit it's implementation into the system is enormously facilitated. The module has two 40khz ultrasonic transducers, one to transmit and one to receive. It also has an onboard PICs microcontroller which is used to generate the ultrasonic signal and listen for the echo, and a handful of other components which condition and detect the signals. Generating a pulse of at least 10us on the trigger line activates the unit. The SRF05 responds with a simple pulse, the duration of which is directly proportional to the distance of the object detected. With this system distance estimation as its duration in microseconds to yields the approximate distance in centimeters. 2.2.1. Calculating the Distance The calculation of distance is based on the speed of sound, (.0343 cm per microsecond). The time in microseconds multiplied by.0343 will give distance in centimeters. Given that the sound

received is an eco (a round trip) the final calculation is distance¼ duration n.0343/2. However distance ¼duration/58 provides a close-enough approximation [57]. The SRF05 can be triggered at 20 Hz. As a delay of 50ms to ensure the ultrasonic “beep” has faded away and will not cause a false echo on the next ranging. This determines the speed of pan and tilt used to scan the surface of the visual area. 2.2.2. Beam Pattern And Beam Width The beam pattern of the SRF05 is conical, with the width of the beam being a function of the surface area of the transducers (an invariant parameter). The actual detection zone will vary based on the size and shape of the object being detected. Smaller, rounder objects present a narrower detection zone, while larger, broader objects will have a wider detection zone. 2.2.3. Pan and tilt platform To increase the range of detection of the sonar, a panning and tilting platform was custom built. Fig. 5A and B. The platform gives the added benefit of pointing the sonar to any specific position in the visual horizon. It uses two hitechs HS311 servomotors with a resolution of 11 radial degree. At a distance of 140 (cm), from the sensors to the object the panning angle corresponds to approximately 501 degrees to cover the width of 114 (cm). The tilting angle from the same position is of approximately 301 degrees. The resultant resolution is blocks of 13 17 pixels (50 13 ¼650 pixels wide, 30 17¼ 510 pixels height). The SRF05 sonar is controlled by a microcontroller using an Arduinos Mega 2516U3 board. The electronics work as an interface

Fig. 4. Sonar SRF05.

Fig. 5. Pan and tilt platform. A and B) Custom platform. C) Scanning pattern.


between the sonar and the computer reading from the time pulses of the sonar and translating the information into strings of characters through the serial port. Each sweep of the sonar takes 15 s, generating a total of 4 full frames per minute (Fig. 5-C). Once the serial port recognizes and configures automatically the device, the assigned serial port starts receiving strings of coordinates with the angles and distance measured as follows: X42Y27V113 In this case the two numbers after the letter “X” indicate the horizontal angle and the panning position of the sonar. The numbers following the letter “Y” indicate the tilt angle and the last numbers after the letter “V” contain the distance at which the sonar is from the detected object.

Fig. 6. Test 1a. (Blue) Average number of steps to reach SGD convergence. (Red) Average distance of he convergence point from the center of mass. N ¼55. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article).

15

Because the information strings are not stored, and are asynchronous, the flow of information from the sonar to the processor can be read, or ignored, without disruption of the sonar's function. This characteristic is relevant because it compartmentalizes the functions of each sensor/input, allowing a true parallel architecture when the amount of inputs or receivers increases, as occurs in biological systems. The Arduinos code used to control the sonar including the pan/ tilt platform is in the (Appendix A). 2.3. Camera – Visual Input The camera used is a CMOS Logitechs webcam HD C270. The maximum resolution of the camera is 1280 720 pixels but in the present application, the only resolution used is 640 480 pixels with a frame rate of 15 frames per second. The acquisition of the data from the camera is done through USB and accessed directly using Processings. The algorithm of the camera filters the images to increase the contrast with the black background of a far wall. This allows a balance between the auto gain and auto white balance of the webcam, to replicate ideal conditions. A secondary advantage to filtering is noise reduction and the possibility to camouflage 3D objects of the same color of the background “visible” to the sonar, but not to the camera. The webcam, in contrast to the sonar, does not require a pan/tilt platform to cover the total visual area and therefore remains static during the experiments. Following code digital filtering, the resulting 640 480 color values are converted into an 8 bit gray scale equivalent. Here color information reduction allows object recognition by brightness analysis with the additional advantage of reducing processor

Fig. 7. Image of the SGD algorithm pathway (Green line) and its final convergence point (Yellow circle). (Red circle) Center of mass of the image. (Thick green line) Distance difference of the convergence SGD point and the center of mass. B) (Blue) Average number of steps to reach SGD convergence. (Red) Average distance of the convergence point from the center of mass. N ¼59. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article).

Fig. 8. A) Image of the SGD algorithm pathway (Green line) and its final convergence point (Yellow circle). (Red circle) Center of mass of the image. (Thick green line) Distance difference of the convergence SGD point and the center of mass. B) (Blue) Average number of steps to reach SGD convergence. (Red) Average distance of the convergence point from the center of mass. N ¼ 38. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article).

16


Fig. 9. Test 2a. A) Detection of two objects of 14 9 30.5 (cm) and 14 9 (cm) by the camera and the sonar. B) Sonar and video camera input. C & D) Image of the SGD algorithm pathway (Green line) and its final convergence point (Yellow circle). (Red circle) Center of mass of the image. (Thick green line) Distance difference of the convergence SGD point and the center of mass. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article).

Fig. 10. The percentage of trials (Blue) 3D object located on the upper left side and (Red) 2D object located on the lower right side. N ¼ 47. B) (Blue) Average number of steps to reach SGD convergence to any object. (Red) Average distance of the convergence point from the center of mass to the upper left object. (Green) Average distance of the convergence point from the center of mass to the lower right object. N ¼ 47. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article).

Fig. 11. Test 2b. Sonar alone off center. A) Signal from the sonar after filtering. B) Image of the SGD algorithm pathway (Green line) and its final convergence point (Yellow circle). (Red circle) Center of mass of the image. (Thick green line) Distance difference of the convergence SGD point and the center of mass. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article).


17

workload. However, if advantageous, full color information may also be implemented. At this point, however, brightness analysis is sufficient to implement stochastic gradient descent localization. The following Processings code is used to control the video:

code written in Processings is in the Appendix B. The libraries written in JAVAs are in the Appendix C.

if(video.available()) { video.read (); video.loadPixels (); for (int i¼ 0; i o numPixels; i þ þ ) { color pixelValue ¼video.pixels[i]; int pixelBrightness¼pixelValue & 0xFF; //Masks the brightness in every pixel if (pixelBrightness 4 threshold) { // If the pixel is brighter than the threshold brillo[i] ¼ pixelBrightness; // threshold value, make it the pixelBrightness } else { // Otherwise, brillo[i] ¼ black; // make it black } } }

The following sets of results describe six (6) different setups in which objects of varying sizes were placed in the receptor field (camera and/or sonar). Among the parameters analyzed in each case were 1) the final point of convergence reached by the SGD algorithm, 2) the number of steps required to reach such a point and 3) the difference with the -separately calculated- center of mass of the image. Other parameters analyzed were the constants of the localization algorithm, such as adaptation rate and average jump size, which can indicate the efficiency of the searching method and its bias, depending on the size of the difference with the center of mass of the image (calculated in Matlabs). However, to obtain a common base of comparison, the sensory adaptation rate and initial jump size values remained constant throughout all the experiments. Each experiment has at least four (4) main figures, including:

Here the use of Processings allows code parallel execution. In the above highlighted part of the code, webcam pixels (video. Pixels[i]) are digitally filtered, and the new matrix (brillo[i]) may be addressed at any time, by other simultaneously running codes. This configuration is particularly convenient when considering that the resulting matrix can be fully or partially accessed. This is significant, when slow processing sensors such as sonar information is ready for parallel information evaluation. The rest of the

3. Results

– The real visual setup, – The filtered image analyzed by the algorithms, – The results of the center of mass and SGD mathematical analysis plotted on the previously filtered image, and – One or two charts to indicate the results of each test. The overall sensory receptive field (SRF) as described previously, has a dimension of 114 86 (cm). The sonar and camera point to the SRF, 140 (cm) in front. The visual and sonar fields are geometrically co-registered, which allows for direct comparison. 3.1. Test 1a

Fig. 12. (Blue) Average number of steps to reach SGD convergence. (Red) Average distance of the convergence point from the center of mass. N ¼ 15. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

The first test was organized such that, a rectangular object of 14 9 (cm) was placed in front of the SRF at a distance of 30.5 (cm). Fig. 1-A. Using the camera background noise filter and sonar signals, the system is able to subtract the shadows and colors from the image; while the resultant 8 bit gray scale is shown to optimize algorithm execution. The result is an image with a normalized visual and sonar image convergence. The image of the final results after filtering and normalization are presented in Fig. 1-D. In Fig. 1-D the image contains two inputs: the camera as a welldefined rectangle at the center of the screen, representing brightness in a gray scale, and the sonar signal represented as small

Fig. 13. test 2 C. A) Image base of two objects, upper left 3D 14 9 30.5 (cm) and lower right 2D 14 9 (cm), detected by the camera. B) Image of the SGD algorithm pathway (Green line) and its final convergence point (Yellow circle). (Red circle) Center of mass of the image. (Thick green line) Distance difference of the convergence SGD point and the center of mass. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

18


rectangular patches representing, with less precision, the reflected ultrasound detected during the panning and tilting process of the sweep distance localization. The intensity of each rectangle (whiter¼ closer, darker¼ farther) determines its image weight, a key factor when running the SGD and center of mass algorithms. The analysis applied on the image first generates the center of mass, red circle, and immediately after draws the pathway of the SGD, which starts in a random point on the image and continues drawing a thin green line as it moves. The final point of the SGD localization path is marked with a yellow circle and the difference with the center of mass is depicted with a thick green line. See Fig. 2. This test was run 55 times and the paths and end points were actually different, but equivalent, in each case. Indeed, the results show large variation in the number of steps, but a relative constant convergence in the distance difference from the center of mass. See Fig. 6. The bar on each graph from Fig. 6 indicates the standard deviation evaluated in the data pool of each test.

3.2. Test 1b The external setup of this test is the same as in experiment 1a, but only the sonar input is addressed. Increased sonar sensitivity was utilized and the decimation used to filter background, reflection and random activation was recalibrated. Under these new conditions the center of mass of the image was still maintained. The result is an image with an increased number of small rectangles and/or and a whiter color concentration in those adjacent to the center of the detected object. The sensitivity of the sonar filtration is maintained from this point on through all the following tests. See Fig. 1-C. As previously described for test 1a, the center of mass and SGD algorithms are implemented resulting in images such as Fig. 7-A bellow. The values of average number of steps of the SGD algorithm to reach the final point (Yellow circle) and the average number of pixels distance from the center of mass of the image are indicated in Fig. 7-B. 3.3. Test 1c The external setup of this test was the same as in test 1a, the only difference being that video is the only input analyzed. The threshold noise filter and the mathematical parameters of the center of mass and SGD algorithms are maintained as in test 1a. The base image is represented in Fig. 1-B. After the algorithms are processed, the new images with pathways are generated. See Fig. 8-A. The statistical analysis is described in Fig. 8-B.

Fig. 14. test 2 c. (Blue) Average number of steps to reach SGD convergence. (Red) Average distance of he convergence point from the center of mass. N¼ 17. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

3.4. Test 2a In this setup, two different objects are utilized. One is located in the center of the upper left quadrant of the image and stands from

Fig. 15. A) Detection of two objects of 14 9 30.5 (cm) by the sonar. B) Image base of two objects, upper left 3D 14 9 30.5 (cm) and lower right 3D 14 9 30.5 (cm), detected by the sonar. C) Image of the SGD algorithm pathway (Green line) and its final convergence point (Yellow circle). (Red circle) Center of mass. (Thick green line) Distance difference of the convergence SGD point and the center of mass. Convergence point in this case is the upper left. D) Image of the SGD algorithm pathway (Green line) and its final convergence point (Yellow circle). (Red circle) Center of mass. (Thick green line) Distance difference of the convergence SGD point and the center of mass. Convergence point in this case is the lower right. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)


19

Fig. 16. A) The percentage of trials (Blue) 3D object located on the upper left side and (Red) 3D object located on the lower right side. N ¼ 65. B) (Blue) Average number of steps to reach SGD convergence to any object. (Red) Average distance of the convergence point from the center of mass to the upper left object. (Green) Average distance of the convergence point from the center of mass to the lower right object. N ¼ 65. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article).

Fig. 17. A) Two 2D objects of the same dimensions, 14 9 (cm) located in the upper right and lower left quadrants of the visual space. B) Image base of two objects, upper right 2D 14 9 (cm) and lower left 2D 14 9 (cm), detected by the camera. C) Image of the SGD algorithm pathway (Green line) and its final convergence point (Yellow circle). (Red circle) Center of mass. (Thick green line) Distance difference of the convergence SGD point and the center of mass. Convergence point in this case is the lower left. D) Image of the SGD algorithm pathway (Green line) and its final convergence point (Yellow circle). (Red circle) Center of mass. (Thick green line) Distance difference of the convergence SGD point and the center of mass. Convergence point in this case is the upper right. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article).

Fig. 18. test 4. A) The percentage of trials (Blue) 2D object located on the lower left side and (Red) 2D object located on the upper right side. N ¼62. B) (Blue) Average number of steps to reach SGD convergence to any object. (Red) Average distance of the convergence point from the center of mass to the upper right object. (Green) Average distance of the convergence point from the center of mass to the lower left object. N ¼62. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

20


Fig. 19. A) Two 2D objects of 5 5 (cm) located in the upper center and 22.5 22.5 (cm) lower center half of the visual space. B) Image base of two objects, upper center 2D 5 5 (cm) and lower center 2D 22.5 22.5 (cm), detected by the camera. C) Image of the SGD algorithm pathway (Green line) and its final convergence point (Yellow circle). (Red circle) Center of mass. (Thick green line) Distance difference of the convergence SGD point and the center of mass. Convergence point is the upper center. D) Image of the SGD algorithm pathway (Green line) and its final convergence point (Yellow circle). (Red circle) Center of mass. (Thick green line) Distance difference of the convergence SGD point and the center of mass. Convergence point in this case is the center down. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Fig. 20. Test 5. A) The percentage of trials (Blue) 2D object located on the center up and (Red) 2D object located on the center down side. N ¼ 74. B) (Blue) Average number of steps to reach SGD convergence to any object. (Red) Average distance of the convergence point from the center of mass to the center up object. (Green) Average distance of the convergence point from the center of mass to the center down object. N ¼ 74. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

the background at 30.5 (cm) of distance. The second is located on the center of the lower right quadrant with no 3D projection. Both objects have the same dimensions, 14 9 (cm), but not the same color. See Fig. 9-A. In this test, the tones of blue are not the same for the two objects, the video camera detects, after a filter is applied, only the one located in the lower right and the sonar only detects the one standing in front of the wall at 30.5 (cm) of distance - upper left. See Fig. 9-B. The purpose of this setup with multiple objects is to evaluate the “preference”, if any, of the SGD algorithm over different distributions of weight at equal distance from the center of mass of the image. The results were divided in two groups. The first group collected the convergence point trials located in the upper left area, Fig. 9-C, and the second targeted the lower right object as its final SGD search algorithm point, Fig. 9-D. The results show that in about 52% of cases, the SGD algorithm reaches the object located in the lower right part of the screen and

48% of the time if the object is located on the upper left. The sample size of this test was 47 trials. See Fig. 10-A. The number of steps required to reach each object was analyzed using the distance in pixels from the center of mass of the image as in the previous experiments. In this particular case, the center of mass is not in the center of the image because the input from the sonar is scattered, in contrast to the camera's, therefore shifting the center of mass towards the lower right of the image, Fig. 10-B. 3.5. Test 2b The external setup of this experiment is maintained as in experiment 2a, and the sonar is the only input. The threshold noise filter and the mathematical parameters of the center of mass and SGD algorithms are maintained as in test 2a. The Fig. 11-A has the base image. After running the image localization algorithm, the conversion point located in the upper left is as in Fig. 11-B. The sample size of


21

Fig. 21. A) Two 3D objects of 5 5 30.5 (cm) located in the upper center and 14 9 30.5 (cm) lower center half of the visual space. B) Image base of two objects, upper center 3D 5 5 30.5 (cm) and lower center 3D 14 9 30.5 (cm), detected by the sonar and camera. C) Image of the SGD algorithm pathway (Green line) and its final convergence point (Yellow circle). (Red circle) Center of mass. (Thick green line) Distance difference of the convergence SGD point and the center of mass. Convergence point in this case is the upper center. D) Image of the SGD algorithm pathway (Green line) and its final convergence point (Yellow circle). (Red circle) Center of mass. (Thick green line) Distance difference of the convergence SGD point and the center of mass. Convergence point in this case is the center down. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Fig. 22. Test 6a. The percentage of trials (Blue) 3D object located on the center up and (Red) 3D object located on the center down side. N ¼ 63. B) (Blue) Average number of steps to reach SGD convergence to any object. (Red) Average distance of the convergence point from the center of mass to the center up object. (Green) Average distance of the convergence point from the center of mass to the center down object. N ¼ 63. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

this test was 15 trials. The results, based on the obtained data, are shown in Fig. 12. 3.6. Test 2c The external setup of this test is the same as in experiment 2a, with video being the only input. The threshold noise filter and the mathematical parameters of the center of mass and SGD algorithms are maintained as in test 2a. The Fig. 13- A has the base image. As previously described, the center of mass and SGD algorithms are applied to the base image and, as a result, the following images are obtained. (Fig. 13-B). The sample size of this test is 17 trials. The results, based on the obtained data, are in Fig. 14. 3.7. Test 3 Two objects of equal dimensions are located equidistant from the center of the image in the upper left and lower right as shown in the

next image. Both objects have a frontal dimension of 14 9 (cm) and stand at a distance of 30.5 (cm) from the background. See Fig. 15-A. The analysis, in this test, addresses sonar signals only. The Fig. 15-B has the resulting base image after threshold filtration. Image symmetry is represented by the reflected sonar detection. Therefore, if the object's surface has small wrinkles that could change the projection of the sound waves, the resultant “perception” of the object will vary as well. The above is a good example of how the system could see two apparently equal objects as different, given their sonar representation. The results were divided in two groups. The first group targeted the upper left object as its final point of destination, Fig. 15-C. And the second group targeted the lower right object as its final SGD search algorithm point, Fig. 15-D. The sample size of this test is 65 trials. Fig. 16-A shows the percentage distribution drawn in Fig. 16-A. The average, standard deviation and distances from the center of mass of each case are represented in Fig. 16-B. There seems to be an inverse relationship

22


Fig. 23. A) Two 3D objects of 5 5 30.5 (cm) located in the upper center and 14 9 30.5 (cm) lower center half of the visual space. B) Image base of two objects, center up 3D 5 5 30.5 (cm) and center down 3D 14 9 30.5 (cm), detected by the sonar. C) Image of the SGD algorithm pathway (Green line) and its final convergence point (Yellow circle). (Red circle) Center of mass. (Thick green line) Distance difference of the convergence SGD point and the center of mass. Convergence point in this case is upper center. D) Image of the SGD algorithm pathway (Green line) and its final convergence point (Yellow circle). (Red circle) Center of mass. (Thick green line) Distance difference of the convergence SGD point and the center of mass. Convergence point in this case is center down. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Fig. 24. Test 6b. A) The percentage of trials (Blue) 3D object located on the center up and (Red) 3D object located on the center down side. N ¼71. B) (Blue) Average number of steps to reach SGD convergence to any object. (Red) Average distance of the convergence point from the center of mass to the center up object. (Green) Average distance of the convergence point from the center of mass to the center down object. N ¼ 71. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

3.8. Test 4

Fig. 25. The averages of tests using sonar and video signals to generate the base image employed by the SGD convergence algorithm.

between the scatteredness of the image and the number of steps required by the SGD algorithm to reach a convergence state. These results will be analyzed further in more detail.

Following the same procedure as in test 3, the objects in this case are 2D only, equidistant from the center of the image and of equal dimensions – 14 9 (cm) each. The localization of the objects is upper right and lower left quadrants of the image, as illustrated in Fig. 17-A. Since the sonar representation is irrelevant in this case, only video results will be analyzed. Using non-3D objects reduces image scatteredness allowing evaluation of artifact number on the number of steps to reach convergence. Fig. 17-B depicts base image after threshold filtering. The results were grouped in two sets. The first set targeted the lower left object as its final point of destination, Fig. 17-C. The second set addressed upper right convergences as its final SGD algorithm search point, Fig. 17-D. The sample size of this test is 62 trials. The results, based on the obtained data, have the percentage distribution of Fig. 18-A. The


23

The sample size of this experiment is 63 trials. The results produced the percentage distribution of the Fig. 22-A. The average, standard deviation and distances from the center of mass of each case are represented in Fig. 22-B. 3.11. Test 6b

Fig. 26. The averages of tests using sonar signals to generate the base image employed by the SGD convergence algorithm.

The setup of this test is the same as in test 6a. Sonar is the only input. The threshold noise filter and the mathematical parameters of the center of mass and SGD algorithms are maintained as in test 6a. Fig. 23-B is the base image from Fig. 23-A. The convergence points in this case is not as definitive as in the previous 6a test, as video signal subtraction reduced small object weight and the center of mass is moved closer to the location of the center down object. However, the SGD algorithm was able to identify a second object located in the upper center more than 40 percent of the time Fig. 23-C. And the second group contains the center down convergences as its final SGD algorithm search point, Fig. 23-D. The sample size of this test is 71 trials. The results, based on the obtained data, have the percentage distribution indicated in Fig. 24-A. The average, standard deviation and distances from the center of mass of each case are represented in Fig. 24-B.

4. Analysis of Results and Conclusions Fig. 27. The averages of tests using video signals to generate the base image employed by the SGD convergence algorithm.

average, standard deviation and distances from the center of mass of each case are represented in Fig. 18-B. 3.9. Test 5 The aim of this test is to evaluate the response of the SGD algorithm with objects -squares- with more than 500%, asymmetrical differences in area, two objects were used: small square, 5 5 (cm) and large square, 22.5 22.5 (cm). None of the objects have a 3D projection. The small square was located in the middle of the upper half of the image and the large one at lower half center. The only sensor used is the camera. See Fig. 19-A. The camera resultant after threshold filtering is presented in Fig. 19-B. The results were divided in two groups. The first group targeted the upper center object as its final point of destination, Fig. 19-C. The second group depicts center down convergences as its final SGD algorithm search point, Fig. 19-D. The sample size 63 trials. The results, based on the obtained data, have the percentage distribution of Fig. 20-A. The average, standard deviation and distances from the center of mass of each case are represented in Fig. 20-B.

Important aspects that require additional analysis concern SGD convergence speed under different conditions and the consistency of such a speed in multiple scenarios. Evaluating these can be addressed, by comparing the average number of steps to convergence in tests with similar input characteristics, regardless of the number of objects detected. In the first scenario the sonar and video inputs are used simultaneously to generate the base image. The tests with such characteristics are 1a, 2a and 6a respectively. Assuming that the standard deviation in each case is not relevant, due to the starting point randomization, the average the number of steps (this unit translates into speed directly depending on the hardware used) can then be compared with tests in which only sonar or video were used. The tests contained in Fig. 25 used both sensory inputs, sonar and video. The resultant average number of steps of the experiments using sonar and video signals is 285.86 steps. The second scenario is based on tests using only sonar to generate the base image regardless of the number of objects, see Fig. 26. The resultant average number of steps using sonar only signals to generate the base image using SGD algorithm is 423.03 steps. The third and last scenario is based on tests using sonar to generate the base image regardless of the number of objects, see Fig. 27. The resultant average number of steps of the test using sonar only signals to generate the base image employed by the SGD algorithm is 689.95.

3.10. Test 6a

4.1. Conclusions

Test 6 aims, to evaluate SGD response to -squares- with more than 500% asymmetrical difference in area. The two objects are used and their dimensions are: small square, 5 5 (cm) and large square, 14 9 (cm). Both objects have a 3D projection of 30.5 (cm). The small square is located mid upper half and the large square in mid lower half. All sensors are used in this setup, video and sonar. See Fig. 21-A. The resultant normalized sonar and video image after threshold filtering is presented in Fig. 21-B. Two groups are presented. One targeted the upper center object, Fig. 21-C, the second addresses the center down convergences as its final SGD algorithm search point, Fig. 21-D.

Efficiency estimation using the number of steps to reach convergence is affected by the number of inputs used, among other factors. The results from the tests described above indicate that, assuming constant step processing time, the speed of the combined sensory inputs image is 50% faster versus the fastest convergence with only sonar and, 250% faster against fastest video convergence. Furthermore, the speed of the average sonar-only image versus the video-only image has a ratio of 1.7:1, which indicates an apparent preference for images with a more scattered pattern over a more detailed one. Further analysis is required to evaluate if the precision over more trials decreases as scatteredness increases.

24


The last conclusion indicates that the rule of summation for the number of steps versus the number of sensory inputs doesn't appear to be linear. More analysis at this point is required to evaluate a larger number of inputs with various pattern distributions and standard deviations. However, given the structure of the algorithm and the biological limitations of energy in the animal model, there is a limiting maximum number of inputs until an asymptotic behavior of inputs versus speed is reached. Basically, the efficiency of the response requires “only so much” information to obtain an optimal solution. As general results that could be compared further against similes in biology we observed:

The apparent non-linear correlation between the number of

sensory inputs, the steps required to convergence and a possible asymptotic behavior. This means that the summation of sensory inputs does not generate a proportional increase in the reduction of time to convergence time. The suspected relationship is non-linear and could have an asymptotic behavior as the number of inputs to the system increase. In biological terms, the solution increases in precision if the afferent signals from the sensors, such as in the optical tectum, are stronger or if the added resultant projections reach an activation threshold. However, the summation is non-linear and the number of afferents required to produce neuronal activation has an asymptotic convergence. In the examples where the solid square or rectangle was exclusively detected by the camera, versus the same examples detected by the sonar, the “scatternedness” rendered by the last one (sonar) demonstrated a rapid convergence time traduced in less steps. Interesting enough, the search patterns of the SGD algorithm had no predisposed programmed preference for density distributions in which the area of concentration had any particular shape or value. However, the tests in which scattered patterns were introduced, the convergence results were faster and equally or more accurate, compared to the center of mass, than the ones in which the detected object had larger contrast areas and peaks of density distribution (tests with camera only).

Acknowledgements Support for this paper was provided by the Office of Naval Research Grant N00014-09-1-1006 to Rodolfo Llinas.

Appendix A

// Servo Sonar // by Alex Porras // This code controls the pan/tilt platform and the SRF05 Sonar #include oServo.h 4 Servo pan; // create servo object to control a servo Servo tilt; // a maximum of eight servo objects can be created int pan_min ¼75; //75 int pan_max ¼122; //110 int tilt_min ¼70; //70 int tilt_max ¼ 97; //90 int pos_pan ¼0; // variable to store the servo position int pos_tilt ¼0; // variable to store the servo position const int laserPin ¼ 11; int echoPin ¼2; // the SRF05's echo pin int initPin ¼3; // the SRF05's init pin unsigned long pulseTime ¼ 0; // variable for reading the pulse unsigned long distance ¼0; // variable for storing distance void setup() { pan.attach(10); tilt.attach(9);

One biological reason could be the immune? Unresponsive? Posture to background noise or the role it plays in correlated signal analysis, such as in the visual cortex [53,54,56,58]. More research concerning these findings is required. The uniqueness of the system in finding density distributions independent of their area, rather than locating the centers of mass when the detection required included more than one object, is rather surprising. False positives are encountered when centroid algorithms are applied to find mass distributions in systems that are not isometric or isomorphic, pointing to a proper geometrical solution, but useless when locating patterns or other arrays. In biology, averaging sensory inputs is not adequate since each sensor has a different modulated weight affected by other sensory inputs and/or external conditions, such as with barn owls and sunset [59], their visual systems are less relevant and their sound detection systems take over the prey localization task. Constant modulation and proper identification of targets has to be precise and not averaged, especially not in real life where isometric distribution is not common. The speed of the SGD algorithm has no direct relationship with system input numbers, but rather, with the size of the image analyzed. Therefore, convergence times required to address bigger areas seem proportional to those shown in the results section of this chapter. Few steps are required when more sensory inputs converge such as the normalization phases and the multiplication for the input weights, if other external conditions are involved in the system. A future area of research concerns the modifying of sensory input dynamically, so that their value becomes a composed parametric function, such as light intensity, level of noise, etc that can optimize convergence evaluation.

// attaches the servo on pin 10 to the servo object


// make the init pin an output: pinMode(initPin, OUTPUT); // make the echo pin an input: pinMode(echoPin, INPUT); // set the digital pin as output: pinMode(laserPin, OUTPUT); // initialize the serial port: Serial.begin(9600); } void loop() { digitalWrite(laserPin, HIGH); pan.write(pan_min); tilt.write(tilt_min); for(pos_tilt ¼tilt_min; pos_tilt otilt_max; pos_tilt þ ¼ 1){ tilt.write(pos_tilt); for(pos_pan ¼pan_min; pos_pan opan_max; pos_pan þ ¼1) // goes from 0 degrees to 180 degrees { // in steps of 1 degree pan.write(pos_pan); // tell servo to go to position in variable ‘pos_pan’ Sonar(); } pos_tilt ¼ pos_tilt þ1; tilt.write(pos_tilt); for(pos_pan ¼pan_max; pos_pan 4 ¼pan_min; pos_pan- ¼ 1) { // goes from 180 degrees to 0 degrees pan.write(pos_pan); // tell servo to go to position in variable ‘pos’ Sonar(); } } for(pos_tilt ¼tilt_max; pos_tilt 4 tilt_min;pos_tilt- ¼ 1){ // tilt.write(pos_tilt); for(pos_pan ¼ pan_min; pos_pan opan_max; pos_pan þ ¼ 1) // goes from 0 degrees to 180 degrees { // in steps of 1 degree pan.write(pos_pan); // tell servo to go to position in variable ‘pos’ Sonar(); } pos_tilt ¼pos_tilt - 1; tilt.write(pos_tilt); for(pos_pan ¼pan_max; pos_pan 4 ¼pan_min; pos_pan- ¼1) // goes from 180 degrees to 0 degrees { pan.write(pos_pan); // tell servo to go to position in variable ‘pos’ Sonar(); } } } void Sonar(){ digitalWrite(initPin, LOW); delayMicroseconds(50); digitalWrite(initPin, HIGH); // send signal delayMicroseconds(50); // wait 50 μs for it to return digitalWrite(initPin, LOW); // close signal pulseTime ¼pulseIn(echoPin, HIGH); // calculate time for signal to return distance ¼ pulseTime/58; // convert to centimetres delay(50); // waits 15ms for the servo to reach the position // if(pos_pan - pan_min o10) Serial.print(“X0”); // print leading X to mark the following value as degrees else Serial.print(“X”); // print leading X to mark the following value as degrees Serial.print(pos_pan - pan_min); // current servo pan position if(pos_tilt - tilt_min o10) Serial.print(“Y0”); // print leading X to mark the following value as degrees else Serial.print(“Y”); // print leading X to mark the following value as degrees Serial.print(pos_tilt - tilt_min); // current servo tilt position Serial.print(“V”); // preceeding character to separate values Serial.println(distance);}

25

26


Appendix B

import gDescent.n; /nn Sonar and Video - Thesis n n by Alex Porras. n n SGD on sonar and video signals. n/ import processing.video.n; import processing.serial.n; // import serial library Serial myPort; // declare a serial port Capture video; PrintWriter outputfile; int numPixels; int blockSize ¼10; int[] brillo; int black¼0; color white ¼color(255); color red¼color(10,220,10); String myString ¼ null; int lf ¼10; // Linefeed in ASCII int pan_min ¼75; //75 int pan_max ¼122; //110 int tilt_min ¼70; //70 int tilt_max ¼ 97; int degree_pan¼0; // servo pan position in degrees int degree_tilt¼ 0; // servo tilt position in degrees int value ¼ 0; // value from sensor int matrixSensor_pan ¼50; int matrixSensor_tilt ¼29; int[][] newValue¼new int[matrixSensor_pan][matrixSensor_tilt]; // create an array to store each new sensor value for each servo position int[][] oldValue ¼new int[matrixSensor_pan][matrixSensor_tilt]; // create an array to store the previous values. int[][] outSonarMatrix ¼ new int[640][480]; int[][] outTotalMatrix ¼ new int[640][480]; int firstRun¼0; // value to ignore triggering motion on the first 2 servo sweeps int flush ¼0; int min_dist ¼90; int max_dist ¼140; int threshold ¼132; // Set the threshold value - 135 sonar int sweepSonar¼0; int indexTrack ¼0; int[][] trackPos ¼new int[2][ 5000 ]; int file_name ¼0; void setup() { size(640, 480); String cameras[]¼ Capture.list(); // println(Capture.list()); // Uses the default video input, see the reference if this causes an error video ¼new Capture(this, cameras[19]); numPixels¼ width n height; brillo ¼new int[numPixels]; video.start(); myPort ¼new Serial(this, Serial.list()[4], 9600); myPort.clear(); //Create a new file in the sketch directory outputfile¼ createWriter(“test6b/sonar_coordinates.txt”);


noStroke(); smooth(); } void draw() { //¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ SONAR¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼ // background(0); if (sweepSonaro 1500) { while (myPort.available() 40) { String xString ¼myPort.readStringUntil(lf); // read the serial port until a new line if (xString! ¼null) { // if theres data in between the new lines xString ¼trim(xString); // get rid of any whitespace just in case if (xString.length() ¼ ¼9 || xString.length() ¼ ¼ 10){ String getX¼xString.substring(1,3); // get the value of the servo position pan String getY¼xString.substring(4,6); // get the value of the servo position tilt String getV¼ xString.substring(7, xString.length()); // get the value of the sensor reading degree_pan¼Integer.parseInt(getX); // set the values to variables degree_tilt¼ Integer.parseInt(getY); // set the values to variables value ¼ Integer.parseInt(getV); if (value o threshold) { // If the pixel is brighter than the value ¼black; // threshold value, make it white } // println(“PAN: “þdegree_pan þ“ TILT: “þ degree_tilt þ“ VALUE: “þ value); oldValue[degree_pan][matrixSensor_tilt-1 - degree_tilt]¼newValue[degree_pan][matrixSensor_tilt-1 - degree_tilt]; // store the values in the arrays. newValue[degree_pan][matrixSensor_tilt-1 - degree_tilt] ¼value; sweepSonar þ þ ; } } } } ExtendMatrix(); //generates a 640 480 matrix from the sonar read outs // //¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼VIDEO ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼ // if(video.available()) { video.read(); video.loadPixels(); for (int i ¼0; io numPixels; iþ þ) { color pixelValue ¼video.pixels[i]; int pixelBrightness ¼pixelValue & 0xFF; //Maskss the blue color in every pixel if (pixelBrightness 4threshold) { // If the pixel is brighter than the brillo[i] ¼pixelBrightness; // threshold value, make it the pixelBrightness } else { // Otherwise, brillo[i] ¼black; // make it black } } } // //¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼AVERAGE BETWEEN SONAR AND VIDEO¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼ averageMatrix(); //generates the average between Sonar and Video and puts in into a 640 480 matrix for (int y ¼0; yoheight; yþ þ) { for (int x ¼0;x owidth; xþ þ) { stroke(outTotalMatrix[x][y]); //average matrix // stroke(brillo[ynwidth þx]); // stroke(outSonarMatrix[x][y]); point(x,y); } } if(sweepSonar 41498){ GradientDescent gd ¼new GradientDescent(); gd.initializeImg(outTotalMatrix);

27

28


gd.run(true); //true ¼ random; false ¼0, 0 //Draw the trajectory of the centroid's search indexTrack¼gd.indexTrack; outputfile.println(“4 4 4 4 FILE: “ þfile_name þ“ ——— JUMPS: “ þindexTrack); for(int i ¼0; i oindexTrack; i þ þ){ trackPos[0][i] ¼gd.trackPos[0][i]; trackPos[1][i] ¼gd.trackPos[1][i]; outputfile.println(trackPos[0][i] þ “,” þtrackPos[1][i]); } paintTrajectory(); // Draw a large, yellow circle at the final pixel fill(255, 204, 0, 128); ellipse(gd.x0, gd.y0, 100, 100); save(“test6b/video_sweepV2_“ þ file_name þ“.tif”); file_name þ þ; //save(“video_sweep10.jpg”); //exit(); } } // ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ END¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼ ¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼¼ // void keyPressed() { outputfile.flush(); // Writes the remaining data to the file outputfile.close(); // Finishes the file exit(); // Stops the program } int sumImage(){ loadPixels(); int sumTotal¼0; for(int i¼ 0; i onumPixels; iþ þ){ sumTotal¼ sumTotal þ(pixels[i] & 0xFF); } return sumTotal; } void ExtendMatrix(){ for(int j¼0; j omatrixSensor_tilt-1; jþ þ ){ for(int i ¼0; io matrixSensor_pan-1; i þ þ){ for(int y¼ 0; yo17; yþ þ){ for(int x¼ 0; x o13; x þ þ ){ if((x þin13) owidth && (yþ jn13) oheight){ outSonarMatrix[x þin13][yþjn17]¼ newValue[i][j]; } } } } } } void averageMatrix(){ for(int j ¼0; joheight; j þ þ){ for(int i ¼0; io width; i þ þ){ // outTotalMatrix[i][j] ¼ (((brillo[jnwidth þi]) þ outSonarMatrix[i][j])/2) & 0xFF; // outTotalMatrix[i][j] ¼ brillo[jnwidth þ i] & 0xFF; //video only outTotalMatrix[i][j]¼(outSonarMatrix[i][j]) & 0xFF; //sonar only } } } void paintTrajectory(){ for(int j ¼0; jo indexTrack -1; j þ þ){ stroke(red); line(trackPos[0][j],trackPos[1][j],trackPos[0][jþ1],trackPos[1][j þ1]); } }

//average


Appendix C

package gDescent; import static java.lang.System.out; import java.lang.String; import java.io.BufferedReader; import java.io.FileReader; import java.io.IOException; import java.util.Random; public class GradientDescent { public int width ¼ 640; public int height ¼480; public int[][] img; //Original image public int[][] cImg; public int blocksize ¼40; public int x0¼0; public int y0 ¼0; public int stepSize ¼20; public int stopCriteria¼0; public double learningRate ¼15; public int[][] trackPos; public int indexTrack¼0;

public void initializeImg(int[][] outTotalMatrix){ img ¼new int[outTotalMatrix.length][]; for(int i¼ 0; i ooutTotalMatrix.length; iþ þ) { int[] aMatrix¼ outTotalMatrix[i]; int aLength ¼aMatrix.length; img[i] ¼new int[aLength]; System.arraycopy(aMatrix, 0, img[i], 0, aLength); } }

//keep a copy of the image public void copyImg(){ cImg ¼ new int[img.length][]; for(int i ¼0; i oimg.length; i þ þ) { int[] aMatrix ¼img[i]; int aLength ¼aMatrix.length; cImg[i] ¼ new int[aLength]; System.arraycopy(aMatrix, 0, cImg[i], 0, aLength); } } // Sum all values from an image public int sumImage(int[][] img) { int temp¼0; for (int i ¼0;i owidth; iþ þ){ for (int j ¼0; joheight; j þ þ){ temp¼ img[i][j]þ temp; } } return temp; } // Black a single square of size blocksize and return new sumImage public int blackPivot(int[][] img, int xPos, int yPos, int w, int h){ for (int i ¼xPos; i o xPos þw; i þ þ ){ for (int j¼ yPos; joyPos þh; jþ þ ){

29

30


img[i][j] ¼0; } } return sumImage(img); } // Get Direction to move public int generateDirection(){ Random random ¼new Random(); return random.nextInt(4–0) þ0; } // Loss function public double lossFunction(int current, int previous){ return 0.5nMath.pow((current-previous), 2); }

// Parametric Function public double calculateParametricFunction(int current, int previous){ double norm ¼(current - previous)n1.0/ (255.0nblocksizenblocksize); double sigmoid ¼norm/(Math.sqrt(1 þ Math.pow(norm, 2))); return sigmoid; } //jump to new position public void jump(int x_ini, int y_ini){ int nX¼ -1; int nY¼-1; while(true){ int dir ¼generateDirection(); switch(dir){ case 0: nX ¼x_ini - stepSize; nY¼y_ini; break; case 1: nX ¼x_ini; nY¼y_ini þ stepSize; break; case 2: nX ¼x_iniþ stepSize; nY¼y_ini; break; case 3: nX ¼x_ini; nY¼y_ini - stepSize; break; } if (nX 4 ¼ 0 && nX o(width-blocksize)) if (nY 4 ¼0 && nYo(height-blocksize)){ x0 ¼nX; y0 ¼ nY; break; } } }

public void readImg(String filename){ img ¼ new int[width][height]; BufferedReader in; try { in ¼new BufferedReader(new FileReader(filename)); for (int i ¼0;i owidth; iþ þ){ for (int j ¼0; jo height; jþ þ){ String line ¼in.readLine();


int temp¼Integer.parseInt(line); img[i][j] ¼temp;

// // // // // // //

// TEMPORAL if (temp4 230){ img[i][j] ¼255; } else { img[i][j] ¼0; } } } in.close(); } catch (IOException ex) { ex.printStackTrace(); } } //Main gradient descent public void run(boolean randomflag) { // TODO Auto-generated method stub trackPos ¼new int[2][5000]; int step ¼ 0; int currentSum ¼0; int previousSum ¼ 0; //load original image //out.println(gd.sumImage(gd.img)); //Copy Image copyImg(); //out.println(gd.sumImage(gd.cImg)); //Black out the square at position (0,0) if (randomflag){ Random randomX ¼ new Random(); Random randomY ¼new Random(); x0¼ randomX.nextInt(width - blocksize - 0) þ0; y0 ¼randomY.nextInt(height - blocksize - 0) þ0;

} blackPivot(cImg, x0, y0, blocksize, blocksize); previousSum ¼ sumImage(cImg); currentSum ¼sumImage(img); //out.println(gd.sumImage(gd.cImg)); //calculate loss function double error¼ lossFunction(currentSum, previousSum); out.println(“ERROR: “þerror); double fw ¼ calculateParametricFunction(currentSum, previousSum); //step¼(int)((gd.stepSizengd.learningRate)nfw); stepSize ¼stepSize þ(int)(learningRatenfw); out.println(“FW¼ “þfw þ “\nStepSize¼“þ stepSize); jump(x0, y0); out.println(x0 þ“ “þ y0); trackPos[0][indexTrack]¼x0; trackPos[1][indexTrack]¼y0; previousSum ¼ currentSum; while (stopCriteriao5){ copyImg();

31

32


//out.println(gd.sumImage(gd.cImg)); //Black out the square at position (0,0) blackPivot(cImg, x0, y0, blocksize, blocksize); currentSum ¼sumImage(cImg); //out.println(gd.sumImage(gd.cImg)); //calculate loss function error¼lossFunction(currentSum, previousSum); out.println(“ERROR: “þerror); if (error4 1E8) stopCriteria þ þ; else stopCriteria ¼ 0; fw ¼calculateParametricFunction(currentSum, previousSum); //step¼ (int)((gd.stepSizengd.learningRate)nfw); step¼(int)(learningRatenfw); if (stepo0) stepSize ¼stepSize þstep; else stepSize ¼stepSize þ0; //out.println(“FW¼“ þfw þ “\nStepSize¼“þ gd.stepSize); jump(x0, y0); out.println(x0 þ“ “ þy0); indexTrack ¼indexTrackþ 1; trackPos[0][indexTrack]¼ x0; trackPos[1][indexTrack]¼ y0; trackPos[0][indexTrack þ1] ¼1000; trackPos[1][indexTrack þ1] ¼1000; previousSum¼currentSum; if (stopCriteria 4 ¼ 4) break; } out.println(“DONE!!”); }

References [1] R.P. Feynman, A.R. Hibbs, Quantum mechanics and path integrals (International series in pure and applied physics), McGraw-Hill, New York (1965) 365 (xiv). [2] A. Bordes, L. Bottou, P. Gallinari, SGD-QN: Careful Quasi-Newton Stochastic Gradient Descent, Journal of Machine Learning Research 10 (2009) 1737–1754. [3] L. Bottou, Stochastic learning, Advanced Lectures on Machine Learning 3176 (2004) 146–168. [4] M.A. Vorontsov, Decoupled stochastic parallel gradient descent optimization for adaptive optics: integrated approach for wave-front sensor information fusion, J Opt Soc Am A Opt Image Sci Vis 19 (2) (2002) 356–368. [5] M.A. Vorontsov, G.W. Carhart, Adaptive wavefront control with asynchronous stochastic parallel gradient descent clusters, J Opt Soc Am A Opt Image Sci Vis 23 (10) (2006) 2613–2622. [6] M.A. Vorontsov, et al., Adaptive optics based on analog parallel stochastic optimization: analysis and experimental demonstration, J Opt Soc Am A Opt Image Sci Vis 17 (8) (2000) 1440–1453. [7] M.A. Vorontsov, G.W. Carhart, J.C. Ricklin, Adaptive phase-distortion correction based on parallel gradient-descent optimization, Opt Lett 22 (12) (1997) 907–909. [8] C.R. Fetsch, G.C. DeAngelis, D.E. Angelaki, Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons, Nat Rev Neurosci 14 (6) (2013) 429–442. [9] A.M. Green, D.E. Angelaki, Multisensory integration: resolving sensory ambiguities to build novel representations, Curr Opin Neurobiol 20 (3) (2010) 353–360. [10] N.P. Holmes, C. Spence, Multisensory integration: space, time and superadditivity, Curr Biol 15 (18) (2005) R762–R764. [11] J. Klemen, C.D. Chambers, Current perspectives and methods in studying neural mechanisms of multisensory interactions, Neurosci Biobehav Rev 36 (1) (2012) 111–133.

[12] S. Tyll, E. Budinger, T. Noesselt, Thalamic influences on multisensory integration, Commun Integr Biol 4 (4) (2011) 378–381. [13] A. Pellionisz, R. Llinas, Tensorial approach to the geometry of brain function: cerebellar coordination via a metric tensor, Neuroscience 5 (7) (1980) 1125–1138. [14] D. Alais, F.N. Newell, P. Mamassian, Multisensory processing in review: from physiology to behaviour, Seeing and perceiving 23 (1) (2010) 3–38. [15] T. Ohshiro, D.E. Angelaki, G.C. DeAngelis, A normalization model of multisensory integration, Nature neuroscience 14 (6) (2011) 775–782. [16] W.C. Hall, A. Moschovakis, The superior colliculus: new approaches for studying sensorimotor integration, CRC Press, Boca Raton (2004) 324 (Methods & new frontiers in neuroscience). [17] R.B. Masterton, Sensory integration (Handbook of behavioral neurobiology), Plenum Press, New York (1978) 579 (xvi). [18] B.E. Stein, et al., The Developmental Process of Acquiring Multisensory Integration Capabilities, in: L.M. Chalupa. (Ed.), Cerebral plasticity: new perspectives, MIT Press, Cambridge, Mass, 2011, p. 415 (12 p. of plates). [19] I.M. Schepers, et al., Functionally specific oscillatory activity correlates between visual and auditory cortex in the blind, Brain: a journal of neurology 135 (Pt 3) (2012) 922–934. [20] H. Lapid, et al., Neural activity at the human olfactory epithelium reflects olfactory perception, Nature neuroscience 14 (11) (2011) 1455–1461. [21] R.J. Nelson, The somatosensory system: deciphering the brain's own body image (Methods & new frontiers in neuroscience), Fla.: CRC Press, Boca Raton (2002) 399 (16 p. of plates). [22] S.E. Boehnke, D.P. Munoz, On the importance of the transient visual response in the superior colliculus, Current opinion in neurobiology 18 (6) (2008) 544–551. [23] S.U. Nummela, R.J. Krauzlis, Superior colliculus inactivation alters the weighted integration of visual stimuli, The Journal of neuroscience: the official journal of the Society for Neuroscience 31 (22) (2011) 8059–8066.


[24] A.B. Butler, W. Hodos, Comparative vertebrate neuroanatomy: evolution and adaptation, Wiley-Liss, New York (1996) 514 (xvii). [25] A.K. Churchland, R. Kiani, M.N. Shadlen, Decision-making with multiple alternatives, Nature neuroscience 11 (6) (2008) 693–702. [26] T. Isa, W.C. Hall, Exploring the superior colliculus in vitro, Journal of neurophysiology 102 (5) (2009) 2581–2593. [27] K. Kaneda, T.I., Superior Colliculus, in: G.M. Shepherd, S. Grillner (Eds.), Handbook of brain microcircuits, Oxford University Press, New York, 2010, p. 514 (xix). [28] M. Kinoshita, et al., Retinotectal transmission in the optic tectum of rainbow trout, The Journal of comparative neurology 484 (2) (2005) 249–259. [29] M. Kinoshita, et al., Periventricular efferent neurons in the optic tectum of rainbow trout, The Journal of comparative neurology 499 (4) (2006) 546–564. [30] R. Nieuwenhuys, H.J.t. Donkelaar, C. Nicholson, The central nervous system of vertebrates, Springer, Berlin; New York, 1998. [31] H. Vanegas, H. Ito, Morphological aspects of the teleostean visual system: a review, Brain research 287 (2) (1983) 117–137. [32] C.S. von Bartheld, D.L. Meyer, Comparative neurology of the optic tectum in ray-finned fishes: patterns of lamination formed by retinotectal projections, Brain research 420 (2) (1987) 277–288. [33] M.T. Wallace, M.A. Meredith, B.E. Stein, Multisensory integration in the superior colliculus of the alert cat, Journal of neurophysiology 80 (2) (1998) 1006–1010. [34] R.H. Wurtz, Superior Colliculus, in: L.R. Squire. (Ed.), Encyclopedia of Neuroscience, Vol. 9, Academic Press, Oxford, 2009. [35] H. Lalazar, E. Vaadia, Neural basis of sensorimotor learning: modifying internal models, Current opinion in neurobiology 18 (6) (2008) 573–581. [36] S. Grillner, et al., Intrinsic function of a neuronal network - a vertebrate central pattern generator, Brain research. Brain research reviews 26 (2–3) (1998) 184–197. [37] C.M. Harris, Biomimetics of human movement: functional or aesthetic? Bioinspiration & biomimetics 4 (3) (2009) 033001. [38] D.R. Humphrey, et al., Motor control: concepts and issues: report of the Dahlem Workshop on Motor Control: Concepts and Issues, Berlin, 1989, December 3-8 (Dahlem workshop reports), Wiley, Chichester; New York (1991) 500 (xiii). [39] A.J. Ijspeert, et al., From swimming to walking with a salamander robot driven by a spinal cord model, Science 315 (5817) (2007) 1416–1420. [40] J.L. Krichmar, H. Wagatsuma, Neuromorphic and brain-based robots, Cambridge University Press, Cambridge, UK (2011) 364 (ix). [41] A. Sproewitz, et al., Learning to Move in Modular Robots using Central Pattern Generators and Online Optimization, The International Journal of Robotics Research 27 (3–4) (2008) 423–443. [42] R. Abraham, C.D. Shaw, Dynamics–the geometry of behavior (The Visual mathematics library Vismath books), Calif.: Aerial Press, Santa Cruz, 1982. [43] S.i. Amari, H. Nagaoka, Methods of information geometry (Translations of mathematical monographs), American Mathematical Society, Providence, RI (2000) 206 (x). [44] S. Ganguli, D. Huh, H. Sompolinsky, Memory traces in dynamical systems, Proceedings of the National Academy of Sciences of the United States of America 105 (48) (2008) 18970–18975. [45] K. Kaneko, Relevance of Dynamic Clustering to Biological Networks, Physica D 75 (1–3) (1994) 55–73. [46] W. Pitts, W. McCulloch, How we know universals the perception of auditory and visual forms, Bulletin of Mathematical Biology 9 (3) (1947) 127–147. [47] C.M. Harris, On the optimal control of behaviour: a stochastic perspective, Journal of neuroscience methods 83 (1) (1998) 73–88. [48] T. Zhang, Solving large scale linear prediction problems using stochastic gradient descent algorithms, in Proceedings of the twenty-first international conference on Machine learning, ACM, Banff, Alberta, Canada (2004) 116.

33

[49] R.P. Feynman, R.B. Leighton, M.L. Sands, The Feynman lectures on physics, Pearson/Addison-Wesley, San Francisco, 2006 (Definitive ed). [50] A. Pellionisz, R. Llinas, Brain modeling by tensor network theory and computer simulation. The cerebellum: distributed processor for predictive coordination, Neuroscience 4 (3) (1979) 323–348. [51] A. Pellionisz, R. Llinas, Space-Time Representation in the Brain - the Cerebellum as a Predictive Space-Time Metric Tensor, Neuroscience 7 (12) (1982) 2949 (&). [52] A. Pellionisz, R. Llinas, Tensor network theory of the metaorganization of functional geometries in the central nervous system, Neuroscience 16 (2) (1985) 245–273. [53] Roy, S. and R. Llinas, Dynamic geometry, brain function modeling, and consciousness. 2007. 168: p. 133-144. [54] S. Roy, R. Llinas, Dynamic geometry, brain function modeling, and consciousness, Prog Brain Res 168 (2008) 133–144. [55] C.M. Bishop, Pattern recognition and machine learning (Information science and statistics), Springer, New York (2006) 738 (xx). [56] J. Bouvrie, J.J. Slotine, Synchronization and redundancy: implications for robustness of neural learning and decision making, Neural Comput 23 (11) (2011) 2915–2941. [57] UK, R.-e., SRF05 - Ultra-Sonic Ranger. [58] B. Girard, et al., Where neuroscience and dynamic system theory meet autonomous robotics: a contracting basal ganglia model for action selection, Neural networks: the official journal of the International Neural Network Society 21 (4) (2008) 628–641. [59] T.T. Takahashi, et al., The synthesis and use of the owl's auditory space map, Biological Cybernetics 89 (5) (2003) 378–387. Alex Porras Ph.D. received the B.S. degree in Electronic Engineering from Pontifical Xaverian University, Bogota, Colombia, in 2002 and his Ph.D. degree in Physiology and Neuroscience from New York University, School of Medicine in 2013. He has experience with mechanical, electrical and control systems projects for the industry as Development Engineer. His research interest includes somatosensory integration, mathematical modeling of biological systems, biomedical devices, and robotics.

R. Llinás M.D., Ph.D. is a Professor at the Department of Neuroscience and Physiology at the New York University (NYU) School of Medicine. His interest expands from ionic channels and single cell physiology to brain circuits and cognitive function. His work in robotics relates mainly to the use of analog hardware computation as heuristic tools to understand the nature of biological neuroscience.