Atomic machine learning

Нейром атем атика и интел лектуальны е вы числения UDC 519.67, 519.23:24, 517.5

Atomic machine learning © Authors, 2018 © Radiotekhnika, 2018

S.Yu. Eremenko − Dr.Sc. (Eng.), Professor, Director of Soliton Scientific Pty Ltd (Sydney, Australia)

E-mail: [email protected]

It is shown how the theory of Atomic functions known since 1970th and their AString generalizations can be expanded to Machine Learning algorithms to represent atomic kernels in Support Vector Machines, Density Estimation, Principal Component Analysis, LOESS regression, and be the foundation of atomic regression, atomic activation function for Neural Networks and atomic computer. Representing a probabilistic weighted uniform distribution, these functions are related to Prouhet–Thue–Morse ‘fair game’ sequence and offer a fast calculation of derivatives. Multiple Matlab, R and Python examples calculate complex regressions, decision boundaries and precise target clusters important for medical, military and engineering applications. In combination, these methods compose Atomic Machine Learning theory which extends the 50 years’ history of atomic functions and their generalisations to new scientific domains.

Keyw ords: atomic function, atomic string, machine learning, regression, support vector machines, neural networks, clustering, atomic computer, Python, MATLAB.

Показано как теория атомарных функций, развивающаяся с начала 1970-х гг., и их AString обобщений могут быть расширены на алгоритмы Машинного Обучения для представления атомарных ядер в алгоритмах опорных векторов, оценки плотности, анализа главных компонентов, локально взвешенной регрессии и быть основой атомарной регрессии, атомарной активирующей функции нейронных сетей и атомарного компьютера. Как вероятностные взвешенные равномерные распределения эти функции связаны с Prouhet–Thue–Morse последовательностями и принципами «справедливой игры», а также допускают быстрые вычисления производных. С помощью программ Matlab, R и Python приведены вычисления сложных регрессий, разграничительных границ и кластеров точных целей, важных для медицинских, военных и инженерных приложений. В сочетании эти методы составляют теорию Атомарного Машинного Обучения, которая расширяет 50-летнюю историю атомарных функций и их обобщений на новые научные области.

Клю чевы е слова: атомарная функция, атомарная струна, машинное обучение, регрессия, метод опорных векторов, нейронные сети, кластеры, атомарный компьютер, Python, MATLAB.

Atomic functions [1–9] (AF) described in multiple books and hundreds of papers have been discovered in 1970th by V.L. Rvachev * and V.A. Rvachev [2, 3] and further developed by many followers [10–16], notably by schools of V.F. Kravchenko [5–9], H. Gotovac [15], V.M. Kolodyazhni [17], for a wide range of applications in mathematical physics, boundary value problems, statistics, radio-electronics, telecommunications, signal processing and others. As per historical survey [7] available online, some elements, analogs, subsets or Fourier transformations of AFs sometimes named differently (Fabius function, hat function, compactly supported smooth function) have been known since 1930th and were rediscovered many times by other scientists from different countries, including Fabius [18], Hilberg [19] and others. But the most comprehensive 50+ years’ theory development supported by multiple books, hundreds of papers, lecture courses, multiple online resources in Appendix and many dissertations have been performed by the school of V.L. Rvachev. AFs, starting from the simplest function up ( x ) featured in http://ru.wikipedia.org/?oldid=82669103, have attracted attention of many scientists due to unique combination of properties like spline-like finite support, absolutely smooth pulse-like shape with derivatives expressed via AF itself, ability to represent a unity (number 1) and other polynomials via superposition of only a few nonlinear pulses, and this cannot be achieved so easily with other trigonometric, hyperbolic or other pulse-like functions. This makes AFs unique and useful for spline-like approximation techniques widely used in finite element methods [10–12] and in AF collocation methods developed by V.A. Rvachev [2, 3, 5], V.F. Kravchenko [5–9], O.V. Kravchenko [6,7], V.M. Kolodyazhni [17], and H. Gotovac [15]. For decades, the attention of AF researchers was mostly allocated to studying approximation and derivation properties and building many new classes of pulse-like AFs called Fups [2–9]. But interestingly, the integral of AF AString’ ( x ) = up ( x ) named AString in the paper [1] offer many useful properties *

Vladimir Logvinovich Rvachev (1926–2005), Academician of National Academy of Sciences of Ukraine, author of 600+ papers, 18 books, mentor of 70 PhDs and 20 Doctors of Sciences including the author. Нейрокомпьютеры: разработка, применение, № 3, 2018 г.

13

Нейром атем атика и интел лектуальны е вы числения

too. First of all, up ( x ) itself can be expressed via a simple combination of AStrings making it somewhat more generic. Secondly, AString looks like a ‘kink’ function used in many theories of solitons [19] and offers some ‘physical meaning’ of up ( x ) as a ‘solitonic atom’ [1,19] – from the translation of which it is possible to compose smooth and curved continua which may model spacetime and other nature continua studied in lattice physics, string theory, general relativity, spacetime physics, and astronomy as observed in [1]. Thirdly, unlike AFs the swing-like AString function can be used to build new non-Archimedean calculus [13,14] (so called atomic  calculus [1]) to describe nonlinear metric of space, offer new methods of modelling the nonlinear physical processes and building the new atomic computer [1] as will be discussed in this paper. For simplicity of description, we would use the common term ‘Atomics’ for both AFs and AStrings. Interestingly, both AFs and AStrings have definite probabilistic meaning deeply related to ProuhetThue-Morse sequence [20, 21] (https://en.wikipedia.org/wiki/Thue–Morse_sequence) and Fabius function [18] (https://en.wikipedia.org/wiki/Fabius_function), both related to independent inform distribution of random variables important to some fractal and statistical concepts of fair division and periodic sequencing useful for chess, soccer penalty, tennis tie-break scheduling and other combinatorial game theory applications [20]. Also, pulse and swing-like shapes of AFs and AStrings make them quite similar to Gaussians and sigmoids widely used in statistics and Machine Learning (ML) [23–29] – vast scientific domain with the number of publications exceeding a few million [25]. It inspired an investigation on the potential application of Atomics for ML, and, as will be shown hereafter, many if not all ML algorithms like linear, logistic and LOESS regression, Neural Networks, Support Vector Machines, Density Kernels Estimations, Principal Component Analysis, Dimensionality Reduction can effectively incorporate Atomics. The group of ML algorithms using Atomics would compose Atomic Machine Learning (AML) theory first introduced in this paper. Let’s note that another invention of V.L. Rvachev – R-functions well published in a few hundreds of papers and books observed in [3, 4, 9–12] allowing constructing exact analytical equations of complex geometrical objects – has already been used in pattern recognition and ML for a few decades, starting from pioneering pattern recognition and AI works of I.B. Sirodzha [16] to more recent researches on Rclouds [23], Rvachev basis functions [34], Rvachev-Kravchenko functions [6–9] and others. R-functions will be out of the scope of this publication in favour of ideas based on atomic functions, and AStrings described hereafter. Atomic Function Atomic functions (AF) [1–9] have been discovered in 1970th by V.L. Rvachev and V.A. Rvachev [2, 3] and further developed by followers, notably V.F. Kravchenko [5–9]. Like well-known splines, the AFs belong to the class of ‘finite functions’ equal to zero everywhere except a local segment of an x-axis (for example, −1 ≤ x ≤ 1 ) and satisfying a special type of linear functional-differential equations with constant coefficients and ‘shifted’ arguments [2–4] y

( n)

( x ) + a1 y ( n −1) ( x ) + … + an −1 y ′ ( x ) + = an y ( x )

M

a > . ∑b y ( ax − b ) , 1  k

k

(1)

k =1

The simplest and the most important is the function up ( x ) depicted in Fig.1, a and represented via Fourier series by the formula [2–5] 1 sin ( t 2− k ) up ( x ) = eitx ∏ ∞k =1 dt ,    up ( x ) dx = 1 −k 2π t 2 −∞ −1 ∞

∫

1

∫

(2)

allowing along with other techniques calculating the tabular and computer code [1–5] representation provided in Appendix. This finite function has a remarkable property – its derivative can be expressed via the function itself shifted and stretched by the factor of 2

( x ) 2up ( 2 x + 1) − 2up ( 2 x − 1)   for x ≤ 1, up′= ( x ) 0 for  x > 1, up′= 14

(3)

Нейрокомпьютеры: разработка, применение, № 3, 2018 г.

Нейром атем атика и интел лектуальны е вы числения

and this equation can be used as a formal definition of AF. In fact, this function has originally been obtained to answer the following question [3–5]: what should be the form of a generic finite pulse function that a derivative of it (consisting of two similar pulses) would be representable via the stretching and shifting of an original function? For such a function, it would be easy to calculate the derivatives through the function itself like for widely used function exp ( x ) or sigmoid. There are a few other properties of AF, for example, it is absolutely smooth, has finite support like a spline and non-analytical (cannot be exactly represented by Taylor's series). Another important property of AF is the ability to exactly represent the unity (number 1) by summing up of individual pulses set at regular points ... –3, –2, –1, 0, 1, 2, 3… as shown in Fig.1, b …up ( x − 2 ) + up ( x − 1) + up ( x ) + up ( x + 1) + up ( x + 2 ) + … ≡ 1.

а)

(4)

b)

Fig. 1. Atomic function with derivative and integral (AString) (a); representation of a constant by combining of atomic function pulses (b)

Like for continuous physical atomic structures consisting of individual atoms, it allows to exactly represent a constant (and linear and other polynomials) by combining only a few nonlinear wavy pulses, and due to this reason, it was named an 'atomic function’ [2, 4]. For 50 years’ history described in the survey [7] and detailed works [1–9], the AF and its variations have been intensively researched and applied in radio electronics, electrodynamics, superconductivity, signal processing, the theory of optimisation, boundary value problems and many others [1–9]. In this paper, we would explore how the AF theory can be extended to important data science domain of Machine Learning. Atomic String Atomic String (AString) introduced in [1] as a separate soliton kink-like function, is an integral from an atomic function (2), (3) which interestingly can be expressed via the up ( x ) section itself [1–4]: x

 x 1 1 AString ( x ) = up ( x ) dx = up  −  −  for  | x ≤ 1, AString ( x ) = ±0.5 for | x ≥ ±1. 2 2 2

∫

(5)

0

An integration constant can be chosen to have a swing height of 1 as shown in Fig. 2, a depicting an AString along with the atomic function up ( x )  as its derivative. This expression also highlights the important fact that the symmetric pulse up ( x ) can be represented by a combination of two opposite swings. The remarkable property of shifted and stretched AF to have the similar shape parts with both derivative Нейрокомпьютеры: разработка, применение, № 3, 2018 г.

15

Нейром атем атика и интел лектуальны е вы числения

and integral functions makes it distinctively unique, and AString inherits some of these properties too. For example, AF itself with all its derivatives can be expressed via AString : up ( x ) = AString′ ( x ) ,  up′ ( x ) = 2up ( 2 x + 1) − 2up ( 2 x − 1) = AString′′ ( x ) . 

a)

(6)

b)

Fig. 2. Atomic String and its derivative – atomic function (a); Atomic function can be exactly represented by a combination of two atomic strings (b)

Importantly, a derivative operation can be replaced by simple stretching and shifting of two AStrings, and Fig. 2,b shows the AF itself (along with all derivatives which are AF combinations) can be exactly represented by combining of two AStrings ′ ( x ) AString ( 2 x + 1) − AString ( 2 x − 1) .  = up ( x ) AString =

(7)

This equation can be considered as a formal definition of AString function the numerical values of which can be easily calculated using a script referenced in Appendix. By integrating the equation (4), it is easy to conclude that AStrings combination can also present a straight segment between joining points with period s as shown in Fig. 3: x ≡ … AString ( x − 2s ) + AString ( x − s ) + AString ( x ) + AString ( x + s ) + AString ( x + 2s ) + …

(8)

It makes it distinct from other kink-like functions like sigmoids, hyperbolic tangent or soliton kink functions which cannot possess [1] such a unique set of properties [1–9]. Basically, like for perfectly matching Lego game pieces, the AString edge swings are shaped in such a way that joining them one after another on a periodic lattice allows composing a perfectly straight line. And namely, this fact has been used in the paper [1] to propose a new AString model of uniform and curved spacetime with the introduction of an atomic quantum which may comprise the fabric of spacetime and other continua. Also, the shape of AString curve in Fig. 2, a close to a linear function in the middle and remote asymptotes allows using AStrings to construct an atomic calculus to describe metric of space within cosmic strings [1] and generally construct multiple non-Archimedean calculi explored in our previous papers [13, 14]. Elementary AString function (5)–(7) can be generalized in the form (9) AString ( x, , , ,  a b c d) = d + c * AString ( ( x − b ) a ) . representing a swing with width 2a , height c and pulse centre positions b, d . Let’s note that Fabius function [19] can be expressed as a shifted AString function (9) with the following parameters and the calculation script referenced in Appendix: (10) Fabius ( = x ) AString ( x= ,  a 0.5,  = b 0.5, 1,  = c = d 0.5 ) .

16

Нейрокомпьютеры: разработка, применение, № 3, 2018 г.

Нейром атем атика и интел лектуальны е вы числения

а)

b)

Fig. 3. Representation of a straight line segment by joining of atomic strings

Unique properties of AStrings – ability to compose an atomic function, represent flat and curved surfaces and have all derivatives expressed through the AStrings themselves – make them quite promising for a wide variety of applications in lattice physics, relativity, cosmology, computing [1], and machine learning where similar swing-like functions are widely used. Commonly used Machine Learning functions and their atomic representations The idea of Atomic Machine Learning theory described here is to explore how atomic functions widely used in other areas can be incorporated into ML algorithms, and substitute/modify/approximate some statistical functions. Using similarity of shapes and numerical experiments shown in Fig.4, it is quite easy to find atomic analogues for the most commonly used ML functions.

a)

b)

c)

Fig. 4. Sigmoid and its AString representation (a); gaussian and its atomic function representation (b); three-cube and its atomic function representation (c)

For example, swing-like Sigmoid function used in logistic regression, cumulative distribution in statistics, neural networks and other ML algorithms [23–29] can be represented by a similarly shaped atomic sigmoid function (Fig.4, a) expressed via AString (9) with the following parameters: 1 (11) = Sig ( x ) = ,  ASig ( x ) AString ( x, 4, 0,1 , 0.5 ) . 1 + exp ( − x ) Gaussian, or radial basis function, used in Support Vector Machines and Density Estimation ML algorithms [24–26] is a pulse-like function which can be well represented by an atomic Gaussian (AGaussian) function (Fig.4, b) based on AF (2)–(4)  − x2  = = Gauss ( x, σ ) exp  2  ,   AGauss ( x, σ ) up ( x / 3σ ) .  2σ  Нейрокомпьютеры: разработка, применение, № 3, 2018 г.

(12)

17

Нейром атем атика и интел лектуальны е вы числения

Three-cube function used in locally weighted LOESS regression [25, 27] is a pulse-like function which can be represented by a closely shaped atomic function (Fig.4, c):

(1 − x ) ,   ACube3 ( x ) = up ( x ) . Cube3 ( x ) = 3 3

(13)

Calculation scripts for all these functions are provided in Appendix. Let’s note that based on recent publication [6] it is possible to use more elaborated presentations for Atomic Gaussian, for example, Gauss ( x, , which may be explored in future publications. σ ) up ( x b )  Atomic Logistic Regression Logistic regression, one of the simplest and commonly used supervised learning algorithm [24, 26–28], is used to calculate linear and nonlinear decision boundaries (separation lines) between known ‘yes/no’ (‘positive/negative’, ‘good/bad’, ‘winning/losing’) binary outcomes of statistical experiments involving training set defined by m features over group of n   measurements

{0 / 1} y1} { x12 …, xm2 ,  y 2 }…{ x1n ,…, xmn ,  y n } ;  yi = {x11 ,…, x1m , , ,

(14)

as shown in Fig.5, a for = m 2, 70 n ≈ . Allocating (unknown) weights w j for features x j leads to the net input variable = z w0 + w1 x1 + … + wm x= wт x  m

(15)

and atomic hypothesis function ( z ) ASig ( w т x ) = h ( z ) ASig =

(16)

which for Atomic Linear Regression (ALR) should include atomic sigmoid (11) instead of traditionally used sigmoid (11). Using function (16) which as per Fig. 4, a is scaled from 0 to 1, it is possible to convert analogue z values (15) into binary 0/1 outcomes according to typical rule [24, 28, 29]:  1, if   h ( z ) ≥ 0.5,  1, if  z ≥ 0, (17) = y =  or  y  0, if   z < 0.  0,  if  h ( z ) < 0.5,  For example, ‘Yes’ outcome is highlighted as a ‘cross’ while ‘No’ as a ‘circle’ in Fig. 5. The unknown weight coefficients w can be found (or learnt in ML terminology [24, 28, 29]) by minimizing sumsquared-error cost function [24, 28, 29] = J (w)

n

2 1 ( i) ( h z −= yi ) 2 1 =i

∑

=i

n

∑ 2 ( ASig ( z ) − y ) 1

i

i

2

,

(18)

1

and this can be algorithmically achieved by numerical gradient descent method [24, 28, 29] which tries to find the values w j using an iteration algorithm involving the gradients ∂J = ∂w j

∂ASig ( z ) . ∑ ( ASig ( zi ) − yi ) n

i =1

i

∂w j

(19)

Let’s note that due to (7) the derivatives from ASig can be conveniently expressed via ASig itself. Modern software tools like Python [24], R [25] or MATLAB [30] offer well-tested libraries for these methods with the ability to replace sigmoids with other functions. Practical implementation of this algorithm programmed in MATLAB for a sample dataset described in Appendix leads to the results shown in Fig. 5, a where linear decision boundary with calculated coefficients w j in (15) splits some ‘winning’ and ‘loosing’ game scores measurements. Using atomic sigmoid hypothesis function (16), trained ALR model allows quick calculation of probabilistic outcomes for future measurements where, for example, h = 0.7 would mean 70% probability of scoring. The model is trained once, but can be reused in future, as per main ML paradigm. 18

Нейрокомпьютеры: разработка, применение, № 3, 2018 г.

Нейром атем атика и интел лектуальны е вы числения

а)

b)

Fig. 5. Linear Decision Boundary calculated by Atomic Logistics Regression separates ‘good’ and ‘bad’ outcomes (a); the similarity of results between Atomic and Standard Logistic Regression (b)

Let’s note that despite the close similarity (see Fig. 4,a) between atomic and logistical sigmoids (11), the probability meaning is a bit different. In logistic regression, it is related to odds ratio for random events [24, 28, 29] while in atomic statistics it is related to ‘fair game’ and Prouhet-Thue-Morse sequence concepts [20, 21] when by some fair rules the opponents are given equal scoring chances which is important for chess, soccer penalty, tennis tie-break scheduling and other combinatorial game theory applications [20]. However, in practical calculations, these nuances may not be that important, and the results and time of calculations for atomic and standard sigmoid-based logistic regression are practically identical to each other (Fig. 5,b). In summary, Atomics can be easily incorporated into logistic regression ML algorithms via AString function similarly shaped with the sigmoid. Regularized Atomic Logistic Regression The main ALR algorithm (14)–(19) described previously can be modified to include regularisation parameter λ   [24, 28, 29] into more elaborated quadratic cost function (18) = J (w)

2 1 ( i) ( h z − yi ) + λ w2j . 2 2m 1 =j 1

n

∑

=i

m

∑

(20)

It allows penalising extreme parameter weights and building more complex nonlinear decision boundaries separating ‘positive’ and ‘negative’ binary outcomes of statistical experiments as shown in Fig.6. In this case, the gradient function (19) would also be modified: ∂J ∂ASig ( z i ) λ = ( ASig ( z i ) − y i ) + wj. ∂w j i =1 ∂w j m n

∑

(21)

But apart from that, the gradient descent ML algorithm [24, 29] stays the same allowing calculating some complex decision boundaries shown in Fig.6 when the sequence of iterations at different λ converges. The results are also quite similar to standard regularized sigmoid regression (Fig. 6,c). As we can see from (20), (21), the AString function is not directly included into regularization factor. Нейрокомпьютеры: разработка, применение, № 3, 2018 г.

19

Нейром атем атика и интел лектуальны е вы числения

a)

b)

c)

Fig. 6. Complex decision boundaries calculated by Regularized Atomic Regression at λ = 0.75  (a) and  2.0  (b); the same λ= set calculated via sigmoid logistics regression at λ = 1 (c)

In summary, Atomic Logistic Regression allows constructing simple and complex decision boundaries for fast prediction of binary outcomes of future observations based on trained atomic machine learning model, with many potential areas of application in medicine, technology and especially artificial intelligence where deep learning of complex features is in great importance. Atomic Support Vector Machines Support Vector Machines (SVM) is one of the most powerful ML classification algorithm [24, 26, 29] applicable for nonlinear decision boundaries separating complex data clusters (Fig. 7, 8) where logistic regression models simply do not work. The core feature of SVM algorithms described in [24, 26, 29] is to build hyperplanes defined by support vectors which separate data clusters with a maximum margin. Modern software tools like R [25], Python [24] or MATLAB [30] include well-tested libraries allowing effective implementation of SVM algorithms with relatively small amounts of code [29]. The core feature of SVM algorithms is the kernel function [23, 24, 26, 28, 29] which estimates the proximity between two data sets x i , x j , typically using the Gaussian Kernel i ) r2 K ( x= σ= , x j ) Gauss ( r , , 

n

∑( x

i k

− xkj

)

2

(22)

k =1

with the Gaussian function (12). For Atomic SVM (ASVM) implementation, it can be replaced by similarly shaped atomic function (12) (Fig. 4,b) bringing the atomic kernel to the form  n  2 ( r , 3 = = K ( x i , x j ) AGauss σ ) up  xki − xkj σ .    k =1 

∑(

)

(23)

The results of calculations presented in Fig.7 for test dataset in Appendix shows how results of calculations are converging at different values of kernel width σ with an obvious pattern of more precise decision boundary for more narrow kernels. As per Fig. 7,c, d, the atomic and Gaussian SVM results are close to each other.

a)

b)

c)

d)

Fig. 7. Convergence of SVM results with Atomic Kernel at σ = 0.07, 0.05, 0.03 (a, b, c); SVM results with Gaussian Kernel at σ = 0.03 (d) 20

Нейрокомпьютеры: разработка, применение, № 3, 2018 г.

Нейром атем атика и интел лектуальны е вы числения

а)

b)

c)

d)

Fig. 8. Underfitting and overfitting of ASVM results at different widths of kernel σ = 1, 0.3,  0.07,  0.03 (a, b, c, d)

Quite interesting to observe in Fig. 8 how ‘underfit’ and ‘overfit’ may be the SVM model for different atomic kernel width. Let’s note that in automated SVM algorithms [23, 24, 29] it is possible to construct an iteration process to find optimal values of σ at which the mean variance between some predicted and trained values would be small enough. But visualization of results like in Fig. 8, 9 is often beneficial. Obviously, once the model is trained, it is possible to reuse it for future predictions. In summary, atomic functions can be quite effectively implemented for robust SVM machine learning algorithms on the level of atomic kernels. Precise Target Clustering with ASVM Interestingly, SVM algorithms offer useful cases of precise target cluster definitions practically important for some military, medical, biological, engineering, navigation, and astronomical applications. Extreme overfitting with very narrow kernels (small σ ) usually bad for classical ML problems [23, 29] may be quite useful to identify the spots/clusters of ‘binary targets’ (‘good/bad’, ‘friend/enemy’, ‘positive/negative’) like shown in Fig. 9 for the atomic kernel (23).

a)

b)

Fig. 9. Precise target clusters definition with ASVM and atomic kernel at σ = 0.005  (a) and σ = 0.01 (b)

Instead of identifying some ‘individual dots’, the SVM algorithm allows uniting targets into small connected clusters upon which the specific ‘actions’ can be performed. Obvious are military applications where targeted strike into the centre of a cluster may be used to destroy targets in short session. In astronomical applications, the observations may allocate the similar stars to one cluster/galaxy/constellation. The circles in Fig.9 may be the clusters of aeroplanes approaching an airport where warnings can be communicated to an individual aircraft dangerously approaching the neighbours. Ocean pollution spots can be calculated and digitised based on aerial observations. In geology, the clusters may represent islands of precious metals. In microbiology, the clusters of ‘bad bacteria’ can be identified and treated acНейрокомпьютеры: разработка, применение, № 3, 2018 г.

21

Нейром атем атика и интел лектуальны е вы числения

cordingly. In medicine, the clusters may identify healthy cells or activity zones in brains; the presentation on AML in healthcare and medicine provided in Appendix. In summary, the ASVM and generally SVM algorithms can not only produce ML models for fast future predictions but also to identify the ‘good’ or ‘bad’ real-time target clusters and hopefully act upon this knowledge. Atomic Locally Weighted Regression Linear Regression [24–30] is one of the simplest and widely used ML techniques to fit a straight line to scatterplots like in Fig. 10, a. The standard procedure is to fit a linear function y =w0 + w1 x =wT x  

(24)

to the dataset { x1 ,  y1} , { x2 ,  y2 } , , yn } to minimize sum-squared-error cost function [24, 28] … { xn ,  J (= w)

n

∑ 2 (w 1

0

+ w1 xi − yi ) . 2

(25)

i =1

It can be achieved by solving simple matrix equations, with many implementations readily available in Excel, R, Python or MATLAB [24, 25, 30]. The approximation error would be normally distributed [24, 28, 29] with Gaussian function (12), and to introduce Atomics into this technique we can incorporate the atomic Gaussian (12) instead which, as we discussed, has different ‘fair game’ bias [20, 21]. However, all linear regression techniques suffer the same low accuracy problem when fit for the nonlinear datasets which typically appear in reality (Fig.10). The more advanced techniques would be preferable, and one of the most robust is locally weighted scatterplot smoothing (LOESS) algorithm described in [27, 25], online https://en.wikipedia.org/wiki/Local_regression and effectively implemented in data science package R [25]. The main idea of the algorithm is to split the research interval into sections/windows (like 10 points) and apply the least-square linear or polynomial regression (24), (25) to the moving windows and join results afterwards. In a small neighbourhood of window points the more weight is given to the points close to a current point, and three-cube weight function (13) from Fig.4, c is typically used for this purpose [27, 25]. Technically, this function introduced in 1979 [27] acts as a kernel (23), (24), and to introduce Atomics in LOESS algorithm, it is possible to replace weight function (13) with atomic function (13) with a similar profile as per Fig. 4,c. The advantages of AF up ( x ) is the ability to provide absolutely smooth connections between ‘moving windows’, exactly represent a unity (Fig. 1,b) and potentially straight-line sections via the closely related AString function (6) like shown in Fig. 3. The implementation of these ideas is the subject of future research requiring significant programming efforts to modify LOESS package in R [25].

a)

b)

Fig. 10. Locally weighted LOESS regression allows fitting a straight line and polynomial splines to data (a); LOESS regression accurately identifies trend data component also allowing decomposing the seasonal and random parts (b)

22

Нейрокомпьютеры: разработка, применение, № 3, 2018 г.

Нейром атем атика и интел лектуальны е вы числения

Local regression LOESS techniques are important not only for the adequate approximation of scatterplots in Fig. 10,a, but also for data series decomposition techniques allowing splitting the data series into Seasonal, Random and Trend components (Fig. 10,b) where Trend part is extracted using LOESS regression in commonly used R STL package [25]. In summary, local regression techniques offer plenty of opportunities for introduction of Atomics as weight functions which have different probabilistic meaning and offer infinite smoothness, derivatives expressed via themselves, ability to represent a constant and basic polynomial, and fast calculations via tabular representations of AFs provided in Appendix. Atomic Neural Networks Neural Networks (NN) Machine Learning algorithms are based on the idea of replication the functionality of a human brain consisting of a huge network of neurons which pass the information from multiple input channels into one output channel which in turn is an input to another network node as shown in Fig. 11. Interestingly, behaviour of each node can be simulated by Logistical Regression model described previously by equations (14)–(19) where input parameters xi supplemented by unknown weights wi are composed into an output factor = z w0 + w1 x1 + … + wm x= w т x.  m

(26)

A node passes the information to another node only if an output signal calculated by an activation function exceeds certain threshold:  1, if   h ( z ) ≥ 0.5,  1, if  z ≥ 0, T ) ( z ) ASig ( w= = h ( z ) ASig = y  = y  x ;   or  0,  if  z < 0.  0, if  h ( z ) < 0.5, 

a)

b)

(27)

c)

Fig. 11. Neural Network model (a); one node of a Neural Network (b); AString as NN Activation Function (c) Sources: https:// c.mql5.com/18/20/NN1__1.gif; https://www.embedded-vision.com/sites/default/files/technical-articles/CadenceCNN/Figure3b.jpg

Typically, this activation function is represented either by sigmoid (11) or similar hyperbolic tangent function [24, 28, 29]. However, for Atomic Neural Network (ANN) implementation it is possible to use AString function (11) which, as has been discussed, produces similar results with sigmoid (Fig. 5, 6). So, complex ANN network would consist of many layers of connected nodes with unknown weight functions wi which can be found by existing NN propagation algorithms implemented in multiple packages including Python [24], MATLAB [30] or R [25]. Practical implementation of ANN is a subject of future research, but presumably some unique AString properties – ability to compose straight line (neural) channels (Fig. 3) and ‘fair game’ probabilistic bias – may yield some benefits, still to explore. In summary, Atomics can be integrated into ML NN algorithms via atomic activation function in the form of AString. Other Machine Learning algorithms suitable for Atomics Booming ML science offers many other important applications and calculation technologies where atomic function and string can be used. They are briefly overviewed here. Нейрокомпьютеры: разработка, применение, № 3, 2018 г.

23

Нейром атем атика и интел лектуальны е вы числения

1. Density Estimation [23, 28, 29] (https://en.wikipedia.org/wiki/Density_estimation) is a statistical technique used in signal processing, data smoothing, image recognition to extract a probability density function for a complex physical process. For example, for LIGO experiment detecting gravitational waves from the collision of black holes to confirm Einstein general relativity theory, the registered signal provided in [31] has come quite noisy, and practically important was to remove the ‘noise’ and extract the meaningful signal. Traditionally, a normal distribution hypothesis with Gaussian function (12) for noise or other random study variable is assumed. However, as has been discussed it may not be the case if the initial process is not totally random but contains some hidden relations which would alter the normality hypothesis in favour of atomic ‘fair sequence’ distribution. Both Gaussian and Atomic Gaussian kernels (12) are close to each other (Fig. 4,b), and in each particular case, it seems interesting to test both hypothesis and kernels. 2. Principal Component Analysis (PCA) is another unsupervised ML theory allowing to remove some unnecessary features/dimensions from complex highly correlated processes to extract truly independent variables. The theory also uses a multivariate Gaussian distribution which may be replaced by close atomic Gaussian function (12). 3. Anomaly detection (AD) technique [24, 28, 29] to find some outlier’s measurements also uses Gaussian (12) while for ‘fairly related’ variables the AGaussian may also be applicable. 4. Edge Detection techniques used in Computer Vision and Image Recognition also intensively use kernels, and atomic kernel (12), (23) used in target recognition Fig. 9 can be one of them. 5. Spatial Regression and Variograms (https://en.wikipedia.org/wiki/Variogram) studying how much nearby points in geology and mining are correlated also use some swing and pulse functions looking quite similar to AStrings and AFs. 6. Gaussian-based ‘White noise’ may be supplemented with ‘Atomic noise’ with AF distribution if there are some doubts about totally random nature of processes generating the noise. 7. It is quite tempting to note that many physical theories including quantum mechanics also incorporate Gaussian and Sigmoid functions which may be altered to include Atomics which offer not only statistical but also deep ‘quantum’ meaning being infinitely smooth finite-based functions able to represent a quantum as discussed in [1]. A brief overview of ML theories provided here shows a wide range of potential applications of Atomic Functions and AStrings to supplement existing ML algorithms. Atomic Computer The idea of an atomic computer (AComputer) based on atomic calculus and AString function has been first proposed in paper [1]. There are two models. Non-Archimedean AComputer inside which all basic arithmetic operations (+, –, *, /) could be replaced by more complex non-Archimedean operations [13, 14] X= ⊕ Y ν (τ ( X ) + τ (Y ) ) ,  X = ! Y ν (τ ( X ) − τ (Y ) ) ,  

(28)

( X )τ (Y ) ) , , = X ⊗ Y ν (τ= X Y ν (τ ( X ) τ (Y ) )

where ν ( x ) is a generating axiom of a non-Archimedean calculus [1, 13, 14]

(

)

, (29) = ν ( x ) 2M * AString = X ⊕Y M * AString AString −1 ( X 2M ) + AString −1 (Y 2M )   ( x M ) , 2

(τ ( x ) ) τ= (ν ( x ) ) x . A combined non-Archimedean calculi theory developed τ ( x ) – inverse function ν= in our previous works [13, 14] and used by followers [32, 33] is based on the idea of replacing Archimedean axiom “responsible” for introducing operation x    + y in classical mathematics with some new operations, for example, those never producing an unphysical infinity like (29). Description of nonArchimedean AComputer is out of the scope of this publication. However, there is a simplified version of AComputer similar to ‘normal’ computer with operations x + y, x − y, xy, x y where the AString function with known tabular representation in Appendix can be implemented on ‘hardware’ or ‘processor’ level for super-fast calculation. In this case, the atomic function (7) can be easily calculated via AString, and all atomic sigmoids, atomic kernels, neural networks activation functions (11)–(13), (16), (22), (27) would 24

Нейрокомпьютеры: разработка, применение, № 3, 2018 г.

Нейром атем атика и интел лектуальны е вы числения

allow super-fast calculations too. This would make all AML algorithms described in this paper very computationally efficient which is especially important for current and future complex Neural Networks, Support Vector Machines, Deep Learning, and Artificial Intelligence models. Let’s remind the unique properties (6), (7) of AStrings and AFs to have their all derivatives expressed via themselves. This can also be very beneficial for boundary value problems algorithms and finite element methods widely used in mathematical physics [3–5, 10–12]. Hardware implementation of AStrings and AFs is still the subject of future theoretical and especially technological research we invite the collaborators into. It may lead to creating the new generation of superfast computers specially designed for advanced current and future ML and AI applications. Summary and future research directions This paper evaluates how atomic functions theory [1–9] with 50 years’ history of applications in mathematics, boundary value problems, geometry, statistics, radio-engineering, signal processing and recent applications [1] in theory of solitons, lattice physics and matter quantization, can be further extended to booming data science domain of machine learning (ML) and artificial intelligence (AI). AString – a swing-like integral from an atomic function (AF) – can be used in atomic logistic regression, atomic activation function of Neural Networks and building the atomic computer. AFs – which can be mutually expressed via AStrings – can be the foundation of atomic Support Vector Machine (ASVM) algorithms, clustering and precise target techniques, atomic LOESS regression, Density Estimation, Anomaly, Edge Detection and other algorithms based on atomic kernels which can supplement another ‘traditional’ kernels like Gaussian. AFs and AStrings have the different probabilistic meaning of ‘fair game’ and Prouhet-Thue-Morse sequence related to the independent inform distribution of random variables important in some fractal and statistical concepts of fair periodic sequencing [20, 21]. However, due to similar shapes with Gaussian and Sigmoids, the AF and AString provide close results with standard ML implementations as demonstrated in practical calculations described earlier. As we can see, many if not all ML and regression algorithms can incorporate AFs and AStrings, and atomic computer where AString can be implemented on a superfast processor level could be important technological innovation to implement AML as a complex of algorithms based on Atomics only. It could also supplement/enhance existing ML algorithms especially those where ‘fair game’ concepts would have more probabilistic relevance to statistical study data. The following research directions would be interesting to explore further. 1. Practical implementation of ALOESS algorithms with Atomics could lead to new local regression algorithms, hopefully having benefits of exact representation of constants and polynomials (see Fig.1–3). 2. Atomic Neural Networks based on AStrings still needs to be implemented and tested. 3. New examples of ‘Fair game’ variable distributions would be interesting to test with AML. 4. Deep Learning algorithms totally based on Atomics would be interesting to research further. 5. Practical implementation of atomic functions and AStrings in commonly used Excel, Python, R and MATLAB libraries may encourage the usage of AML by some practitioners and scientists. 6. Further testing of advantages of AML for very complex and long-running ML problems would be interesting to explore further. 7. Creating of Atomic Computer with the super fast calculation of Atomics may be important technological innovation. 8. In paper [6], V.F. Kravchenko and co-authors have proposed some alternatives like Gauss ( x, , σ ) up ( x b ) for some statistical functions, and further investigation could be an interesting exercise. AML theory with real calculations overviewed here may significantly expand the 50 years’ history of atomic functions and closely related AStrings to new popular scientific domains. Acknowledgements This paper is dedicated to the memory of the author’s teachers – Academician Vladimir Logvinovich Rvachev, the founder of Atomic and R-Functions, and Igor Borisovich Sirodzha, one of the first followers who started using R-functions for complex pattern recognition in 1980th in early days of ML and AI. Нейрокомпьютеры: разработка, применение, № 3, 2018 г.

25

Нейром атем атика и интел лектуальны е вы числения

The author also thanks Oleg Victorovich and Victor Filippovich Kravchenko for providing some support materials, comments and recent papers on the theory of atomic functions. Appendix and Web Resources The Python script representing an Atomic function, AString, AGaussian and ASigmoid is presented in https://solitonscientific.github.io/AtomicMachineLearning/AtomicMachineLearning.html and downloadable from GitHub project https://github.com/SolitonScientific/AtomicMachineLearning. Test datasets from Machine Learning course https://www.coursera.org/learn/machine-learning to produce calculations in Fig. 5–9 is downloadable from https://github.com/worldveil/coursera-ml and https://github.com/vugsus/coursera-machine-learning. Atomic Machine Learning and Atomic Strings ResearchGate projects: https://www.researchgate.net/project/Atomic-Machine-Learning-and-Artificial-Intelligence-withAtomic-Soliton-and-R-functions https://www.researchgate.net/project/Atomic-Computer-new-generation-superfast-computing https://www.researchgate.net/project/Atomic-Strings-Quantum-of-Spacetime-and-Gravitation https://www.researchgate.net/project/Atomic-String-and-Atomic-Function-New-Soliton-Candidates Machine Learning for Medical and Healthcare Applications presentation https://www.researchgate.net/publication/322499525, DOI 10.13140/RG.2.2.32213.12002. Other Atomic Functions Web Resources: Atomic functions on Wikipedia http://ru.wikipedia.org/?oldid=82669103 Atomic functions Web www.atomic-functions.ru Atomic functions founders and contributors: V.L. Rvachev https://ru.wikipedia.org/w/index.php?oldid=83948367 V.A. Rvachev http://m.mathnet.ru/php/person.phtml?personid=23932 V.F. Kravchenko https://ru.wikipedia.org/w/index.php?oldid=84521570 O.V. Kravchenko https://www.researchgate.net/profile/Oleg_Kravchenko H. Gotovac https://www.researchgate.net/profile/Hrvoje_Gotovac S.Yu. Eremenko https://www.researchgate.net/profile/Sergei_Eremenko References 1. 2. 3. 4. 5. 6.

7. 8. 9. 10. 11. 12. 13. 14.

26

Eremenko S.Yu. Atomic Strings and Fabric of Spacetime. Foreign Radioelectronics: Achievements of modern radioelectronics. 2018, Accepted for publication. https://www.researchgate.net/publication/320457739. Rvachev V.L., Rvachev V.A. About one Finite Function, DAN URSR. A (6). 705–707. 1971. Rvachev V.L., Rvachev V.A. Non-classical methods in the theory of approximations in boundary value problems. Kiev: Naukova Dumka. 1982. 196 p. Rvachev V.L. Theory of R-functions and their applications. Kiev: Naukova Dumka. 1982. Kravchenko V.F., Rvachev V.A. Application of atomic functions for solutions of boundary value problems in mathematical physics. Foreign Radioelectronics // Achievements of modern radioelectronics. 1996. № 8. P. 6–22. Kravchenko V.F., Kravchenko O.V., Pustovoit V.I, Churikov D.I. Atomic, WA-Systems, and R-Functions Applied in Modern Radio Physics Problems: Part 1. Journal of Communications Technology and Electronics. 2014, V. 59. №10. P. 981– 1009. https://www.researchgate.net/publication/298406834. DOI: 10.1134/S1064226914090046. Kravchenko V.F., Kravchenko O.V., Pustovoit V.I, Pavlikov V.V. Atomic Functions Theory: 45 Years Behind. https://www.researchgate.net/publication/308749839. DOI 10.1109/MSMW.2016.7538216. Kravchenko V.F. Lectures on the theory of atomic functions and their applications. Moscow: Radiotechnika. 2003. Kravchenko V.F., Rvachev V.L. Logic Algebra, atomic functions and wavelets in physical applications. Moscow: Fizmatlit. 2009. Eremenko S.Yu. Finite Element Methods in Mechanics of Deformable Bodies. Kharkov: Osnova. 1991. 272p. https://www.researchgate.net/publication/321171685. Eremenko S.Yu. Natural Vibrations and Dynamics of Composite Materials and Constructions. Kiev: Naukova Dumka. 1992. 182 p. Eremenko S.Yu., Rvachev V.L. A survey of variants of the structural finite-element method // Journal of Mathematical Sciences. 1998. V. 90. № 2. P. 1917–1922. Eremenko S.Yu. Combined Non-Archimedean Calculi. Foreign Radioelectronics // Achievements of modern radioelectronics. 1996. № 8. P. 57–65. https://www.researchgate.net/publication/320616255. Eremenko S.Yu., Kravchenko V.F., Rvachev V.L. Combined Non-Archimedean Calculi and New Models of Relativistic Mechanics. Foreign Radioelectronics // Achievements of modern radioelectronics. 1997. № 9. P. 26–38. https://www.researchgate.net/publication/320616269. Нейрокомпьютеры: разработка, применение, № 3, 2018 г.

Нейром атем атика и интел лектуальны е вы числения

15. Gotovac H., Kozulic V, Gotovac B. Space-Time Adaptive Fup Multi-Resolution Approach for Boundary-Initial Value Problems // Computers, Materials and Continua. 2010. 15(3). P. 173–198. https://www.researchgate.net/publication/287948119 16. Sirodzha I.B. Quantum model and methods of artificial intelligence for solution taking and control. Kiev: Naukova Dumka. 2002. 490 p. 17. Kolodyazhni V.M., Rvachev V.A. Atomic functions. Generalizations to the case of several variables and perspective directions of practical application // Cybernetics and Systems Analysis. 2007. V. 43. № 6. P. 155–177. 18. Fabius J. A probabilistic example of a nowhere analytic C ∞-function. Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete, 1966, 5 (2): 173–174, doi:10.1007/bf00536652, MR 0197656. 19. Hilberg W. Impulse und Impulsfolgen, die durch Integration oder Differentiation in einem veranderten Zeitmastab reproduziert warden. AEU. (25). 1. 1971. P. 39–48. 20. Palacios-Huerta I. Tournaments, fairness and the Prouhet–Thue–Morse sequence. Economic inquiry, 2012. 50 (3): 848–849. DOI:10.1111/j.1465-7295.2011.00435.x. 21. Kravchenko V.F., Kravchenko O.V., Konovalov Ya.Yu. Prouhet-Thue-Morse sequence and atomic functions in applications

22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34.

of physics and techniques. Journal of Measurement Science and Instrumentation, 2015, 6(2)： 128–141. https://www.researchgate.net/publication/280316794. DOI: 10.3969/j.issn.16748042.2015.02.005. Filippov A.T. The Versatile Soliton. Modern Birkhauser Classics. 2000. 261 p. Bougaev A.A., Urmanov A.M., Gross A.M., Tsoukalas L.H. Method of Key Vectors Extraction Using R-Cloud Classifiers. Springer, 2006. DOI 10.1007/1-4020-5263-4_15. Raschka S. Python in Machine Learning. PACKT Publishing, 2016. R-documentation on https://www.r-project.org. Cortes C., Vapnik V. Support-vector networks. Machine Learning. 1995, 20 (3), p.273–297. doi:10.1007/BF00994018. Cleveland W.S. LOWESS: A program for smoothing scatterplots by robust locally weighted regression. The American Statistician. 1981, 35 (1): 54. doi:10.2307/2683591. JSTOR 2683591. Data Science and Big Data Analytics: Discovering, Analyzing, Visualizing and Presenting Data. Wiley: EMC Educational Services, 2015. 432 p. Machine Learning Course, Stanford University, https://www.coursera.org/learn/machine-learning. MATLAB documentation on www.mathworks.com. LIGO Gravitational Wave detector experiment on https://losc.ligo.org/events/GW150914/, https://losc.ligo.org/s/events/GW150914/GW150914_tutorial.html. Scheglov E.G., Arutynov P.A. Non-Archimedean Calculus in the Mathematical Modelling of Digital ICs // Russian Microelectronics. 2002. V. 31. № 6. P. 384–395. Arutynov P.A. Non-Archimedean Calculi in Theoretical Metrology. Izm. Tekh. (Metrologia). 2001. № 2 P. 3–18. Varvak M. Ellipsoidal/radial basis functions neural networks enhanced with the Rvachev function method in application problems // Engineering Applications of Artificial Intelligence. 2015. V. 38. P. 111–121. DOI: 10.1016/j.engappai.2014.09.017.

Arrived 18 сентября 2017 г.

Атомарное машинное обучение © Авторы, 2018 © ООО «Издательство «Радиотехника», 2018

С.Ю. Еременко − д.т.н., профессор,

директор научно-технологической компании Soliton Scientific Pty (г. Сидней, Австралия) E-mail: [email protected] Описано как теория атомарных функций (АФ) развивающаяся с начала 1970-х гг. и применяемая во многих отраслях математики и физики может быть применена в теории Машинного Обучения. АФ в форме пульса может представлять атомное ядро в алгоритмах опорных векторов, оценки плотности распределений, анализа главных компонентов, локально взвешенной регрессии и других. AString, как интеграл от АФ который может представить АФ через комбинацию, может быть основой атомарной логистической регрессии, атомарной активирующей функции нейронных сетей и атомарного компьютера. АФ и AString обладают уникальными свойствами финитности и компактной основы, быстрого вычисления производных через себя, способности представлять секции многочленов и аппроксимировать важные статистические функции. АФ как вероятностное взвешенное равномерное распределение глубоко связано с Prouhet– Thue–Morse последовательностями и принципами «справедливой игры», что может быть важно для некоторых приложений. Реальные расчетные примеры в программах Matlab, R и Python показывают вычисления сложных регрессий, разграничительных границ и кластеров точных целей, потенциально важных для медицинских, военных и инженерных приложений. В сочетании эти методы составляют теорию Атомарного Машинного Обучения, которая расширяет 50-летнюю историю атомарных функций и их обобщений на новые научные области.

REFERENCES 1.

Eremenko S.Yu. Atomic Strings and Fabric of Spacetime. Foreign Radioelectronics: Achievements of modern radioelectronics. 2018, Accepted for publication. https://www.researchgate.net/publication/320457739.

Нейрокомпьютеры: разработка, применение, № 3, 2018 г.

27

Нейром атем атика и интел лектуальны е вы числения 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21.

22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34.

28

Rvachev V.L., Rvachev V.A. About one Finite Function, DAN URSR. A (6). 705–707. 1971. Rvachev V.L., Rvachev V.A. Non-classical methods in the theory of approximations in boundary value problems. Kiev: Naukova Dumka. 1982. 196 p. Rvachev V.L. Theory of R-functions and their applications. Kiev: Naukova Dumka. 1982. Kravchenko V.F., Rvachev V.A. Application of atomic functions for solutions of boundary value problems in mathematical physics. Foreign Radioelectronics // Achievements of modern radioelectronics. 1996. № 8. P. 6–22. Kravchenko V.F., Kravchenko O.V., Pustovoit V.I, Churikov D.I. Atomic, WA-Systems, and R-Functions Applied in Modern Radio Physics Problems: Part 1. Journal of Communications Technology and Electronics. 2014, V. 59. №10. P. 981–1009. https://www.researchgate.net/publication/298406834. DOI: 10.1134/S1064226914090046. Kravchenko V.F., Kravchenko O.V., Pustovoit V.I, Pavlikov V.V. Atomic Functions Theory: 45 Years Behind. https://www.researchgate.net/publication/308749839. DOI 10.1109/MSMW.2016.7538216. Kravchenko V.F. Lectures on the theory of atomic functions and their applications. Moscow: Radiotechnika. 2003. Kravchenko V.F., Rvachev V.L. Logic Algebra, atomic functions and wavelets in physical applications. Moscow: Fizmatlit. 2009. Eremenko S.Yu. Finite Element Methods in Mechanics of Deformable Bodies. Kharkov: Osnova. 1991. 272p. https://www.researchgate.net/publication/321171685. Eremenko S.Yu. Natural Vibrations and Dynamics of Composite Materials and Constructions. Kiev: Naukova Dumka. 1992. 182 p. Eremenko S.Yu., Rvachev V.L. A survey of variants of the structural finite-element method // Journal of Mathematical Sciences. 1998. V. 90. № 2. P. 1917–1922. Eremenko S.Yu. Combined Non-Archimedean Calculi. Foreign Radioelectronics // Achievements of modern radioelectronics. 1996. № 8. P. 57–65. https://www.researchgate.net/publication/320616255. Eremenko S.Yu., Kravchenko V.F., Rvachev V.L. Combined Non-Archimedean Calculi and New Models of Relativistic Mechanics. Foreign Radioelectronics // Achievements of modern radioelectronics. 1997. № 9. P. 26–38. https://www.researchgate.net/publication/320616269. Gotovac H., Kozulic V, Gotovac B. Space-Time Adaptive Fup Multi-Resolution Approach for Boundary-Initial Value Problems // Computers, Materials and Continua. 2010. 15(3). P. 173–198. https://www.researchgate.net/publication/287948119 Sirodzha I.B. Quantum model and methods of artificial intelligence for solution taking and control. Kiev: Naukova Dumka. 2002. 490 p. Kolodyazhni V.M., Rvachev V.A. Atomic functions. Generalizations to the case of several variables and perspective directions of practical application // Cybernetics and Systems Analysis. 2007. V. 43. № 6. P. 155–177. Fabius J. A probabilistic example of a nowhere analytic C ∞-function. Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete, 1966, 5 (2): 173–174, doi:10.1007/bf00536652, MR 0197656. Hilberg W. Impulse und Impulsfolgen, die durch Integration oder Differentiation in einem veranderten Zeitmastab reproduziert warden. AEU. (25). 1. 1971. P. 39–48. Palacios-Huerta I. Tournaments, fairness and the Prouhet–Thue–Morse sequence. Economic inquiry, 2012. 50 (3): 848–849. DOI:10.1111/j.1465-7295.2011.00435.x. Kravchenko V.F., Kravchenko O.V., Konovalov Ya.Yu. Prouhet-Thue-Morse sequence and atomic functions in applications of physics and techniques. Journal of Measurement Science and Instrumentation, 2015, 6(2) ： 128-141. https://www.researchgate.net/publication/280316794. DOI: 10.3969/j.issn.16748042.2015.02.005. Filippov A.T. The Versatile Soliton. Modern Birkhauser Classics. 2000. 261 p. Bougaev A.A., Urmanov A.M., Gross A.M., Tsoukalas L.H. Method of Key Vectors Extraction Using R-Cloud Classifiers. Springer, 2006. DOI 10.1007/1-4020-5263-4_15. Raschka S. Python in Machine Learning. PACKT Publishing, 2016. R-documentation on https://www.r-project.org. Cortes C., Vapnik V. Support-vector networks. Machine Learning. 1995, 20 (3), p.273–297. doi:10.1007/BF00994018. Cleveland W.S. LOWESS: A program for smoothing scatterplots by robust locally weighted regression. The American Statistician. 1981, 35 (1): 54. doi:10.2307/2683591. JSTOR 2683591. Data Science and Big Data Analytics: Discovering, Analyzing, Visualizing and Presenting Data. Wiley: EMC Educational Services, 2015. 432 p. Machine Learning Course, Stanford University, https://www.coursera.org/learn/machine-learning. MATLAB documentation on www.mathworks.com. LIGO Gravitational Wave detector experiment on https://losc.ligo.org/events/GW150914/, https://losc.ligo.org/s/events/GW150914/GW150914_tutorial.html. Scheglov E.G., Arutynov P.A. Non-Archimedean Calculus in the Mathematical Modelling of Digital ICs. Russian Microelectronics. 2002. V. 31. № 6. P. 384–395. Arutynov P.A. Non-Archimedean Calculi in Theoretical Metrology. Izm. Tekh. (Metrologia). 2001. № 2 P. 3–18. Varvak M. Ellipsoidal/radial basis functions neural networks enhanced with the Rvachev function method in application problems // Engineering Applications of Artificial Intelligence. 2015. V. 38. P. 111–121. DOI: 10.1016/j.engappai.2014.09.017.

Нейрокомпьютеры: разработка, применение, № 3, 2018 г.