NeuDB'95: An SQL Based Neural Network ... - Semantic Scholar

NeuDB'95: An SQL Based Neural Network Environment Erich Schikuta Institute of Applied Computer Science and Information Systems, Dept. of Data Engineering, University of Vienna, Rathausstr. 19/4, A-1010, Vienna, Austria, [email protected]

Abstract| The NeuDB'95 system is a novel approach to the physical and conceptual in-

tegration of neural networks into an object-oriented database systems. In the context of the database system neural networks are seen as basic objects and are administrated by the standardized SQL interface of the system. The network paradigm of a neural network object is de ned by the type hierarchy of the general neural net database type. The structural information is stored using a data oriented approach. The dynamic components of the neural networks are triggered by conventional SQL statements for insertion, update, deletion and access. They are processed by an independently running arti cial neural network simulator.

1 Introduction In the last few years many dierent systems for the easy and software supported creation and administration of neural networks were presented. Some of the system tackle only special types of neural networks as for example Aspirin/MIGRAINES [5] or SOM-PAK [13]. Other try to deliver a comprehensive tool as AXON [3], SNNS [15] or NeurDS [14]. In general all systems provide the user with a proprietary software environment, which reaches from highly sophisticated interactive systems to programming language extensions. They confront the user with the problem to cope with a new and/or complex tool. Further most of these systems present a standalone environment, which is not capable to interconnect to other software systems. A further problem of all these systems is the lack of a generalized framework for handling data sets and neural networks homogenously. During the training phase and the evaluation phase of a neural net the user has to feed the net with large amounts of data. Conventionally data sets are mostly supported via sequential les only and the de nition of the input stream, output or target stream into a neural net is often extremely clumsy, static and complex.

2 Database Approach Object oriented database systems have proven very valuable at handling and administrating complex objects. The object-oriented approach seams (and in our opinion has proven) the most comfortable and natural design model for neural networks [4]. In the context of object-oriented database systems neural networks are treated generally as complex objects. These systems showed very valuable at handling and administrating such objects in dierent areas, as computer aided design, geographic databases, administration of component structures, etc. It is our objective to consider neural networks as conventional data in the database system. From the logical point of view a neural network is a complex data value and can be stored as a normal data object. The usage of a database system as an environment for neural networks provides both quantitative and qualitative advantages. 



Quantitative Advantages. Modern database systems allow the administration of objects eciently. This is provided by a 'smart' internal level of the system, which exploits well studied and well known data structures, access paths, etc. A whole bunch of further concepts is inherent to these systems, like models for transaction handling, recovery, multi-user capability, concurrent access etc. This places an unchallenged platform in speed and security for the de nition and manipulation of large data sets at users disposal. Qualitative Advantages. The user has powerful tools and models at hand, like data de nition and manipulation languages, report generators or transaction processing. These tools provide a uni ed framework for both handling neural networks and the input/output data streams of these networks. A homogeneous and comprehensive user interface is provided to the user. This spares awkward tricks to analyze the data of his database with a separate network simulator system.

A further important aspect (which is beyond the scope of this paper) is the usage of neural networks as part of the rule component of a knowledge base database system [9]. Neural networks represent inherently knowledge by the processing in the nodes [6]. Trained neural networks are similar to rules in the conventional symbolic sense. A very promising approach is therefore the embedding of neural networks directly into the generalized knowledge framework of a knowledge based database system.

3 The NeuDB'95 System The NeuDB'95 system is based on the Postgres'95 database system [7] and an adapted version of the NeurDS neural network simulator [14]. The predecessor to the NeuDB'95 System was the original NeuDB System [10]. This was the rst approach of integrating neural networks into a database system. The main drawback of this system was the propriatory interface language of the Postgres 2.0 database system, called Postquel, which was dicult to understand and to use (in spite of or, maybe due to, its logical soundness). The user acceptance of the whole system suered from the clumsy interface language. This led to the replacement of the Postquel language by the SQL[2] interface in the latest Postgres'95 system. The replacement of the interface language of the Postgres system triggered a comprehensive redesign of the NeuDB system too and resulted into the new SQL based NeuDB'95 system.

3.1 System design Postgres'95 is a highly extensible database system with object oriented features, like type inheritance. The data model of Postgres'95 consists of classes (relations) containing tuples, which represent real-world entities. A class has attributes of a xed type that represent properties of the objects stored in the class. The type inheritance feature allows a natural and comfortable modeling of the neural network data type according to our embedding framework. A neural network class is embedded into the Postgres'95 class system and the rule system of Postgres'95 is heavily exploited to emulate the dynamic neural network component. We chose Postgres'95 because of its capability to de ne triggering rules on database operations [12]. Using the rule system a data-driven request-action strategy was established. The insertion or update of neural network objects noti es the neural network simulator by inserting a request record into a dedicated communication class the server is listening to. This approach realizes represent the dynamic component of the system. The Arti cial Neural network simulator runs as an independent server process (ANNSserver) to the database system in parallel and performs the actual network evaluation and training. It provides a user interface, which allows to monitor the state of the simulator. This proved extremely useful for the training phase of a network. The user can create trace les, which can be used for the design of succeding training phases. The following gure shows the process structure of the NeuDB'95 system. Database frontend

insert delete results

requests Postgres’95 data base

actions

ANNS server

Figure 1: The NeuDB'95 process structure The Postgres'95 database server administrates the data on disk and requests actions of the ANNSserver corresponding to database commands (insert, update, eval). The ANNSserver creates an arti cial neural network according to the data description, performs the requested actions and updates the database accordingly. The user at the computer console directs both processes and controls interactively the system status. The distribution of the tasks to 2 separate and independently running servers gives the advantage of exploiting the inherent parallelism of the system. While the ANNSserver is processing the neural networks (training or evaluation), the database server is providing the input data streams in parallel. Generally it can be seen as a pipelined process structure. Postgres'95 is designed to support the database server paradigm, where an independent database server runs on a dedicated machine and communicates with the front-end processes via the network.

We exploit this paradigm by a 'request - action' strategy. The user connects to the database via a frontend process (this can be the usual Postgres'95 'monitor' or any other available front-end program). User inputs trigger actions (e.g. the insertion of a training object triggers a training phase), which are resolved by a call to the ANNSserver. The server processes the requests (e.g. the training phase) and performs dependent data actions (e.g. the insertion of link weights). The ANNSserver can physically distribute its tasks among available processing units of the underlying network. So it can exploit idle workstations of the network to perform the very time consuming neural network actions. The user has not to wait for the accomplishment his requested actions; he just initiates it and continues with his work. This allows to parallelizes the requests, for example multiple training phases. The whole NeuDB'95 design exploits heavily parallelism to speed-up the program execution. More speci c we distinguish between three levels of parallelism, the processes(user monitor, database system, ANN server), inter-operation (SQL requests), intra-operation parallelism (training). A comprehensive description and performance analysis can be found in [11].

3.2 Data types The 'NeuralNet' type is a subtype of the general object type of the database system. Subtypes of this NeuralNet type can be classi ed into specialized neural network types according to their network paradigm. It is also possible that a network paradigm is the supertype of another more specialized paradigm. In the NeuDB'95 system a neural network is generally de ned by the 'NeuralNet' class, which provides a unique identi er for referencing the neural network object. It also describes the basic structure of the network by a Layer attribute. This attribute contains a sequence (basically an array) of integer values, which de nes the number of processing elements in each layer. In the following we give the SQL statement for the respective class creation for clari cation. These are no user actions. The classes are prede ned in NeuDB'95. create table NeuralNet ( NNId char16, Layer int4[] );

At this de nition level nothing is said about the network paradigm. It is de ned by a specialization, a subtype of 'NeuralNet'. This subtype (which inherits all characteristics of its supertype) provides the speci c and necessary attributes dependent on the network paradigm. Combined with the de nition of the paradigm is the dynamic behavior of the network. For example, a feedforward network paradigm with backpropagation training algorithm is provided by the system class, create table BPN ( ConnectInToOutput bool ) inherits ( NeuralNet );

The characteristics of a trained network is de ned by the 'Train' class. This class provides all attributes for the training phase of the referenced neural network object. It is organized as a hierarchical type structure similar to the basic neural network class. Generally a 'Train' class consists of the neural object reference (NNId), a unique 'Train' object identi er (TId) and a de nition of the input and target data stream. The notation '[ ]' denotes an array of values. create table Train ( TId char16, NNId char16, InputA float8[], TargetA float8[], InputF text, TargetF text );

The data values can be de ned explicitly by a value sequence or implicitly by an SQL statement (see the following section). A 'Train' sub-class provides object characteristics dependent on the network paradigm. create table TrainBPN ( InitWeight float8, ActivationF char16, ...

)

inherits ( Train );

This class de nes all describing attributes for the training phase respective to the network paradigm. The training action is started by the insertion of a training object into the specialized class. After a completed training phase the user can use the 'trained' neural network object to analyze his data sets. This can be done via two mechanisms, the creation of an 'Eval' (evaluate) object or the use of the 'eval'-function. The

'eval' function allows the user to evaluate a neural network object according to the training characteristics responding to an input data stream. This function can be seen as an embedded operator and can be used in any SQL statements. The second possibility is the insertion of an 'Eval' object into the respective class. The insertion triggers an eval action, which is resolved by the ANNSserver. The result is inserted into a result class. The user can directly access this class or via a composed 'Output' attribute (an SQL function) de ned on 'Eval'. The creation of an 'Eval' object has the advantage that the result of the neural network evaluation is stored in the database and has not to be calculated again. It can easily be accessed via the conventional database operations.

3.3 Functional data stream de nition The functional data stream de nition allows to specify the data sets in a comfortable way. It is not necessary to specify the data values explicitly, but the data streams can be described by SQL statements. The well known apparatus of the SQL database manipulation language is at hand. Thus the same tool can both be used for administration and analysis of the stored information. So it is easily possible to use 'real world' data sets as training set for neural networks and to analyze other (or the same) data with trained networks (see Section 4). The main design principle of the NeuDB'95 system is that the handling of a neural network has to be simple. That means that there has not to be a complex or arti cial procedure to use it. In contrary the environment has to supply functions to manipulate neural network in a natural and (more important) commonly known way. In the formalism of the database system both types of information (the 'realworld' data and the neural networks) are 'just' data. These data objects are therefore administrated and handled homogeneously within the same framework. To perform a useful training phase the user has to provide certain training phase controlling parameters. The type and the number of parameters are highly dependent on the network paradigm. All paradigms expect input and target data. In an increasing number of neural network applications these data sets show tremendous sizes, like in cluster analysis, pattern recognition, and others. Therefore the access costs can not be neglected and have to be considered in the processing times. In these cases the database facilities of the NeuDB'95 system can improve the performance of the data analysis process tremendously. Besides the mentioned data pipelining of the independent processes, the access mechanisms of the database system provide a powerful tool for the handling of these large data sets.

4 An example: The XOR problem The following section gives an example of the creation, training and usage of a neural network object which solves the well known XOR-problem [1]. All user inputs are marked by a preceding > prompt. The shown examples represent actual screen dumps. The sometimes intricate format is a 'specialty' of the basic Postgres'95 monitor program. 0

0

4.1 Creation and administration To solve the XOR-problem we use a feedforward multi-layer network with the backpropagation training algorithm [8]. First we have to create a neural network object within the database system. This is easily done by an insert to the back-propagation network class BPN. We de ne the network identi er (for future references to this object), the number of layers (3), and the number of processing elements for each layer (3, 1, 1 respectively). > insert into BPN values ( 'XOR-Net', ' 3, 1, 1 ', 't' );

The links between the processing nodes can be inserted explicitly. In can also be done automatically by setting the 'ConnectInToOutput' ag to true. This triggers a Postgres'95 rule, which inserts the correct links to the Link class automatically. Actually the ANNSserver performs these insertions. The status of the new neural network object is untrained. Via the network identi er NNId the object can be referenced and administrative database tasks can be performed, like updating > update BPN set Layer = ' 3, 5, 1 ' where BPN.NNId = 'XOR-Net';

or deleting

> delete from BPN where NNId = 'XOR-Net'; .

During the update and the delete operation the consistency is kept in the database system in accordance to the status or properties of the neural network. Both operations trigger rules, which change the operation dependent objects accordingly. This can be an update of the connections of the processing elements represented by entries in the Link class (see 'update' example) or the deletion of training characteristics of neural network objects (see 'delete' example). All these 'consistency-keeping' operations are performed automatically without a forced activity by the user.

4.2 Training The training phase is started by the insertion of a training object. This object contains the necessary training parameter as attribute values. Besides the neural network identi er, the training identi er and the Input and Target data stream the other values are set to default values. Thus it is not necessary to specify all values explicitly (but it is good programming practice). > insert into TrainBPN ( NNId, TId, InputA, TargetA, ... ) values ('Train1', 'XOR-Net', 'f 1.0, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 1.0, 0.0, 1.0, 1.0, 1.0 g', 'f 0.0, 1.0, 1.0, 0.0 g', ... );

The above example uses input and target values explicitly. But as mentioned in Section 3.3, it is also possible to apply a functional data stream de nition and use input and/or target functions. These functions are SQL statements. The result of these statements are sequential data streams with the same properties as the respective explicit value de nition. A training object with a functional data stream de nition can therefore be inserted by (comments are denoted by '- -' in SQL) > insert TrainingBPN ( NNId, TId, InputF, TargetF, ... ) values 'XOR-Net', 'Train2', 'retrieve(A.x, A.y) from A in InputValues', -- input function 'retrieve(B.z) from B in TargetValues', -- target function ... );

The result of a training phase is a new object in the 'weight' class. This object can be accessed by the following select statement > select * from weight where NNId = 'XOR-Net' and TID = 'Train2';

which produces as result --------------------------------------------------------| NNId | TId | Weights | cWeights | --------------------------------------------------------| XOR-Net | Train2 | 'f1.93072,4.76279,-5.74234, -3.07542,-5.74085,-3.075,-7.72916g'| 'f0.000641426,0.00224068,-0.00127401,-0.00165155,-0.00126514, -0.00171889,-0.00279093g'| ---------------------------------------------------------

4.3 Usage As mentioned above, neural networks can be used to analyze test data by the insertion of an 'Eval' object, > insert into Eval ( EId, NNId, TId, InputA ) values ( 'E1', 'XOR-Net', 'Train1', 'f 1.0, 1.0, 1.0 g' );

or by the 'eval' function, > select eval('XOR-Net', 'Train1', 'f 1.0, 1.0, 1.0 g');

which produces --------------| x | --------------| '0.199699' | ---------------

We have the possibility to use a functional data stream de nition (an SQL statement) instead of explicit input values, too.

5 Conclusions We used the NeuDB'95 system, and its predecessor, for a certain while and it proved extremely well with practical applications. We tested it with the administration and analysis of medical data sets and it reached a high degree of acceptance throughout the group of test user. The homogeneity of the database and neural network simulator interface found general acceptance and a high attraction. The next milestone will be the replacement of the NeurDS based neural network simulator with a novel parallelizing arti cial neural network simulation system. We aim for a highly parallel system design, which can employ equally well a underlying workstation cluster environment or a specialized parallel hardware architecture.

References [1] Beale R., Neural Computing: an introduction, Hilger, 1990 [2] Melton J., Simon A.R., Understanding the new SQL: A Complete Guide, Morgan Kaufmann Publishers, 1993 [3] Hecht-Nielsen R., Neurocomputing, Addison/Wesley, 1989 [4] Heileman G., et al., A General Framework for Concurrent Simulation of Neural Networks Models, IEEE Trans. Software Engineering, 18, 7, pp. 551-562, 1992 [5] Leighton R., The Aspirin/MIGRAINES Neural Network Software, user manual, MITRE Corp., 1992 [6] Pao Y.-H., Sobajic D.J., Neural networks and Knowledge Engineering, IEEE Knowledge and Data Engineering, 3, 2, pp. 185 - 192, June 1991 [7] Rowe L., Stonebraker M., The POSTGRES data model, Proc. 1987 Conference on Very Large Database Systems, Brighton, 1987 [8] Rumelhardt, D.E. et al., Learning internal Representation by Error Propagation, in Rumelhardt D.E. et al., Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol 1, MIT Press, 1986 [9] Schikuta E., The Role of Neural Networks in Knowledge Based Systems, In Proc. Int. Symp. on nonlinear theory and applications, IEICE, Hawaii, Dec. 1993 [10] Schikuta E., The NeuDB-system: Towards the Integration of Neural Networks and Database Systems, In Proc. KI'94, Saarbruecken, Springer-Verlag, Sept. 1994 [11] Schikuta A., Parallelism in the NeuDB System, In Proc. 2nd Int. Conf. on Massively Parallel Computing Systems, Ischia, IEEE Computer Society Press, May 1996 [12] Stonebraker M., et. al., On Rules, Procedures, Caching and Views in Database Systems, In Proc. 1990 ACM-SIGMOD Conf. on Management of Data, Atlantic City, June 1990 [13] SOM Team, SOM-PAK, The Self-Organizing Map Program Package, user guide, Helsinki, 1992 [14] Wecker D.B., The Neural Design and Simulation System (NeurDS), Digi TR. 589, May 1989 [15] Zell A. et al., SNNS, Stuttgart Neural Network Simulator, User Manual, Tech.Rep.No. 3/92, Univ. Stuttgart, 1992