objects, neural network paradigms) and the exploit of available computing .... Resource Broker, Replica Manager. Data Grid. N2Grid Simulation Service.
© 2004 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.
N2Grid: Neural Networks in the Grid Erich Schikuta and Thomas Weish¨aupl Department of Computer Science and Business Informatics University of Vienna Rathausstraße 19/9, A-1010 Vienna, Austria E-mail: {erich.schikuta, thomas.weishaeupl}@univie.ac.at
Abstract— We present a framework for the usage of neural network resources on a world-wide basis. Our approach employs the novel infrastructure of the Grid as a transparent environment to allow users the exchange of information (neural network objects, neural network paradigms) and the exploit of available computing resources for neural network specific tasks leading to a Grid based, world-wide distributed, neural network simulation system, which we call N2Grid. Our system uses only standard protocols and services aiming for a wide dissemination of this Grid application.
TABLE I G RID L AYERS – U SE C ASE S CENARIO M APPING Knowledge Grid
2-dimensional information analysis and mining of information providing problem solution mechanisms
Information Grid
1-dimensional information semantic interpretation
Data Grid
0-dimensional information data administration
I. I NTRODUCTION A Grid [6] based computational infrastructure couples a wide variety of geographically distributed computational resources (such as PCs, workstations, and supercomputers), storage systems, data sources, databases, libraries, and special purpose scientific instruments, and presents them as a unified integrated resource which can be shared transparently by communities (virtual organizations). The Grid started out as a means for sharing resources and was mainly focusing high performance computing. By the integration of Web Services as inherent part of the Grid infrastructure the focus evolved to the sharing of knowledge to enable collaborations between different virtual organizations or subjects. Accordingly to the definition of the notion of “information” of Gundry we propose a layered Grid architecture based on the dimensionality of the information in focus which allows to differentiate three different Grid layers: •
•
•
Data Grid, 0-dimensional. The Data Grid stores data which represent just facts. Information Grid, 1-dimensional. The Information Grid collects this data in a structured manner and attributes it with semantic contents. Knowledge Grid, 2-dimensional. The Knowledge Grid (similar to [2]) provides problem solution mechanisms on the administered information allowing a human for acting, deciding or planning.
In this architecture each layer (starting from the data layer) provides its functionality to the next layer in form of specific services. In this paper we present N2Grid, which is a Grid application providing services spanning all three layers. N2Grid implements a highly sophisticated connectionist problem solution environment within the Knowledge Grid.
II. M OTIVATION The driving stimulus for development is the exchange of information and resources between researchers. This principle is just as valid for the neural information processing community as for any other research community. As described by the UK e-Science initiative [13] several goals can be reached by the usage of new stimulating techniques, as • enabling more effective and seamless collaboration of dispersed communities, both scientific and commercial, • enable large-scale applications comprising of 10,000 computers, large-scale pipelines etc, • transparent access to “high-end” resources from the desktop, • provide a uniform “look & feel” to a wide range of resources, and • location independence of computational resources as well as data. However we see the problem that in the “connectionist” community these current developments are not used to the maximal possible extent. As justification of our hypothesis we want to put a light on the situation of artificial neural network simulators. In the last few years many different systems were proposed, but only a few got accepted by the community. There are systems developed specifically for certain network paradigms, as Aspirin/MIGRAINES [10], SOM-PAK [12], and some to deliver a comprehensive environment, as AXON [8], SNNS [18], NeurDS [17] to name only a few. Basically all these systems, reaching from highly sophisticated interactive systems to programming language extensions, share the same common problems: • A clumsy software environment, which mostly do not present an intuitive interface to the user. So the user often
•
•
TABLE II G RID L AYERS – N2G RID C OMPONENT M APPING
faces the problem to cope with a complex tool. Most of these systems present a proprietary system, which is not capable to interconnect to other software systems and makes any information exchange difficult or impossible. All these systems lack a generalized framework for handling data sets and neural networks homogenously. During the training phase and the evaluation phase of a neural network the user has to feed the network with large amounts of data. Conventionally data sets are mostly supported via sequential files only and the definition of the input stream, output or target stream of a neural network is often static and extremely complex.
These problems lead to the situation that large number of rarely used simulation systems exist, because most scientists, scared of existing programs, develop their own systems for their specific neural network applications. We believe that this situation is one of the reasons of an obstructed open information and data exchange within the scientific community. We see a solution to this problem by the N2Grid system proposed in this paper. N2Grid is an artificial neural network simulation environment. It provides basic neural network functions like creating, training and evaluating neural networks. The system is Gridbased in order to make it open to the growing Internet/Grid community. The simulator interacts with Grid data resources (as databases) to store and retrieve all relevant data about the static and dynamic components of neural networks objects, and with Grid computing resources to harness free processing cycles for the ”power-hungry” neural network simulations. Further the system allows to integrate additional paradigms found on the Grid, provided by arbitrary users, into the simulator. To get acceptance from the community the development of the N2Grid system is based on widely accepted standards only. Thus we made the main decision that only methodologies based on Grid technology are used for the design and development of the system. N2Grid uses the Grid on the one hand as a communication and collaboration platform for neural network scientists and users to exchange neural network specific knowledge (e.g. network models, data sets), on the other hand it provides access to unused resources (e.g. processors, main memories, disk space) for neural network specific simulation tasks. The focus of the presented paper is the development of N2Grid, a neural network environment based on Grid technology. In a former project the NeuroWeb system [11] was developed, a much simpler Internet based neural network simulator, which passes on several proven design principles to N2Grid. The layout of the paper is as follows. In the following section we give an overview of the N2Grid system, describe the design principles and justify them. Next we present a working prototype implementation. The paper closes with a look at future developments and research directions.
Knowledge Grid
N2Grid Paradigm Serv. (Replica Mng.) N2Grid Java Application/Applet clients N2GPort web portal client
Information Grid
N2Grid Paradigm Archive Service Resource Broker, Replica Manager
Data Grid
N2Grid Simulation Service N2Grid Data Services
III. N2G RID A RCHITECTURE The N2Grid system is a neural network simulator using the Grid infrastructure as deploying and running environment. It is an evolution of the existing NeuroWeb [11], [9] and NeuroAccess [1] systems. The idea of these systems was, to see all components of an artificial neural network as data objects in a database. Now we go ahead and see them as parts of the arising world wide Grid infrastructure. N2Grid is based on a service oriented architecture and spans all three layers of the Grid layer architecture defined in Section I. In Table II we give the mapping of the specific N2Grid services to the Grid layer architecture. In this section we describe all components of the N2Grid system on which the actual running prototype implementation is based (see Section V). A. N2Grid Services N2Grid Services are Grid Services hosted by the Grid infrastructure. They execute neural network simulation tasks (as creation, training and simulation), which are submitted to the Grid and do not consume local client resources. By the submission the work load is minimized on a local machine. Nevertheless the N2Grid system gives also the possibility for local artificial neural network simulations. We can run the following tasks of neural network simulation remotely in the Grid: 1) 2) 3) 4) 5)
Training of neural networks Evaluation of neural networks Processing of data by a neural network Archiving of paradigms Archiving of network objects (nodes, structure, weights, etc.) 6) Using data sources and archives
Task 1, 2 and 3 are integrated into one component in the N2Grid system. The Simulation Service accomplishes the training, evaluation and propagation function of the neural network simulator. A selected paradigm and network instantiation is executed on a Globus Gatekeeper (Version 2). Migrations are planed for Globus 3 [7]. The necessary data are provided by other N2Grid services, described below. Task 4 is implemented as N2Grid Paradigm Archive Service. As follows in the next subsection, the uses can find paradigm archive services over the paradigm services (replica manager).
Web Portal N2GPort Java Applet N2Grid
N2Grid Clients
Simulation Services Globus Gatekeeper
Data Request Local Simulations
N2Grid GridServices Paradigm Archive Services GridFTP, OGSADAI
Java Application N2Grid
n io at t pu m Co
ry ve co is D
R eg is tra tio
Re qu es t
n
Data Services OGSADAI, GridFTP
Database
Broker Services Resource Broker, Replica Manager
Grid Information Service MDS, GMA, R-GMA
Fig. 1.
N2Grid Application Model
More information about the available paradigm implementations provides the paradigm archive service by a directory implemented as OGSADAI database (XML), residing on the same site. Paradigms are implemented as Java classes using the Java Commodity Grid (CoG) Kit [16], [15]. They are transferred by the GridFTP protocol. In special cases the execution of the class will only be allowed on a specific (e.g. one specific remote site) N2Grid Simulation Service, to protect securely intellectual property of the paradigm implementation. Task 5 and 6 are unified by the N2Grid Data Services. OGSADAI provides the access to a database storing all training-, evaluation-, propagation-data and network objects (nodes, structure, weights, etc.). To provide more flexibility, references (GridFTP URLs) to flat files can be registered in the database which can be accessed directly by the neural network simulation system. B. Overview – Infrastructure Figure 1 shows the overall application model of the N2Grid system in a final implementation phase. We assume a sophisticated Grid infrastructure, including independent resource brokers and replica manager. A client does not care about the execution hosts or data sources. It can control a local simulation or a simulation in the Grid infrastructure in a transparent way. Parts of the remote execution are the authentication to the Grid resources, the processing of the job descriptions and the usage of Grid data sources. For local simulation runs the client can also use local arbitrary data sources. The components of the N2Grid are derived from the clientserver model of NeuroWeb to a novel, service oriented,
tripartite Grid application. N2Grid Services realize Simulation Services, Paradigm Archive Services and Data Services. The end user can choose between three different N2Grid clients. These are respectively a Java application with the possibility of direct data base connections, a Java Applet running in a web-browser, and a Web portal. N2Grid services and clients are interconnected by standard Grid broker services, using standard Grid information services. The N2Grid system is based on the common middleware Globus [4], [5], and some extensions of Globus by the DataGrid project [3] and the “Open Grid Services Architecture Data Access and Integration” (OGSADAI) project [14]. Thus we only use existing standard protocols and services to guarantee the dissemination and acceptance of the N2Grid system. As independent and transparent service the resource broker accepts simulation jobs from the client. The job submission for the client is the same in the case of submitting it to a resource broker or directly to a Globus gatekeeper (simulation service). The N2Grid system uses the DataGrid [3] resource broker. Thus for huge data files, as training-, evaluation-, and propagation-data for artificial neural networks, the replica manager of the DataGrid project provides optimized data file access. The replica manager implements also a N2Grid key component, the N2Grid Paradigm service. It gives the clients the possibility to find N2Grid Paradigm archive services providing paradigm implementations. The services establish the mapping between logical paradigm names and the physical location. The user can search for different types of paradigms (e.g. Back propagation, Quick propagation, Jordan, ART, etc.) and after this she/he can browse for a specific implementation in the
TABLE III G RID L AYERS – U SE C ASE S CENARIO M APPING Knowledge Grid
IV-A.1 IV-A.2
Search Paradigm Search Net Object
Information Grid
IV-B.1 IV-B.2 IV-B.3 IV-B.4 IV-B.5
Directed Remote Execution Data-driven Remote Execution Computation-driven Remote Ex. Paradigm-driven Remote Exec. Stand-alone Local Execution
Data Grid
IV-C.1 IV-C.2 IV-C.3 IV-C.4 IV-C.5
Data Pull (GET) Data Push (PUT) Paradigm Pull (GET) Net object Pull (GET) Net object Push (PUT)
Paradigm archive service. C. N2Grid Clients We propose three different clients as shown in Figure 1. First, there exists a N2Grid Java Application client for an advanced, high end user, who can run also a local database with storing his own data. She/he can extend the functionality of his client without influencing the rest of the N2Grid system by including own paradigm Java classes. Second, we provide a N2Grid Java Applet having a similar user interface as the Java application but with limited functionality. Because of the sandbox principle local database and file accesses are not allowed. The user can use the builtin client functions (paradigms) and shared functions of the N2Grid system, as the N2Grid services. The applet can access remote data of the N2Grid Data Service for local simulation runs. Third, for the purpose of thin clients a simple web browser can be used as a front end of the N2Grid system by accessing a web-portal called N2GPort. It provides control over running simulation jobs on the N2Grid services and presents their results. All the functions are implemented by Java Server Pages (JSP) and Servlets using the client Java classes of the Java Application and Applet. IV. U SE C ASES – S CENARIOS For the N2Grid system we propose several use cases, depending on the state of the dynamic and changing Grid infrastructure. Table III shows the categorization of the scenarios according to the Grid layers (see Table I). A. Knowledge Grid – Scenarios 1) Search Paradigm: The Search Paradigm scenario allows users to find a paradigm by the paradigm services (replica manager) in a paradigm archive (N2Grid Paradigm Archive Service). The available neural network paradigms are categorized and mapped to paradigm archive services offering corresponding implementations. In the paradigm archive service directory (OGSADAI database) also different usage patterns or other characteristics of neural network paradigms can be stored.
The user can locate the physical location of a paradigm by this directory. Depending on its properties one of the remote execution scenarios (see IV-B.1, IV-B.2, IV-B.3, IV-B.4) can be chosen. Scenario IV-C.3 can be executed only, if the run time configuration allows the download of the found paradigm (no Java Applet client and no proprietary restrictions). 2) Search Net Object: A specific neural network is defined by its paradigm and its describing attributes (as number and types of nodes, structure, weights of the connections, etc.). Trained neural network objects are solutions to specific problems. Therefore an ever increasing number of neural network objects, each representing a specific problem solution, will be stored by N2Grid and, in turn, will be available as high level problem solver to the users. The Search Net Object scenario uses the N2Grid paradigm archive services to find specific instantiations of neural network paradigms. These net objects are stored in the N2Grid data service. B. Information Grid – Scenarios 1) Directed Remote Execution: If a client knows about the available remote execution hosts (Globus Gatekeeper) it can direct a simulation run directly by the Directed Remote Execution. No interaction with the resource broker is necessary. The used data can be sent with the job or are located by references in the Grid infrastructure. 2) Data-driven Remote Execution: The Data-driven Remote execution scenario uses the resource broker to execute remote simulation runs. The client does not care about the used host machines. Depending on the location of the needed data the resource broker aligns the job on the computation nodes next to the data. This is an advantage for data intensive simulation runs, because transfer of data to other compute nodes is communication intensive and slow. 3) Computation-driven Remote Execution: In opposite to the data-driven remote execution the Computation-driven Remote execution scenario applies another align policy. For less data intensive simulations the resource broker chooses free, power-full compute nodes. Some paradigms can also have parallel implementations. To execute these, the resource broker has to assign the jobs to front ends of clusters or other parallel architectures. 4) Paradigm-driven Remote Execution: In the Paradigmdriven Remote execution scenario the resource broker and simulation client have to choose a defined host for the execution of the specific paradigm simulation. This is due to proprietary restrictions and copyrights of the paradigm, which allows the owner to sell the program with restricted access and execution rights. In turn this scenario gives the user the advantage of more economic access, because she/he has only to pay for the usage of the paradigm and not for the paradigm itself. 5) Stand-alone Local Execution: In the case of standalone execution of the N2Grid client (Java Application or Applet) a comprehensive set of standard build-in functions and paradigms are available for local simulation runs. The
user can define specific network instantiations of the build-in paradigms and execute them. At the time being the Backpropagation, Quickpropagation, Resilient propagation, Self organizing maps, Counter propagation, Hopfield net, ART 1/2, Kohonen net, Jordan net, Elman net, Cascade Correlation (with Quickpropagation) and CNN paradigms are supported by the N2Grid prototype. C. Data Grid – Scenarios 1) Data Pull (GET): Besides the neural network instantiations also the input data is stored in the Grid infrastructure. The N2Grid Data Service provides an interface to these data by OGSADAI to relational and XML databases or by GridFTP to simple files. The Data Pull scenario establishes the connection between a local simulation and remote input data source, which can provide training-, evaluation- or process (propagation) data. 2) Data Push (PUT): In opposite to the Data Pull case the N2Grid Data Service can also be used to store and write output data into the Grid by the Data Push (PUT) scenario. A local client can save the neural network simulation results (output) by the OGSADAI interface or directly to a file over GridFTP. 3) Paradigm Pull (GET): As mentioned in Scenario IV-A.1 it is possible to download and install a paradigm locally on a client by the Paradigm Pull (GET) scenario, if the running configuration allows the download. 4) Net object Pull (GET): Like Scenario IV-C.3 also specific neural network objects (instantiations) can be downloaded on a local client by the Net object Pull (GET) scenario. 5) Net object Push (PUT): If a neural network simulation run has produced a trained network object of a given paradigm for a specific problem, it is possible to store this new network instantiation by the Net object Push (PUT) scenario. The N2Grid paradigm archive service and the data services are used to register the new settings and save the describing attributes of the network objects (as nodes, structure, and connection weights). V. N2G RID P ROTOTYPE The prototype is under development, three scenarios (see Section IV) are already fully implemented, which are the scenarios Data Pull (GET), Data Push (PUT), and Stand-alone Local Execution. The other scenarios are implemented partly until now. Figure 2 shows the concept of the actual prototype implementation of the N2Grid system. When the simulator N2Grid starts up the N2Grid Client window is shown (see Figure 3). In the upper left tree all build-in neural network paradigms are listed. These paradigms are those neural network types that are always part of the simulator. To find other paradigms the paradigm services are used. In the lower left corner paradigm categories are listed. By choosing a category and pressing the find button the replica manager is asked for N2Grid paradigm archives providing corresponding paradigms. The upper right corner of the window is used to control the N2Grid Data Services
Fig. 3.
Actual N2Grid Prototype User Interface (Java Application)
Fig. 4.
The N2Grid net creation window
(Grid OGSADAI or JDBC/ODBC). Below the user controls the N2Grid Simulation Services. After choosing a network paradigm the user can define the structure of the network according to the selected paradigm in the create net window (see Figure 4). In the shown example we build a classical XOR backpropagation network with 2 input, 5 hidden and 1 output neuron, which is depicted in the graph section of the create net window. The net structure can be loaded and saved by the N2Grid Data service. VI. C ONCLUSION We presented the N2Grid project, as a next step in the program evolutions for neural network simulation. It is a framework for the usage of neural network resources on a world-wide basis by the upcoming Grid infrastructure. Our system uses only standard protocols and services to allow a wide dissemination and acceptance. The prototype implementation is still under development and will be extend to the proposed final implementation in a
Java Applet N2Grid
Simulation Services Globus Gatekeeper
N2Grid Clients
N2Grid GridServices
Processing Work
Paradigm Archive Services GridFTP, OGSADAI
Java Application N2Grid
Data Service OGSADAI, GridFTP
Database
Paradigm Server (NPS) Java RMI Server
Internet
Database ODBC/JDBC Relational Database
Simulation Server (NSS) Java RMI Server
Fig. 2.
Actual N2Grid Prototype Implementation
short period. To reach the full capability of our system and to develop more sophisticated systems further research has to be done in the following two areas, • The description of the paradigm has to be enhanced, to establish easier sharing between paradigm providers and customers. These semantic description is a key concept specified only rudimentary until now by a directory implementation. We will define a semantic paradigm description language by a pattern and/or scheme approach using XML. • The actual N2Grid client controls single simulation runs. To allow the building of large connectionist systems consisting of several neural network instantiations (possibly of different paradigms) an extension of the N2Grid system is on the way, which allows to control a flow of simulations by a specific neural network workflow language. R EFERENCES [1] C. Brunner and C. Schulte. NeuroAccess: The Neural Network Data Base System. Master’s thesis, University of Vienna, Vienna, Austria, 1998. [2] M. Cannataro and D. Talia. The Knowledge Grid. Communications of the ACM, 46(1):89–93, January 2003. [3] European DataGrid Project. The DataGrid Project. http://www. edg.org. R The Open Source Solution for Grid [4] I. Foster. The Globus Toolkit°: Computing. Keynote, GlobusWorld, San Diego, January 14 2003. [5] I. Foster and C. Kesselman. Globus: A Metacomputing Infrastructure Toolkit. Intl J. Supercomputer Applications, 11(2):115–128, 1997. [6] I. Foster and C. Kesselman, editors. The Grid: Blueprint for a New Computing Infrastructure. Morgan Kaufmann, 1999. [7] Globus Project. Globus Project. http://www.globus.org. [8] R. Hecht-Nielsen. Neurocomputing. Addison/Wesley, 1989. [9] E. Hombauer and S. Zach. Neuroweb: A Webbased Neural Network Simulator. Master’s thesis, University of Vienna, Vienna, Austria, 1999.
[10] R. Leighton. The aspirin/migraines software tools user’s manual. Technical report, The MITRE Corporation, Virginia, 1991. Technical Report MP-91W00050. [11] E. Schikuta. NeuroWeb: an Internet-based neural network simulator. In 14th IEEE International Conference on Tools with Artificial Intelligence, pages 407–412, Washington D.C., November 2002. IEEE. [12] SOM Programming Team SOM-PAK. The self-organizing map program package, 1992. user guide. [13] UK e-Science. UK e-Science programme. http://www. escience-grid.org.uk. [14] UK e-Science Grid Core Project. Open Grid Services Architecture Data Access and Integration OGSA-DAI. http://www.ogsa-dai.org. uk. [15] G. von Laszewski, B. Alunkal, K. Amin, J. Gawor, M. Hategan, and S. Nijsure. The Java CoG Kit User Manual. MCS Technical Memorandum ANL/MCS–ATM-259, Argonne National Laboratory, Mathematics and Computer Science Division, 9700 S. Cass Ave, Argonne, IL 60439, U.S.A., 2003. [16] G. von Laszewski, I. Foster, J. Gawor, and P. Lane. A Java Commodity Grid Kit. Concurrency and Computation: Practice and Experience, 13(8-9):643–662, 2001. http:/www.cogkits.org. [17] D. Wecker. The neural design and simulation system (NeurDS). Technical report, Digital Research, 1989. [18] A. Zell, G. Mamier, M. Vogt, N. Mache, R. H¨ubner, S. D¨oring, K.U. Herrmann, T. Soyez, M. Schmalzl, T. Sommer, A. Hatzigeorgiou, D. Posselt, T. Schreiner, B. Kett, G. Clemente, and J. Wieland. SNNS Stuttgart Neural Network Simulator user manual. Technical report, University of Stuttgart, 3/92 1992.
