Software Distributed Shared Memory – New Applications and Scalability Mats Brorsson Department of Information Technology, Lund University P.O. Box 118, S-221 00 LUND, Sweden email:
[email protected] May 17, 1999
Summary Software Distributed Shared Memory (S-DSM) can be used to build cost-effective high-performance computing platforms out of clusters of individual computers. In this project we propose to extend this to new applications, such as database and web servers, that currently have not been studied extensively for S-DSM . We also propose to study large-scale S-DSM systems for embedded platforms. An example application for such a large-scale system is a distributed surveillance system with several hundreds of intelligent nodes where each node collects data from its environment, processes the data and puts it in a shared database system. The project is a cooperation with Axis Communications AB in Lund which will contribute in a reference group and with equipment to build an embedded S-DSM system.
1
Introduction and motivation
Clusters of PCs or workstations have emerged as a cost-effective alternative to high-end compute servers. They are regularly used for scientific applications and are rapidly becoming mainstream, partly thanks to the Beowulf effort [1]. Traditionally, clusters have been programmed with a message passing programming model but with Software Distributed Shared Memory (S-DSM) systems and OpenMP, a shared address space model is now viable, also for clustered systems [4, 5, 11]. Research on S-DSM have so far concentrated on small systems and scientific and engineering applications. For these applications, S-DSM can be very cost-effctive. The research proposed here aims to build on the existing knowledge to investigate new application areas and the scalability of software DSM for clustered systems. Applications such as database servers and web servers are becoming increasingly common in parallel systems and it is therefore important to explore these applications in cluster systems as well. The design of S-DSM systems for these applications is largely unexplored. We also expect that this type of applications will be used in embedded distributed parallel systems in a not so distant future. Another new important application domain is distributed parallel applications. A distributed computing system can be seen as a special case of cluster systems and an example of an application for such a system is a distributed surveillance system used, e.g., for the surveillance of a factory plant etc. We envision systems that in the future can consist of several hundreds of nodes where each node has an number of input devices such as cameras, microphones, finger print scanners, etc. and one or more processors that processes the information that the local input devices produces. This information should be shared with other nodes and possibly also with a database server system that may, or may not, be distributed itself. A decision support system can then be used to process the information in the database or information retrieved directly from the nodes. Similar systems are already under study, but not from the viewpoint of system architecture [10]. In order to make it manageable to program large-scale distributed surveillance systems it is important that the system support a shared address space. The ability of distributed systems to be able to manage shared information has also been emphasised by several internationally recognised researchers, e.g. in [3]. We believe that a shared address space model will provide a tremendous advantage in terms of programming support and these systems could therefore benefit from a S-DSM system. The system should be programmed as one system, and not as hundreds of separate systems. The main problem here is the scalability of S-DSM which is largely unexplored as of today.
-1-
2
Problem statements
2.1
Software DSM for new application domains
This part of the project will study the feasibility and performance of software DSM for commercial applications such as database and web servers. A special focus will be on software DSM for embedded clusters with these applications. More specifically, the problems that we will study are: • What requirements do new applications such as database and web servers put on S-DSM systems • How should S-DSM systems for these new applications be designed? • How are database and web servers best parallelised for S-DSM? • Can embedded clusters be effectively built for server applications? • What network interface support is needed to improve performance? 2.2
Scalability of Software DSM
This part of the project will focus on highly parallel and distributed applications such a distributed surveillance system. The main research problems here are: • How do current S-DSM systems scale to several hundreds of processors? • What application support is needed to program large-scale clusters effectively? • How should S-DSM systems, hardware and software, be designed to support scalability more effectively? • What interconnection network support is suitable for large-scale embedded clusters with software DSM?
3
Approach
The group already has a considerable amount of experience in software DSM systems for scientific and engineering applications [7, 8, 9, 12, 13]. We now intend to expand this to new applications, especially database and web servers. We intend to carry out the project in three parts over a period of five years where the first two parts overlap each other in time. • Study of server applications and their performance on state-of-the-art software DSM systems (2.5 years) • The design and implementation of embedded clusters with software DSM (2.5 years) • Scalability studies of large software DSM embedded systems (2.5 years) 3.1
Study of server applications
The first task will be to study the behaviour of exisisting database servers on a clustered software DSM system. For this purpose we intend to use public domain software such as MySQL and GnuSQL. These are well known database servers for which the source code is publicly avilable. MySQL is already multithreaded for parallel transaction processing. These applications will be ported to a new S-DSM system developed at Lund University [8]. This system will be programmed with OpenMP or with a standard Posix threads interface, both of which makes porting of applications already running on shared memory computers such as SMPs easier. An implementation of OpenMP for S-DSM has been done at the department [4] and will be used in this study. The platofrm in this study is a cluster of dual Pentium II machines, interconnected with a high-speed SCI-network that we have available at the department. The software DSM system being developed for this cluster will particularly make use of the remote memory capabilities of the SCI-network. The results of this study will guide us to design improvements of the S-DSM system which will be the next major part of the study. 3.2
The design and implementation of embedded clusters
The nesecond part of the project deals with the design of software DSM for embedded systems using development systems for the Etrax processor provided to us by Axis (see section 10). There are numerous new research issues here. The main challenge is to design an S-DSM system that does not rely on the elaborate memory management system of commercial microprocessors but which is missing in many embedded processors. In order to do this, we will probably use some sort of fine-grain memory coherence similar to what is used in Shasta [14]. The upside of using embedded processors is that many of them, including Etrax designed at Axis, have extensive I/O-support on the chip which could be used for high-performance low-latency communication in an S-DSM system. The development system from Axis will, in addition to a processor, contain programmable logic devices and we will therefore be able to experiment with hardware features that can facilitate the implementation of software DSM. A prototype consisting of a small number of processors will be built and evaluated in this part of the project. The actual S-DSM system for this platform will as far as possible be built on the system developed as described previously and we will use the same server applications for evaluation purposes. We will in the first year start with a four-processor prototype to be extended to 16 processors in the second year.
-2-
3.3
Scalability studies of large-scale software DSM systems
The first embedded software DSM system will only consist of a few tens of processors. In order to carry out a meaningful scalability study, we intend to expand this system to 256 processing nodes. This will in particular stress the interconnection technology needed in order to deliver reasonable performance for the software DSM system. The structure and design of a scalable S-DSM itself is also a major focus of this part since this has not been studied yet. In order to obtain meaningful applications to evaluate the system we will work closely with Axis in this part of the project. It would be impractical to use a full-blown distributed surveillance system as we envision because of the large number of input devices needed. Instead we will use data files that represent input devices and besides being much less costly, this will ensure the repeatibility of performance experiments.
4
Expected results and impact
We expect that the project will produce a wide range of research results in the area of software DSM, parallel database servers and embedded parallel systems, in particular: • Protocols and techniques to improve performance of software DSM systems for server applications • Strategies to parallelise database and web servers for clustered systems with software DSM • Knowledge of design alternatives for embedded software DSM systems • Performance evaluations of embedded software DSM systems • Evaluation techniques for scalable software DSM systems • Interconnection technologies for scalable software DSM • Software technologies for scalable software DSM
5
Project plan
The project will be carried out by two Ph.D. students during five years. One student (Sven Karlsson) has started on the project on faculty funding. This funding will, however, end by his licentiate degree expected during the winter 1999/2000. A funding from PAMP will ensure that he can continue until a Ph.D. He will mainly work in the first part of the project. A second student is needed to complete parts two and three. The project will be carried out according to the following timeline in which we assume 20% departmental duties, such as teaching, for the students. Ph.D. thesis
Student 1
Software DSM and studies of server applications Licentiate thesis
Student 2
Embedded software DSM systems
Year 1
Ph.D. thesis
Scalability issues with embedded software DSM
Year 2
Year 3
Year 4
Year 5
We propose that the project starts in January 2000 for Sven Karlsson (Student 1) and in October 1999 for Student 2 provided that there is a suitable Ph.D. candidate available. The results from the project will be published in internationally renowned conferences and journals. From the first year of the project we expect to present two state-of-the-art reports according to the following: • A report on the design of software DSM systems suitable for commercial applications. A performance evaluation has been carried out on the dual processor PC cluster at the department of Information Technology. • A report describing the design, implementation and performance evaluation of a small cluster of embedded processors with a software DSM system. In addition to these reports, we will produce deliverables and reports according to the CFP of PAMP. It is also expected that around three M.Sc. students each year will do their thesis project within the context of this project proposal.
6
Preliminary budget
All costs are in kSEK. Salary costs have been adjusted for 3% annual increase. The supervision cost is calculated as 10% of the salary of an Associate Professor for each student. Cost item Student 1 (Sven Karlsson)
1999
2000
2001
320
330
-3-
2002 170
2003
2004
Total 820
Cost item
1999
2000
2001
2002
2003
2004
Total
Student 2
78
320
330
340
350
270
1688
Supervision (Mats Brorsson)
15
110
114
87
60
46
432
Computers and software
50
50
50
50
50
30
280
75
75
75
50
50
325
396
4569
Travelling costs Experimental platform Sum
16
48
192
256
512
159
923
1091
978
1022
1024
The cost for the experimental platform provided by Axis is estimated to 4 kSEK per processor and the net funding asked for from PAMP is 3545 kSEK.
7
Related research
Many references to related research are found in the introductory section of this proposal. Software DSM for scientific and engineering applications has been studied by many groups for some time now. A good survey of the current state-of-the-art has been made by Iftode and Singh in [5]. They also identify several new research directions, most notably the support for fine-grained memory coherence and scalability studies, both of which we intend to study within this project. Studies of server applications for software DSM platforms are, however, scarce. Scales and Gharachorloo have preliminary data on the performance of a commercial database server on fine-grained software DSM [14]. Járai et al. have studied data mining algorithms on hardware and software DSM platforms [6]. Bryhni et al. [2] have performed a preliminary simulation study of a clustered database system with an SCI interconnect and DSM support in hardware. Even though this is not on a software DSM system it is of interest because of the similarities to the PC cluster available for us. There is quite some activity in the area of embedded systems for high-performance processing. There is a yearly workshop devoted to this area in conjunction with International Parallel Processing Symposium (now merged with Symposium on Parallel and Distributed Processing to IPDPS – International Parallel and Distributed Processing Symposium): Workshop on Embedded HPC Systems and Applications. We will of course build our research on these results but as far as we know, there is very little done in the area of server applications on embedded clusters, especially with a shared address space.
8
Relation to PAMP profile
We believe that this project proposal fits very well in the profile of PAMP. The envisioned application of a distributed surveillance system is a real-time application for a large-scale system. A shared address space programming model has the potential to make the development of these system much easier than what is possible with current practice. Results from the project will contain methods to parallelise applications, methods for performance evaluation and methods to design I/O systems (interconnection networks); all of these are explicitly mentioned in the profile.
9
Industrial relevance
First of all, it is expected that the project will directly contribute to the technical know-how of Axis Communications in terms of distributed parallel applications and system design for a distributed shared address space in software. However, the results are of course also applicable in standard cluster of workstation systems and as such they will contribute to highly cost-effective parallel execution and programming of applications such as database and web servers. This will especially benefit small and medium sized enterprises since it will bring high-end computing power to a fraction of the cost of high-end parallel computer servers.
10
Context
The group conducting research on parallel computer systems at department of Information Technology at Lund University currently consists of: • Mats Brorsson, Ph.D., Associate Professor and project leader • Sven Karlsson, M.Sc., Ph.D. student • Andreas Rodman, M.Sc., Ph.D. student
-4-
The research in the group has during the last few years focused on programming models for distributed shared memory architectures, especially the trade-offs between shared address space and message passing, and on protocol improvements for software DSM systems. Mats Brorsson is representing Lund University as a member in EuroTools, a working group for the promotion of European tools and research on high.end computing. The group also has informal, but regular, contacts with Department of Computer Science at University of Copenhagen (Eric Jul and Povl Koch). Mats Brorsson will together with Pallas (http://www.pallas.de) and other partners form a consortium for a project proposal to the new IST programme financed by the European Union. This project proposal will focus on programming tools and instrumentation for OpenMP and is complementary to the proposal as described here. This project will be done in cooperation with Axis Communications AB. Axis has so far been a network client product company and now also want to expand in the technology sector. As part of this, they are interested in how new server applications can be used in embedded systems. The cooperation consists of two parts. The first part is to form a reference group consisting of group members from department of Information Technology and technical personell from Axis. The purpose of this reference group is to transfer information in both directions in a structural manner. The project will gain most of its input regarding distributed applications for highly-scalable software DSM for embedded clusters from this group. The second part is that Axis communication will contribute with equipment for the development of a large-scale embedded cluster with software DSM. The equipment will consist of hardware (processor development boards), software (operating system and communications software) and a limited amount of support. The processor development boards will contain an FPGA that can be used to study hardware features that may facilitate the design and implementation of the scalable S-DSM system. Contact person at Axis is Niklas Jonsson, Axis Communications AB, Scheelevägen 16, 223 70 Lund, tel. 046270 18 00.
References [1]
D. J. Becker, T. Sterling, D. Savarese, J. E. Dorband, U. A. Ranawak, C. V. Packer, Beowulf: A Parallel Worstation for Scientific Computation, in Proceedings of the International Conference on Parallel Processing, 1995.
[2]
H. Bryhni, S. Gjessing and A. Schinco, Preliminary Simulation results of an SCI based Clustered Database Machine, in Proceedings of the SCIzzL-3 conference, Crete, October 3, 1995.
[3]
J. Carter, New Directions and Challenges for Distributed Shared State Management, in NSF Workshop on New Challenges and Directions for Systems Research, St. Louis, Missouri, July 31-August 1, 1997. http://www.cs.utexas.edu/users/new_directions
[4]
John Hård, OdinMP - A proposal on how to implement the OpenMP standard on Networks of Workstations, M.Sc. thesis, Department of Information Technology, Lund University, February 1999.
[5]
L. Iftode and J. P. Singh, Shared Virtual Memory: Progress and Challenges, Proceedings of the IEEE, March 1999, Vol. 87, no. 3, pp. 498-506.
[6]
Z. Jarai, A. Virmani and Liviu Iftode, Towards a Cost-Effective Parallel Data Mining Approach, in Proceedings of Workshop on High Performance Data Mining, (held in conjunction with IPPS'98), March 1998.
[7]
HS. Karlsson and M. Brorsson, Producer-Push - a Protocol Enhancement to Page-based Software Distributed Shared Memory Systems, in Proceedings of the 1999 International Conference on Parallel Processing (ICPP’99), Aizu-Wakamatsu, Japan, September 1999. (to appear)
[8]
S. Karlsson and M. Brorsson, An Infrastructure for Portable and Efficient Software DSM, Technical report, Department of Information Technology, Lund University, P.O. Box 118, SE-221 00 Lund, Sweden, April 1999. To be presented at the 1st Workshop on Software Distributed Shared Memory (WSDSM’99), June 1999.
[9]
S. Karlsson and M. Brorsson, A Comparative Characterization of Message Communication in Applications using MPI and Shared Memory on an IBM SP2, in Proceedings of 1998 Workshop on Communication, Architecture, and Applications for Network-based Parallel Computing, Las Vegas, January 31 - February 1, 1998, pp. 189-201.
[10]
MIT AI Lab VSAM Home Page, A Forest of Sensors, Massachussetts Institute of Technology, Artificial Intelligenc Lab, http://www.ai.mit.edu/projects/vsam/
[11]
OpenMP consortium, OpenMP: A Proposed Standard API for Shared Memory Programming, White paper, http:// www.openmp.org.
[12]
E. W. Parsons, M. Brorsson and K. C. Sevcik, Predicting the Performance of Distributed Virtual Shared Memory Applications, IBM Systems Journal, Volume 36, No. 4, 1997, pp. 527-549.
[13]
A. Rodman and M. Brorsson, Programming Effort vs. Performance with a Hybrid Programming Model for Distributed Memory Parallel Architectures, in Proceedings of EuroPar’99, September 1999. (to appear)
[14]
D. J. Scales, K. Gharachorloo, Towards Transparent and Efficient Software Distributed Sahred Memory, in Proceedings of the 16th ACM Symposium on Operating System Principles, St. Malo, France, October 1997.
-5-
MATS BRORSSON Curriculum Vitae Personal data Full name:
Mats Håkan Brorsson
Email address:
[email protected]
WWW home page:
http://www.it.lth.se/~matsbror
This CV is an abbreviation. For the full CV see http://www.it.lth.se/~matsbror/cv.html.
Employment • Lund University, 1995 – present, Associate Professor • Lund University, 1992 – 1995, Assistant Professor • Lund University, 1991 – 1992, Research Assistant (Ph.D. student) • Telesoft AB, 1989 – 1991, Systems Engineer • Lund University, 1985 – 1989, Research Assistant (Ph.D. student)
Education • Ph.D. in Computer Engineering, 1994 • M.Sc. in Electrical Engeering, 1985
Publications The following list contains refereed contributions since 1997.
Books • M. Brorsson, Datorsystem – program- och maskinvara, Studentlitteratur, 1999. This is an undergraduate textbook in Swedish on computer systems organisation.
Refereed journal contributions • M. Brorsson and M. Kral, Visualisation for Performance Tuning of DVSM System Applications on Networks of Workstations, The Journal of Supercomputing, Volume 13, No. 3, 1999.
Curriculum Vitae
May 17, 1999
1
• E. W. Parsons, M. Brorsson and K. C. Sevcik, Predicting the Performance of Distributed Virtual Shared Memory Applications, IBM Systems Journal, Volume 36, No. 4, pp. 527- 549. • Per Stenström, Mats Brorsson, Fredrik Dahlgren, Håkan Grahn, and Michel Dubois: Boosting Multiprocessor Program Performance using Optimized Cache Coherence Protocols, IEEE Computer, July 1997, pp. 63-70.
Refereed conference contributions • S. Karlsson and M. Brorsson, Producer-Push - a Protocol Enhancement to Page-based Software Distributed Shared Memory Systems, in Proceedings of the 1999 International Conference on Parallel Processing (ICPP’99), Aizu-Wakamatsu, Japan, September 1999. (to appear) • A. Rodman and M. Brorsson, Programming Effort vs. Performance with a Hybrid Programming Model for Distributed Memory Parallel Architectures, in Proceedings of EuroPar’99, September 1999. (to appear) • S. Karlsson and M. Brorsson, A Comparative Characterization of Communication Patterns in Applications using MPI and Shared Memory on an IBM SP2, in Proceedings of 1998 Workshop on Communication, Architecture, and Applications for Network-based Parallel Computing, Las Vegas, January 31 - February 1, 1998, pp. 189-201. • M. Brorsson, Performance Tuning of Small Scale Shared Memory Multiprocessor Applications using Visualisation, in Proceedings of the 10th International Conference on Parallel and Distributed Computing Systems, New Orleans, October 1997, pp. 155162.
Student supervision I am currently supervising two PhD students and three MSc students. I have supervised more than 20 MSc thesis projects.
Professional activities • I am engaged as a reviewer for both international conferences and journals. I also contribute as program committee member and organiser of international conferences and workshops • I am project leader for a large project developing a system for course administration
Curriculum Vitae
May 17, 1999
2
Letter of Support Axis Communications AB will support the project “Software DSM – New Applications and Scalability” headed by Mats Brorsson at the Department of Information Technology at Lund University. The support will take the form of 1. A reference group for discussions around the project. 2. Hardware for building a test system with up to 260 processor nodes for the project. Contact person at Axis Communications AB will be Niklas Jonsson.
Axis Communications AB reserves the right to withdraw the support if the project is cancelled or does not receive funding. Axis Communications AB reserves the right to approve further involvment in the project by other industrial partners. Axis Communications AB reserves the right to withdraw the support if such involvment is made without the approval from Axis Communications AB.
LUND, May 17, 1999
Johan Ericsson General Manager Technology Division
