Paralex: An Environment for Parallel Programming in Distributed

Paralex:

An

Environment

for

Parallel

Distributed labao~lu

Ozalp

Alessandro

Alvisi

Davoli

Luigi

Department

of

University Piazza

Alberto

Bologna

S.

Donato,

Bologna,

Italy

true Modern

distributed

systems

and

are

an

per

computers.

addressed

communication,

fault

software

ming

provide

these

issues

and

the

plication

programmer

plexities

of distributed

programming

entific

the it

this

All

achieve

fault

paper

we give

tolerance

and

sci-

for

and

disrepli-

dynamic

by the

of Paralex

a prototype

use of

code

a network

generated

com-

parallel

necessary

an overview

with

extensive debug

load

system. and

dozens,

be

super

harness

the

for

single,

is

general in

present

to

through

exploiting

that

power

parallelism.

be

One

of

future

possible the

only

many

networks, geneity,

“This

work was supported Communities

Action

Number

tems),

the

under 3092

United

University,

and

to

without

Permission granted direct

copy

provided commercial

title

of the

that

copying

that

ad-

Machinery.

fee

the copies

tion

the

sII or part

otherwise,

of this

are not made

the ACM

and

ACM

0-89791

Ministry

-485-6/92

material

of the Association

notice notice

and

As

or to republish,

lack

/0178

There

man’s

netsuper-

architectural

but

(ii)

linguistic

com-

homo-

of communicafall

under

or weeks

also

in fault

not

multiple

of computation

can be wasted

off, Given

In these

programs

only

tolerance.

(e.g.,

of computations

hardware

parallel

expertise

applications

distributed

(vi)

developing

requires

[14].

uine

and

and

with

high-latency

probability

failures,

par-

asynchrony

communication,

increased

a consequence,

is turned

special-purpose

exhibit

domains.

systems

genome

A large

number

analysis)

require

on a network

with

applications,

many

not

dozens

if there

but or

disconnected

from

most

common

components

the

system

are

the

if one of the

workstations

control

of

and

multiple

hours

are gen-

failures, that

also

only

in

in distributed

rebooted

under

domains events

(typically are

and the

a fee

much

processors

that

the

netof a

they

are

administrative

We claim

that

programming

to

assembly

quential primitives

. ..$1 .50

178

individuals more

who

plausible

“own”

and

them),

frequent

these

than

real

failures.

lel

munication, /0007

‘(a poor

to sys-

before

low-bandwidth,

of workstations

is for

is given

requires

(i)

and

(iv)

processor

typically

of

for Computing

and/or specific permission. ICS ‘92-7 /921D.C., USA ~ 1992

Eu-

contract

or distributed

copyright

appear,

these

Sys-

under

Italian

to try

in

remain

from

they

relatively

administrative

hardware

and its date

is by permission To copy

Computing

Research

and

of

Research

Technology.

advantage,

publication

Naval

of

Corporation

Research

Basic

Dependable

Office

IBM

by the Commission Programme

(Predictably

States

NOO014-91-J-1219,

in part ESPRIT

special-

applications. that

differ

in that

(v) exhibit

work. ropean

work-

expensive,

available

can become

over

computing,

significant

will

power system

it is tempting

obstacles

computation

municate

such

implementation,

agreement

computing

computing

an

Thus,

systems

computers

respect

In

Introduction

increases

for ele-

distributed

compute-intensive

of workstations

potential processing

of general-purpose to

parallelism

several

Distributed allel

days

There

hundreds,

computer.

however,

of important

1

of raw modern

comparable

massive

their

multiple

amount

if not

purpose tems

the

in a typical

may

is

across

computer.”

is a complete

across

is automatically

experiences

and

of the

computation

to

makes

execute

fact,

with

is ap-

the

In

stations

systems

execution

is present

works

fault-

parallel

from

Paralex

and

edit,

in

distributed

that

are,

system

the

liberated

bal-

towards

Paralex

to which

be

environment

balancing our

extent

a

program-

be experts

The

in

load

parallel support

to

be

high-latency

dynamic

for

systems.

applications.

cating

users

to

of

parallel

ments,

su-

need inherent

automatic

can

to define,

tributing

and

computing.

at exploring

graphics

no

require

distributed

aimed

parallelism

systems

or

that

heterogeneity,

tolerance

little

powerful networks

special-purpose

issues

the

include

ancing.

Current

to

technical

exploiting

system

of

interconnection

alternative The

in

distributed

consisting

high-speed

economical

tolerant

5

vantages

ABSTRACT

Amoroso

Giachini

Mathematics of

Porta

40127

workstations

in

Systems*

Lorenzo

Renzo

Programming

current in

language

systems

—

technology systems

programming the

to accomplish remote

software

distributed user data

execution,

must

for

for

paral-

is comparable traditional

resort

to

encodingldecoding, synchronization,

se-

low-level comfailure

detection

and

technologies lems

recovery. already

individually. these that

plications

Paralex

mer

can be isolated

to

systems. will

Our

tolerance

has

realize of the

using

the

addresses

can

reuse

The

issue

tation

and

for

application

paradigm

supported

data

of nodes

flow

and

(functions, cate

alized

having

links the

contain node

they

new

relations

out

large-grain

can

allel

flow,

While

model

cation allel.

parallelism have

programmer

has

pro-

so-called

by the highis

available

Only ratio

reasonable

for

by keeping

the

above

in its notation

to deal

with

How

on

nodes.

of

with

input

“pure”

data in that

persistent components are

library

not

return

Another

and

pack

types

such

as structures,

option

out

pos-

to

com-

architec-

3.3.

for multi-functions

used

to

program

the

is to implement

the

results

call-by-reference

through

results records

next

be

as C,

through

subject

being

multiple

in the

such

in Section

possibility

tions this

languages

are specified

language

may is available,

or Lisp 1. It is also

functions

outputs

and

computation notation

to be carried

possibility

procedures

Filter

functions

is to

use simple

into

composite

or arrays.

as

funcdata

We

pursue

section.

par-

programming

Filters

permit functions.

— all nodes

can execute

in par-

abstractions

that from

se-

multi-functions

sult

to be picked

this

manner,

tion

may

tered”

appli-

Nodes

simple

parallel

are familiar

be

external

of the

as discussed

the

One

from

First,

or

multiple

four

They

subsets

be sent

to

by some by

of the

data

different

single

For

by the

destinations

in

a single

each

of the

In func-

the

com-

for

data-

paradigm large

in the computation

to be processed

re-

outputs.

produced

example,

using

(structured)

multiple

is a principal

node

extracting

sub-matrixes

the

to produce

This

computing.

to be implemented

allow

apart

graph.

produced

properties.

number

binaries

putation

performance

the

computations

Modula

computations

compatibility

to reasonable

systems.

desirable

small

This

comincoming

code

a Paralex

that

interdependencies

the

of

of the

particular,

programming

Pascal,

tural

2.2

systems.

is explicit

no data

Second,

flow

[6] is motivated

simple,

has several

visu-

data

sequential

nodes

network

in such

extremely

paradigm that

we expect

are

the

computations.

in distributed

applications

can

indeed

representing

communication-to-computation

levels

programs

for

parameters.

indi-

computations.

significant

low-bandwidth

computations links

In

appropriate

sequential Fortran,

depends as

whatever

piled

a

classified

the

of ordinary

the

flow

to and

(and

in

is composed

Paralex

graphs

a collection

data

communication

the

Thus,

screen)

data

be

correspond

no-

programming

program

programs) data.

claasical

carry

can

Paralex

Nodes

to indicate

Unlike

the

A

on the

plus

latency,

[1].

Paralex

obeys

associated

must

other

specification

using

sible

parallelism The

all

with

and

outThe

syntax

transforming

effects.

devices

actual

done

Paradigm

[11].

no side

multi-

and

respectively.

computation

functions

the

of data-driven

when

or interactions

as files,

The

also

and

the

to

inputs

graphical

rule”

that

execution

have state

such

[10].

paradigm

values,

Paralex

can

internal

enabling sense

starts

links,

this

is

inputs

for

of the

outgoing

program of

permitted.

rewrite

Paralex

the

number

that

with

“strict in

to outputs.

exist-

to

such,

harnessing

of as directed

fragments

gram

in

by

links.

as such

sections.

representation and

and

associated

so-called

as

of inter-

reusing

the

by “glue”

interesting

reusability”

domain

of (typed)

be thought

sup-

and facilitates

of a Paralex

some

graphical is a node

are incoming

putations

are expressed

by

programming

procedures,

flow

tolerance

unit

mapping

itself

semantics

and

The

limit,

As

puts

parallel

consists

without code.

fundamental

particular

static

by interconautomatic

in subsequent

computational The

function

fault

supported

basic

outputs.

computations

that

Programming

made

are

allows

Nodes

a multi-junction

that

execution

be constructed

of “software

Paralex

choices

as discussed

are no new

or synchroniza-

programs

and fault

Computation

The

activity

programs.

In the

2.1

C++, The

parallel

computations

for heterogeneity

there

communication

composing

sequential

including 2

In particular, for

program-

use of graphics

relations

of traditional

the

Finally,

soft ware

to explore

paradigm

flow

software

line

tolerance

and

extensive

of parallel

notation.

sequential

port

distribution,

computations

data

programs

a single

ap-

environment

for

sequential

and

of reliable

tion.

of distributed

controlling

view

combining

a graphical

parallel ing

makes

of ordinary for

an

programming

promotes

dependencies

necting

in distributed

computations,

necessary

constructs

en-

programming

support

Paralex

collages

programming.

task

fault

complexities

heterogeneity

The

quential linguistic

is the

application

is to

applications.

Paralex

prob-

support

undertaken

phases

for

debugging.

been

the

automatic

expressing

and

parallel

from

goal

and

a challenge in software

parallelism

the

all

provide

reasonable

systems.

which

encompass

and

the

project

extent

that

each of these

easy development

by distributed

The

remains

permit

the

belief

technologies

to exploit

offered

is our

to address

What

of integrating vironments

It

exist

may

matrix be “fil-

quadrants

to produce

in parallel

at the next

level. 1 The

exact

computations patibility

179

list

of programming

is determined with

C

by

languages type

and

permissible calling

convention

for

node com-

Conceptually, as regular

filters

nodes

through

sequential

of the

data

context

the

than

with

the

programs.

single

producer

process

practice, are

that

data

spare,

SunOS,

rel=4,

rev=l,

fpu,

gfxn2,

however,

all

xenia

spare,

SunOS,

rel=4,

rev=l,

fpu,

spec=13

the

fyodor

spare,

SunOS,

rel=4,

rev=l,

fpu,

gfx,

nabucco

mips,

tosca

m68020-Sun

violetta

m68020-A/UX,

turandot

RS6000,

AIX,

carmen

vax, gfx,

memory=

jago

fyodor

executed

in the

Associating

not

it also economizes

elettra

produced

processes.

of the

Subgraph

Paralex

only

on data

data filters

saves

network

buffers

necessary

rather

Any

than

tion.

primitive

graphs node

may

may

sequential

These

to

nodes

nodes

themselves.

Subgraphs

are

to

abstraction

that

understand,

programmed

also

may are

to

de-

Paralex —

programs

easier

as

be further

what

a structur-

not

only

construct

easier

using

pre-

components.

2.4

Cycle

At

the inter-node

Nodes

Any

be invoked outputs The

level, single

(in

termination

node

computation

of

the

during

number

for

explicitly

graphs

graph,

however,

execution

and

condition

as it is defined The

type) the

by the

cycle

and

function

evaluates and

erwise, are

the

as long

exactly

cycle

user

the

false,

input

outputs

has the

values

produced

around”

Paralex

appear

become

cycle

the

as an “architecture the

host

of

the

host.

name

of the

separated shown

list

(e.g.,

nary

attribute

se-

for

is

and

host

name same

for

tor

aud

a runtime

gra!uliling (’ollll)ll~ WI(II

support

are in[egratmi

next,

input

to

ecuting old

The

II)

this

ollaracterisLics

first

t II ree

graphical

]Jro-

possible dSO

frOl”ll

section

IllaC])i

we illust

through

a iIlc

011 wllicb

I IFS

ritl e

vax)

and

( allt,d

paralex

site.

is to

rllll

Tills

is iilv

value load

tlous

with

Note

raw

type,

broadcast

ate

interest

used

by

for

ex-

scheme.

must

power two

of a host

is used

balancing

algorithms

by

are

(such

its

spare, in

share not

the

binary

as the

oper-

be included

them. the

and

measured

hosts

but

distinguish

This

contain (e.g.,

A/UX ) must

The

Inapping

in

SPECand

to associate

dy-

computa-

hosts.

the

definition

systeln.

and

The

area

Iletworliing

~ould

easily

uniform.

us.

This

the

information

of Paralex

include and

the

of immedi-

ot’ gigabit site expliclt

pfirlll]t

ashosts

is supported are

dvent [17],

to

that

inforof the

u-m between

assumljt,ron

systenls

technologies

Lopology

velsion

communicant

With

be exteuded

no explicit

characteristics

current

LAN-based to

contains

communication

t hat vach pair-wise

by

( atlojl

to

are

suitable

family

family

at-

inter-

user.

a host

information SunOS,

nor

be introduced

by the

case

architecture

additional

abol]t

distributt,d

They

processor In

same

defines

to

to share

the

matching

of

assignanother

hosts

has

neither

at will

Numeric

through

the two

attributes

or nu-

presence,

names.

[13].

t]] at the site

rnatio]l

180

the

chrrracterizatioll

nalriic

is

of a bi-

Indicating

architecture

(spec)

processor

mark

is

file

fpu) name

sets of hosts

description

system

of a site

j ago

a pattern

new

processor

SPECmarks

sulllc~

of the user interface

Paralex

its

values

to select

through

minimum

mips,

line

is a comma-

the

signals

as attribute

loader

to be modified

the

Paralex

of the

spare,

integer

Paralex

allows

is possil)lv

systel)l

a host

(e.g.,

name,

to edit,

Definition

distributed

cleiined

The

a single

prO~riiJTIS

support,

an exec u-

example.,.

Site

3.1

environment

name

(e.g.,

permits

nodes

ones

the

A gral)ljlcs

a compiler.

It is, however,

~’iMalFX

graphics

MJI]I

11-

h{wl.d.

)1

(, * Point4Y-@nt ,wwlvtim of the