Limits to low-latency communication on high-speed networks

Limits to Low-Latency Communication High-Speed Networks CHANDRAMOHAN A. THEKKATH University of Washington, Seattle

The throughput

of local

ATM

networks

Other

network

few years. often

and

mented this

system

procedure

call

to several

hardware

platforms

to achieving

low

system,

Our

RPC

network We

(ATM,

impact

latency,

and

which

networks

demonstrate

and

that

controllers

RPC

times

more

than

ever

to achieving

Subject

Descriptors:

crucial

Categories

and

put/Output

to near

Devices—channels

Network

Protocols—protocol

tributed

Systems—distributed

tems]:

Systems]:

General Additional

Key

munication,

This

Design,

work Digital Intel

Authors’

WA

Permission not made of the

with

us to

respect

architecture

and

(170

pseconds

those

and

on an ATM

components

software

design

network

of next-

communication.

and

can

controller

reduce design

and

Data

Communications]

[Computer-Communication operating

sending,

Networks]:

systems;

network

In-

Networks]:

D.4.4

Dis-

[Operating

communication;

Sys-

D.4.7

[Oper-

systems

Performance networks,

Systems

level

in part

of Washington,

host-network

interfaces,

interprocess

com-

protocols

Science

Foundation

by the

Research

Computer.

is supported

University

host

of low-latency

making

Output

CCR-9200832,

and Apple

H. Levy

under

Washington Center

C. Thekkath by a Fullbright Department

and

grants

Technology External

is supported research

Research

in part award

of Computer

CCR-8619663, Center,

Science

and

by

Program,

by a fellowship

and by the INRIA. and Engineering,

98195. to copy without

or distributed

publication

Association specific

Corp.,

Corp.

differ-

allows

communication.

network

transport

Corporation

several

systems

C.2.2 [Computer-Communication

by the National

and

impleported

system

the

times

way

technology limits,

[Input/

ATM

calls,

in part

Equipment

address:

Seattle,

Phrases:

CCR-8907666,

Hewlett-Packard from

and

procedure

was supported

CCR-8703049, the

Words

remote

and

We then

with

these

to isolate

in the

Design—dwtributed

Measurement,

call

us

stand

Management—message and

latency.

SPARCstation)

controller,

reduced

C.2.4

is

high-speed

software.

low-latency

applications,

in a

throughput

we designed

low

Comparing

network

processor

B.4.2

of magnitude

of newer

in the communication

allow

and controllers;

Organization

Terms:

the

network-imposed truly

effects

of new

of Ethernets.

increased

software,

and

runtime

still

architecture;

Communications

ating

that

new-generation

small-packet

and

designs

hosts),

that

order

than

at providing

(DECstation

substantially

5000/200

the bandwidth than

communications.

and Ethernet).

network,

rather

system-level

targeted

user-level

achieves

DECstation

system FDDI,

the and

the

hardware

of alternative

e.g.,

the kernel

system,

using

generation

both

example, greater

of yet another

latency

cross-machine

of influences,

and controllers

increase

lowered

examines

on low-latency,

the performance

cache

paper

For

of magnitude

a bandwidth

remote

ent networks

increasing.

is an order

systems,

This

a number

a new

explore

promise

concern.

and HENRY M. LEVY

is rapidly

rings

in distributed

technologies

To evaluate

networks

token

technologies

However,

of primary

network

area

FDDI

on

and

fee all or part

for direct its

for Computing

date

of this

commercial

appear,

Machinery.

and

material

is granted

advantage,

the ACM

notice

is given

To copy otherwise,

that

provided copyright

copying

or to republish,

that notice

the copies

is by permission requires

are

and the title of the

a fee and/or

permission.

!3 1993 ACM

0734-2071/93/0500-0179 ACM

Transactions

$01.50 on Computer

Systems,

Vol. 11, No. 2, May 1993, Pages 179-203.

180

.

C. A, Thekkath and H M. Levy

1. IINTRODUCTION In

modern

computer

systems, slow communication links and softwarehave imposed a stiff penalty for cross-machine commucosts limit the extent to which a network of computers

processing overheads nication. These high can

be used

related

Several way

as a solution

problems recent

in which

for

either

structuring-related

developments,

networks

improvement over Ethernet ment to 1 gigabit per second

however,

have

the

potential

such as FDDI (ATM), offer

expected

the bandwidth

Taken

together,

communication

lying

on

these

system

technologies

systems

other

These

operating system system mechanisms reduce the software

the

that

should

can

exploit

of networked

nodes

[2], and B-ISDN [34] a 10-fold bandwidth

could

be

improve-

have already given us power, with 100 MIPS

processors

will

be capable

of

provide. research has led to extremely low [4, 22, 27, 33]. Such low-overhead latency that has typically limited

primitives.

be possible, example, consider

[ 11]. In a cluster idle

the year.

new networks

of operating

would not otherwise As a motivating server

within that

—Performance-oriented overhead operating mechanisms greatly the performance

to change

[23]. Another order-of-magnitude seems close behind.

—New processor technologies and RISC architectures an order-of-magnitude improvement in processing using

performance-

are used:

—New local area network technologies, using Asynchronous Transfer Mode

performance

or

in applications.

the

allow novel design

us to produce distribution

that

of a distributed-memory

workstations,

paging

much

than

faster

low-latency schemes

to spare paging

to

memory disk.

A

distributed server could manage the idle memory in the network; on a page fault, the page fault handler would communicate with the server and transfer pages to or from remote nodes. The effectiveness of this scheme is highly influenced by the communication latency between client and server. As another example, reduced-latency communication would greatly facilitate the use of networked workstations as a loosely coupled multiprocessor or shared virtual-memory system [7, 21]. Such configurations would have significant cost performance, flexibility, and scalability benefits over tightly coupled systems or over dedicated message-based multiprocessors (e.g., cubes). These systems are characterized by frequent exchanges of small synchronization packets and data transfers—attributes that are well served by low communication latencies. The common thread in these examples is that latency and CPU overhead are at least as critical as network throughput. With the advent of very high-bandwidth networks, the throughput problem relatively more tractable, leaving latency as the preeminent issue. 1.1 Paper

Organization

The objectives (1) To evaluate ACM

TransactIons

of this

and Goals paper

are threefold:

the fundamental, on Computer

Systems,

underlying Vol.

11. No. 2, May

message 1993,

costs for small-packet

is

Low-Latency Communication communication

on new-technology

on High-Speed

networks,

when

Networks

compared

181

. with

Ether-

net technology. (2)

To describe and evaluate the the face of different networks,

design of a new low-latency RPC controllers, and architectures.

(3) To draw lessons for network and ence. We believe (as do others) controllers are often poorly tributed systems, particularly Overall,

our

generation interface.

objective

is

to

take

system-level

of modern disRPC systems.

view

of RPC

FDDI,

connected

by an Ethernet

and

ATM

applications;

(2) general

new-

local

area

networks,

and

on the

combinations, connected by SparcStation

I

network.

We have used Remote Procedure model for several reasons: (1) it known [5, 27]; communication

on

in particular the software and hardware designed, implemented, and measured a

RPC on several different network and workstation on the MIPS R3000-based DECstation 5000/200

Ethernet,

cantly;

a

in

design based on our experinew generation of network

matched to the demands in the face of low-overhead

networks, examining To do this, we have

low-latency specifically,

controller that the

system

Call (RPC) as the remote is a dominant paradigm

techniques

for achieving

communications for distributed

good performance

are well

(3) it is user-to-user, in contrast to kernel-to-kernel memory that tends to underestimate communication latencies signifi-

and (4) it is close in spirit

to other

communication

models,

such

as V

our observations widely applicable. Our objective is [81 or CSP [17], making not to attempt a completely new RPC design, but rather to assess the latency impact

of modern

required

the

networks,

design

and

controllers,

processors,

implementation

and software.

of a new

RPC

system,

To do this because

existing high-performance RPC systems (such as SRC RPC [27]) do not run on the experimental hardware base we required. The paper is organized around the objectives listed previously. Section 2 describes the network cost for cross-machine ines the RPC design ment

of our RPC

of the controller nication existing

principles

system

Complex appear

that we used and details the minimum in these environments. Section 3 exam-

that

make

on a variety

has a significant

overheads. machines

environments communication

low-latency

RPC feasible.

of controllers

influence controllers

indicates

on efficient and

to add to the overall

how

cross-machine

host/controller overhead

Measurethe design commu-

interfaces

on

in two ways—latency

inherent in the controller and latency required by the host software to service the controller. Finally, Section 4 examines in greater detail the implications of low-latency communication for networks, controllers, and operating system abstractions. For example, we show the latency effects interrupt handling, and data transfer techniques. Our

experience

demonstrates

that

new-generation

of cache management, processor

technology

and performance-oriented software design can reduce RPC times for small packets to near networkand architecture-imposed limits. For example, using DECstation 5000/200 workstations on an ATM network, we achieve a simple user-to-user that RPC

round-trip RPC in 170 microseconds. Our experiments software overhead contributed by marshaling, stubs, ACM

Transactions

on Computer

Systems,

suggest and the

Vol. 11, No. 2, May 1993

182

C. A, Thekkath and H. M, Levy

.

packet exchange protocol need not be the bottleneck to low-latency remote communication on modern microprocessors. However, as software overheads decrease, network controller design becomes more crucial to achieving truly low-latency communication. 2. LOWER

BOUNDS

This

section

from

two important

FOR CROSS-MACHINE

evaluates

the

overhead

sources:

COMMUNICATION

added

to cross-machine

(1) the network

hardware

(the

communications link) and (2) the controller, memory, This allows us to examine our RPC software (the stubs, protocol,

and

runtime

support)

costs for cross-machine network and processor nents

may

change;

relative

to the

by isolating

the

lower-bound

components,

to extend

networks,

and

other

and

with

ware

and

explain

those

second

measurements,

to compare

Ethernet. our

In the

first

different

experimental

past

we briefly

testbed

2.1 Overview of the Networking subsection

capabilities Ethernet. which controller

is

summarizes

and

The [1].

network

Ethernet

accessed

on

However,

networks

controller

DECstations

the

[ 18, 27]. newer

However, technology

networks

with

characterize

measurement

each

our hardmethodology.

controller and network communication.

controller

we used and the particular

used to access each network.

is a 10 Mbit/see

our

the perfor-

Environment

the various

of the specific

Given different of these compo-

to examine

Then we analyze our performance results and discuss issues that specifically impact latency in cross-machine

This

message-passing

we can see how

high-speed

following

and the

and CPU interfaces. RPC packet exchange

communication on our hardware. technologies, the relative importance

mance of each layer scales with technology change. Similar measurements have been reported in the we wish

communication controller

CSMA/CD and

local

SparcStations

is packaged

area by

network, a LANCE

differently

on the

two

machines. On the DECstations, the controller cannot do DMA to or from host memory; instead, it contains a 128-Kbyte on-board packet buffer memory into which the host places a correctly formatted packet for transmission [ 14]. Similarly, the controller places incoming packets in its buffer memory for the host to copy. In the case of the SparcStation, the controller uses DMA to transfer

data

to and from

host memory.

In this

case, a cache flush

is done on receives to remove old data from the cache. packets are described by special descriptors initialized by send) or the controller (on receive). Descriptors are kept the SparcStations while they are in the special on-board

operation

On both machines, either the host (on in host memory on packet memory on

the DECstations. Two message sizes were used in our experiments, a minimum-sized (60 bytes) send and receive, and a maximum-sized (1514 bytes) send and receive. FDDI.

FDDI

is a 100-Mbit/sec

fiber

token

tion by the DEC FDDI controller. Like the the FDDI controller cannot perform DMA ACM

TransactIons

on Computer

Systems,

Vol

11, No

ring,

accessed

DECstation from host

2, May

1993.

on the DECsta-

Ethernet memory

controller, on message

Low-Latency Communication transmission;

instead,

ever, the FDDI

it relies

controller

on an on-board

can perform

on reception of a packet from share descriptors as described an unloaded

private

token

passing

ment,

token-passing

Packet with

FDDI

is kept

DMA

ring

delay

packet

transfers

Networks buffer

with

two

would

have

1514 bytes

hosts.

the

overhead

in a more

to be added

were

chosen

How-

to host memory

and host software Ethernet. We used

Thus,

minimum;

183

.

memory.

directly

the network. The controller above for the DECstation

to an absolute

sizes of 60 and

on High-Speed

due

realistic

to the

to facilitate

overall

direct

to

environlatency.

comparison

the Ethernet.

ATM.

ATM

munication exchanged order

(Asynchronous

standard

Transfer

Mode)

used to implement

between

entities

of a few tens

in fixed-length

of bytes.

is an international

B-ISDN.

An ATM

In an ATM

parcels network

called

telecom-

network,

data

cells, usually

typically

consists

is

on the

of a set of

hosts connected by a mesh of switches that form the network. In an ATM network, user-level data is segmented into cells, routed, and then reassembled at the destination The particular ATM

using header information we used has 140-Mbit/sec

contained in the cells. fiber optic links and

cell

sizes of 53 bytes and is accessed using FORE Systems’ ATM controller [16]. The controllers on the two DECstation hosts were directly connected without going through a switch; thus there is no switch delay. Unlike the Ethernet and FDDI controllers, the ATM controller uses two FIFOS, one for transmit and

the

simply

other

for

receive.

reads/writes

tions

that

cells

arrive

choosing

correspond in

how

The

complete to the

the

receive

often

controller ATM

FIFOS. FIFO.

it should

has

cells The The

no DMA

by accessing host

is notified

host

be interrupted.

facilities. certain

has

via interrupt

considerable

Further,

The

host

memory

locawhen

flexibility

the controller

in

does not

provide any segmentation or reassembly of cells; that is the responsibility of the host software. Each ATM cell carries a payload of 44 bytes; in addition, there are 9 bytes of ATM and segmentation-related headers and trailers. In our experiment

we chose packet

well

close

as being

enough

sizes that to

the

were an integral

Ethernet

and

FDDI

number packet

of cells as sizes

for

link,

we

comparison. 2.2 The Testbed In order

and Measurement

to isolate

built a minimal on the network.

the performance

stand-alone testbed, There is no operating

software is executing workstations (either through processor

an isolated rated

at

Methodology of the controller

which simply sends and receives packets system intervention since only minimal

on each machine. two DE Citations network. 18.5

The

SPECmarks,

and the network

The testbed hardware consists of two or two SparcStations) connected

DECstation and

the

uses

a 25 MHz

SparcStation

MIPS

I uses

R3000 a Spare

processor rated at 24.4 SPECmarks. The DECstations were connected in turn to an Ethernet, an FDDI ring, and an ATM network. The SparcStations were connected to an Ethernet. The DECstation network devices are connected on the 25 MHz TURBOChannel [13] while SparcStations use the 25 MHz SBUS [29]. We measured

the performance ACM

of each configuration

Transactions

on Computer

Systems,

in sending

a source

Vol. 11, No. 2, May 1993.

184

C. A, Thekkath and H, M.

.

packet from one node to the Packets are sent and received data

over the host

are enabled, packet dedicated

the

While

fashion

other from

bus is included

so both

arrival.

Levy

and

the

is generally

by disabling

in receiving a packet in response. memory, so the cost of moving the

in our measurements.

sender

it

and host

receiving

possible

network

Network

hosts

to

access

interrupts,

provide

periodic

interrupts.

These

on the DECstations (about SparcStations (once every

were

on the

Wire.

the propagation

This

time

clock

set to provide

chips

interrupts

Measurements trip

can be broken

is the transmission

because

were

time

it is negligible

on

network

in

a

time-sharing

once every 244 microseconds) and 625 microseconds). No significant

involved in fielding a timer interrupt. least 10,000 successful repetitions. The component costs for the round —Time

the

conventional

access will involve interrupt-processing overheads. Both the DECstation and the SparcStation have

interrupts

are interrupted

that

can

at 4096

Hz

1600 Hz on the processing is averaged

over

at

up as follows:

of the packet.

for the length

We ignore

of cable

we are

using. —Controller the sending has made the data

Latency. controller the data

This is the sum of two times: (1) the time taken by to begin data transfer to the network once the host available

at the receiving

to it, and (2) the delay

controller

and the time

between

the arrival

it is available

of

to the host.

—Control / Data Transfer. Data has to be moved at least once over the host bus between host memory and the network. Some of our controllers use DMA incurs transfer describe the

to move

no

data

data

over

transfer

the

host

overhead.

bus to the

However,

network;

the

overhead because it has to use special the location of the data to the controller.

actual

Latency

time

to do the

data

transfer

CPU

thus incurs

the

CPU

a control

memory descriptors to With such controllers,

is captured

in

the

Controller

item.

—Vectoring the packet

the Interrupt. arrival interrupt

The overhead —Interrupt

involved

Service.

some essential

On the receive side, host to the interrupt handler

is a function On taking

controller-specific

of the CPU

an interrupt, bookkeeping

software must vector in the device driver.

architecture.

host

software

before

the

must interrupt

perform can be

dismissed. 2.3

Performance

To determine

Analysls the latency

of the controller

between a pair of hosts controller’s status registers of data, expected simply

we ran separate

experiments

host polled the indicated arrival

a new transmission was begun. In the cases where the to copy data to the controller’s memory before transmission, programmed

giving it any data. data for the host ACM

itself,

with interrupts disabled. Each in a loop. As soon as the register

Transactions

the controller Similarly, to copy,

on Computer

to start

the data

transfer,

without

host was the host actually

when the controller indicated the arrival of new the host ignored the data and began the next

Systems,

Vol.

11, No. 2, May

1993.

Low-Latency Communication transfer.

In

started.

addition,

In these

by the host

descriptors

circumstances

per round

trip.

were very

on High-Speed prefilled

before

few machine

The time

required

Networks the

data

instructions

to execute

185

.

transfer

are executed

these

as well

as the

time This

spent on the wire was subtracted from the total measured round trip. method gives satisfactory results in most cases but has the disadvantage

that

it does not account

This

artifact

multicell

for any pipelining

is particularly

packets

transmission

are exchanged.

of data

transmission instance, in

visible

between

that

in the Typically,

its

the controller

a controller

internal

buffer

the

perform.

controller

chip

and

between host (or on-board memory) sending a multicelled packet through

host can fill the FIFO with a cell while from the FIFO onto the network.

might

case of our ATM

when

can overlap

network,

the

with

the

and the chip itself. the ATM controller,

the controller

sends

For the

the previous

cell

Table I shows the cost of round-trip message exchanges on the host/network combinations described above. A few points of clarification are in order here. First, which

in the

case of the

perform

flushing

the

overhead. from

DMA cache

While

higher-level

is kept

software

data. Second,

arrived

only

two interrupts. row

in the named

Most of the microseconds have

cache flushes

network,

it interrupted

Sum

of

in

the

necessary

locations,

means

this

Table

reported

include

In addition, each

is therefore

is the

if data that

the

cost of accessing

I does not

experiments,

cost of

Service

are not strictly

packets.

Components

the

Interrupt

the host only when

in our

controller,

receives,

pay a performance

multicell

Thus,

SparcStation

sum

this

the cost

the controller

a complete

packet

round

incurs

trip

a lower

of the

bound.

rows

above

it.

time, the sum of our component measurements is within 1–9 (about 270) of the observed round-trip time. The only exception

case of the

transfers

2.3.1

that

The performance

overestimated

ATM

network

in sending

the round-trip

time

of the amount

of overlap

Latency

and

on the

packets,

12%. The most

between

where likely

we

cause

the host memory–FIFO

transfer. It

Latency.

to Time

multicell

by about

and the FIFO-network

Throughput

Controller

is included

eventually

FIFO.

is an underestimate data

DMA

the

on packet

in uncached-memory

will

so that

had

and

memory

the

and reassembling

was programmed

is in the

controller

to host

in the case of the ATM

for segmenting

The

after

it is true

the network

FDDI

directly

is

interesting

Wire.

to

For small

compare

packets,

the

this

ratio

is 0.4 on

the DECstation Ethernet, 0.8 on the SparcStation, 7 for the FDDI controller, and 3.2 for the ATM network. For larger packets of approximately 1514 bytes, these ratios are respectively 0.02, 0.04, 0.9, and 1.0. While these numbers are specific to the controllers we use, we believe they are indicative of a trend: The

tant

bandwidths

packet

difference

exchange

between

are improving times

on

throughput

packets is the goal, then exchange on our DECstation

dramatically

Ethernet,

we can Ethernet

and

FDDI,

latency.

while and

If

latencies

ATM

show

low

latency

achieve a round-trip in only 253 pseconds;

ACM TransactIons

on Computer

Systems,

are not. the

for

impor-

small

60-byte message FDDI on similar Vol. 11, No. 2, May 1993,

186

.

C. A. Thekkath and H. M. Levy Table

I.

Hardware-Level

Round-Trip

Component Ethernet (DIK) 60/60 1514/1514 Time

on the Wue

Controller

Latency

Control/Data Vecmrmg

Transfer the Intemupt

Interrupt

Service

Sum of Components

Round Tnp Time

Measured

hardware,

despite

single

cell

lower

bound

checking

in ,useconds

ATM (DEC) 53/53 1537/1537

2442

115

2442

13

245

6

176

53

89

103

97

230

16

161

40

600

6

6

40

253

17

45s

25

25

12

12

25

25

25

25

26

26

42

42

92

140

9

20

257

3146

264

2605

267

893

73

840

253

3137

263

2611

263

894

73

7.$6

10-fold

bandwidth

is

in

about

because

whether

Times

51

which

transfers

Exchange

115

its

the same operation,

Packet

Round-Trip Time (f(seconds) Packet Size in bytes (serrdk’ecv) FDD1 (DEC) Etbcrnct (Spare) 60/60 1514/1514 60/60 1514/1514

we

longer.

4~0

73

have

the cell is part

advantage, The ATM

~seconds,

ignored

takes network

However

switching

of a larger

message

263

pseconds

is capable

this

delays, that

for

of doing

is an optimistic and

the

needs

cost

of

fragmenta-

tion/reassembly. On the other hand, the high bandwidth of ATM and FDDI is significant for large packets. For example, latency for a 1514-byte packet on the DECstation Ethernet

is 4.2 times

packets

even

larger

worse than

than 1514

ATM

and

3.5 times

worse

bytes,

the

Ethernet

situation

than

FDDI.

For

is relatively

worse, because FDDI will require fewer packet transmissions. The ATM comparison is slightly more complex; as pointed out earlier, we have ignored ATM segmentation and reassembly costs in arriving at the figures in Table I. The particular software implementation of the segmentation/reassembly we use requires

about

11 ~seconds

per cell.

If this

is included,

then

a 1537-byte

packet (29 cells) takes about 1065 ~seconds, which is about 1.2 times the FDDI time. The situation further improves in favor of FDDI for packet sizes beyond

this

2.3.2

limit.

Controller

Structure

and

Latency.

As noted

earlier,

both

the

DEC-

station and the SparcStation use the same Ethernet controller. However, their performance is not identical. Recall that on the SparcStation the controller uses DMA transfers on the host bus. The overhead for this is included in Controller Latency; the Control / Data Transfer costs include

only

the

cost

of the

instructions

to program

troller is able to overlap the data transfer the network. Thus the sum of Controller Transfer is relatively unchanged by the

the

controller.

The

con-

over the bus with the transfer on Latency and Control/ Data packet size. In contrast, the DEC-

station controller incurs a heavy latency overhead because the host first has to copy the data over the bus before the controller can begin the data transfer. Consequently, for larger packets, the SparcStation controller performance is likely to be better even though for small packets both have comparable round-trip times. Controllers that use on-board packet buffers instead of DMA or FIFO will generally incur this limitation. However, controllers with DMA or FIFO are not without problems; we will defer a more detailed discussion ACM

until

TransactIons

Section on Computer

4. Systems,

Vol.

11, NrI

2, May

1993

Low-Latency Communication It

is

also

interesting

to compare Both

various controllers. interrupt-handling

times.

the The

on High-Speed

the

Networks

interrupt-servicing

Ethernet

latency

controllers

SparcStation

figure

.

have

is slightly

187 on the

comparable

higher

because

a cache flush operation is included in the cost due to the DMA transfer. except the ATM controller use descriptor-based interfaces between the and the interface. than

Programming

the simpler

FIFO

this

interface

requires

many

on the ATM.

more

The FDDI

All host

accesses to memory controller

is the most

complex one to service and is 7– 10 times as expensive as the ATM. While figure is for a specific pair of controllers, we believe that FIFO-based trollers

will

in general

reduce

Our experiments with lowered latency, simple the more

traditional

of controllers

programming

overhead.

low-level message passing seem to suggest that for FIFO-based controllers have some advantages over

types

of controllers.

on user-to-user

However,

cross-machine

we describe the impact

a higher-level of controllers

message-passing

performance,

user-level

to

with

concerned

additional

ultimately

it is the impact

communication

the next section, RPC and explore be

this con-

that

communication in this context. cross-machine

issues

like

is crucial.

system Unlike

communication

protection

and

In

based on low-level has

input

packet

demultiplexing. 3, THE DESIGN,

LOW-LATENCY The

IMPLEMENTATION, RPC

performance

system

of

the

hardware

are two of the three

If the

performance

poor,

this

twofold. systems

of the

can easily

and

message-passing

components third

dominate

Second,

of the

low-level

of user-level

component, the overall

we wish

used

by higher-level

software

ture

of the network

controller.

the

high-level

how

might

system,

is

section

is

of this

exists for building RPC in the previous section

some of the low-latency these

latency.

RPC

cost. The purpose

to outline and

communication

interact

techniques

with

the

struc-

Design and Implementation

The RPC system tion

OF A

First, we wish to show that the technology that are so efficient that the costs described

are significant.

3.1

AND PERFORMANCE

[30]

[27] built

has demonstrated

optimization. Our where performance tions

system

system

differs

(1) We differ performed: promising RPC. (2) Our

system

for the DEC the

SRC Firefly

latency-reducing

multiprocessor

effect

of a large

RPC system closely follows the SRC is a goal, many details in the structure

must

be

from

SRC RPC in several

dictated

by

in the way stubs we use a scheme protection differs

kernel

in the way but ACM

hardware

of

design; however, of a communica-

environment.

Thus,

our

respects:

are organized that minimizes

between

(3) We do not use UDP/IP,

the

workstanumber

and

in which

and the copying user

control

instead

use network

Transactions

on Computer

way marshaling is costs without com-

spaces

as is done

transfer

is done.

datagrams Systems,

in SRC

directly.

Vol. 11, No. 2, May 1993.

188

.

These

C. A. Thekkath and H M. Levy

differences

Our

are described

low-latency

ning

the Ultrix

The

system

RPC operating

is

in assembly RPC

system

is entirely

during

face previously common byte

the

binding

by the

to impact

Multipacket

SunOS other

that

is linked

process,

server. transfers

code. The

the user’s

address

on different

imports

the

inter-

optimized

machines

for the

with

the same

code path

so as not

is reduced

by ensuring

that only the client, but not the server, swaps byte order when needed. The underlying communication in the RPC relies on a simple request response

packet

however,

there

beyond what data copying,

exchange

similar

are

fundamental

three

is required for con trol transfer,

to that

described aspects

dis-

the need to program

specially

overhead

or

high-quality

by a separate

case. Byte-swapping

systems

networked

client

between

run-

SunOS.

operating

into

the

We have

RPCS are handled

the common

not felt

generate

5000

I running

that is integrated into the kernel. clients and servers can be placed

case [4] of single-packet

order.

and

in C; we have

subsections.

DECstation

impacting

our compilers

runtime

exported

on the

Ultrix

component

space and another component Like other RPC systems, machines;

the without

because

has a runtime

in the following

and on the SparcStation

into

machines

language,

detail

is prototype

system

integrated

executing on these tributed services. Our implementation

in more

system

in Section

to RPC

and

2. In addition,

that

add

overhead

a simple message exchange: marshaling and and protocol processing. Our system achieves

its low latency by optimizing are somewhat interrelated, and

each of these areas. The optimizations we consider them in turn in the following

subsections. 3.1.1 simple specific

Marshaling procedural and depend

procedures

marshal

and

Data

interface

Copying.

for the

client

on the particular the

call

RPC

stubs

and

server.

service

arguments

into

function and

out

create Stubs

the

illusion

of a

are application-

that

is invoked.

of the

message

These packet.

Even in highly optimized RPC systems such as SRC RPC, marshaling time is significant. Marshaling overhead depends on the size and complexity of the arguments. Typically the arguments are simple—integer, booleans, or bytes; more involved data structures are used less frequently [4]; therefore simple byte copying is sufficient. Related to the cost of marshaling

is the cost of making

the network

packet

available to the controller in a form suitable for transmission. A complete transmission packet contains network and protocol headers, which must be constructed by the operating system, and user-level message text, which is assembled by the application and its runtime system. There are several strategies for assembling the packet for transmission, with the cost depending on the capability of the controller and the level of protection required in the kernel. With a controller that does scatter-gather DMA over the bus, like the SparcStation controller, the data can be first marshaled in host memory by the host (in one or more locations) and then moved over the bus by the controller. This is the scheme we use on the SparcStations—the data packet comes from user space, and the header is taken from kernel space, a techACM

Transactions

on Computer

Systems,

Vol. 11, No. 2, May

1993

Low-Latency Communication nique way

commonly

used

SparcStation

and from

in such

DMA

a fixed

range

on High-Speed

an environment

is architected, of kernel

the

virtual

is more

expensive

than

our RPC design uses mapping One general drawback with

addresses.

accesses packet

of the data and

Likewise,

again given

performs

This

the

when

the

controller with

copying

it

of the user

over

to the

approach is to relax kernel/user user space and allow the user copies

to one. However,

there

controller

ment-marshaling into

the

To do this

kernel

generate

and

optimized

rather

executed.

to the

Code for

network.

or a FIFO, into

two

technique

Another

retains

synthesis

specific

which

has been

situations

all the

used

argu-

space,

is then in the

to achieve

and

buffer into number of

address

code on the fly,

the

a buffer

copies.

that

in the user’s

two

of the

the cost of two copies. call arguments, we perform

than

we synthesize

routines

the data requires

is an alternative

in the kernel

conventional.

pieces

conthis

at least

the pieces

memory

to

a threshold,

protection and to map the packet direct access. This reduces the

benefits of the protection without incurring In an effort to minimize copying of the

only mapping

accessible to the primitives for

it involves

the

on-board

marshaling

due to the

necessitates

the host builds

transfers

special

189

DMA

sizes below

is that

over the bus: once when

technique

kernel

for packet

only selectively. a DMA scheme

a controller

straightforward

copying

.

[18, 25]. However, controller

the user’s data buffer into kernel addresses that are troller. Since the cost of using SunOS virtual-memory mapping

Networks

as is linked

past

high

to

perfor-

mance [20, 22]. Our focus is slightly different: we are more concerned with avoiding the copy cost than with generating extremely efficient code for a special situation. At bind time, when the client imports the server’s interface, the client calls into

the

directly

kernel

pointers

1 As only

procedure instructions. each input

a template

of the

simple-valued

to bytes.

procedure. involves

with

supports

Using

this

mentioned

assignments

is nothing

earlier,

more

than

passed

procedure.

as words, kernel

assembling

the task

the

procedure

contains This

right

simple

synthesized

code to check

procedure

and a

of primitive the validity

has the benefit

and reply are known in advance, the can be avoided if arguments and results

the

and

of synthesizing

sequence

approach

kernel

bytes,

a marshaling

is typically

Thus,

The

halfwords,

synthesizes

marshaling

copying.

at runtime.

since the sizes of the request general multipacket code path a single network packet. The kernel then installs

the

the

and byte

marshaling

such

template

The marshaling argument

types

as a system

of that

more fit in

call

for

subsequent use by this specific client. Thus, the stubs linked into the user’s address space do not do marshaling; they merely serve as wrappers to trap into the kernel where the marshaling is done. A client RPC sees a regular system call interface with approach has the benefit required

without

1 We do not yet have

all the usual of performing

compromising

a template

the

compiler, ACM

protection rules the minimum

safety

of a “firewall”

and so our kernel

Transactions

that go with it. This amount of copying

stubs

on Computer

between

are currently

Systems,

hand

the

user

generated.

Vol. 11, No. 2, May 1993.

190

.

C. A. Thekkath and H, M, Levy

and the kernel, or the user and the network controller. impose the overhead of probing the validity of pointers copied,

but

this

In addition RPC

system

is not a significant scheme,

provides

another

interface

into

a specialized

case. Instead

of calling

which

client calls into a fixed-kernel tors. Each descriptor describes type—IN descriptors space ing

and the

done using cally

handles

controller’s

exports

memory

compromising

this

several

cases, our general

procedure,

the

The kernel can use these directly between the user

thereby

eliminating

Unmarshaling

on the

server

above.

A server

interface

with

received packets. We have used kernel-level

common

for a more

kernel-marshaling

or FIFO,

procedures

the kernel

of the

is designed

supported. arguments

safety.

general-purpose

provides

RPCS.

most

that

scheme does data can be

entry point, passing an array of data descripa primitive data type (including its parameter

or OUT) that is directly to marshal and unmarshal

cost without

server

cost for most

to this

Our before

as described

with

different

a generic

types

template

marshaling

that

of

extra

is

typi-

arguments.

applies

and unmarshaling

copyside

The

to all types

with

of

the DECsta-

tion Ethernets, but a hybrid scheme is used with the FDDI controller. On transmissions the usual scheme is used, but on receives, the controller’s DMA engine copies the data over the host bus and hands it to the kernel. The kernel unmarshals the data either by copying or by virtual-memory mapping if the

alignments

controller. driver

performs

could

read

required, fact,

are

In this

suitable.

case, before

reassembly

only

the

A similar the

RPC

from

the

network

accessed

method, but reassembly is such ATM that we did not feel justified

is used

receives

in a page-aligned

header

and if not let the RPC layer

in a non-ATM

approach

layer

buffer.

FIFO

and

the data

via FIFOS,

this

and this

the the

Furthermore,

determine

unmarshal

a common in making

with

the packet,

ATM device

the driver

if reassembly from

would

the FIFO.

is In

be the preferred

frequent operation optimization.

on the

3.1.2 Control Transfer. Researchers have reported that context switching causes a significant portion of the overhead in RPC [27]; in addition, there is a substantial impact on processor performance due to cache misses after a context

switch

switching and finally the

[24].

the client switching

server

out—can

An

RPC

call

typically

requires

four

context

out, switching the server in, switching the client back in. Two of these—switching be

overlapped

with

the

transmission

the

switches:

server out, the client or

of the

packet.

Systems with high-performance RPC usually have lightweight processes that can be context switched at low cost, but unless there is more work to do in the client and the server, or no work elsewhere, a process context switch usually occurs. Both the DECstation that can be significant context

switches,

our

and the SparcStation have context-switching times to the latency of a small packet. To reduce the cost of RPC

system

defers

blocking

the

client

thread

on the

call. Instead, the client spin-waits for a short period before it is blocked on the sleep queue. If the service response time is very small, the reply from the server is received before the client’s spin-waiting period has expired. When ACM

TransactIons

on Computer

Systems,

Vol

11, No

2, May

1993

Low-Latency Communication there

is no other

no penalty done,

work

to be done,

to spinning

the

caller

the caller

spins

for

i.e., when

round-trip

time

are

be to block

a short

greater

than

otherwise. using

available.

the

time

relative

An estimate

an estimate.

hint,

In general,

are processes

control

trip

technique

queue

switch

a thread

This

leads

often

with

In

contrast,

packet

directly

is done

from

layer.

performance.

the lowest

directly

within

the

Protocol

Processing.

layer

trip

is

and to spin statically

response

by

times

for latency

as

if there

protocol

handler,

layering.

The

the incoming packet on the interrupts are then used to to a modular we

to the destination

interrupt

scheme

round

either

past

from

wake

in that

is

to be

turn.

arises to queue software

of control

there quantum

penalty

throughput

traditional approach is for each layer input queue of the next higher layer; unacceptable

191

work

current

expected

be obtained

their

overhead

to the

by using

trades

waiting

transfer

is empty, is useful

round-robin

if the

could

or dynamically,

this

on the run

Additional

extension

to the context

of the round

a user-supplied

to its

spinning

related

there

.

of the server ensures that the process can be improved if estimates of the

without

some threshold

Networks

queue

When

A simple

caller

the run

indefinitely.

before blocking. In most cases the low response time is never put to sleep. This approach would

on High-Speed

try

approach

but

to dispatch

the

process.

yielding

This

a path

dispatch

of very

low

latency. 3.1.3 nate

communication

overhead

of protocol

processing

costs if general-purpose

The

protocols

are used

the usual case of a homogeneous and low service response times, employed to optimize There are typically protocol

like

specialized procedure

call semantics

Several multiple

that

protocol

aspects layers

increasing

layers

provides

to provide

in RPC

a basic a close

systems:

unreliable

a transport-level transport,

approximation

and

a

of conventional

for RPC.

of protocols

of protocol

the number

In

environment, with frequent remote requests special-purpose protocols can be effectively

the latency. two protocol

UDP/IP RPC

can domifor RPC.

contribute

tend

to RPC latency.

to add to the overhead

of context

switches

As mentioned of RPC

and increasing

above,

in two

the number

ways: of data

copies between layers. Further, the primary cost of using protocols such as UDP/IP is the cost of checksums. Calculating checksums in the absence of hardware support involves manipulating a packet as a byte stream; this can nullify any advantage gained by the controller or host processor in assembling the packet using scatter-gather or wordlength operations. For efficient RPC implementations, then, the checksum must be either calculated in hardware themselves transmission

or made

optional.

to the former medium

Most

conventional

approach.

The latter

is reliable

and that

protocol approach

the transport

formats

do not lend

presupposes protocol

that

is used

the only

for routing, not for reliability. These factors argue for the use of a simpler protocol and hardware checksumming. Our RPC uses a simple and efficient unreliable transport protocol and relies on a specialized RPC protocol to provide robustness, which is similar to ACM

Transactions

on Computer

Systems,

Vol. 11, No. 2, May 1993.

192

.

C. A. Thekkath and H. M, Levy

that used by the protocols reflects the

SRC or Xerox [5] RPC systems. In general, the choice of a set of assumptions about the location of clients and

servers

and

servers

are expected

area networks heavy

loads

error

characteristics

have good packet due to overruns.

the same LAN,

raw

by the controller. controller. If the

of the

to be on the same local

network.

loss characteristics,

In the case when

network

datagrams

Typically

area network.

clients

dropping

packets

the RPC destination

are used with

and

Furthermore,

local at only is within

the checksum

provided

An erroneous packet is simply dropped by the receiving target is not on the local network, it is a straightforward

extension to use UDP/IP without checksums. The choice can be made at bind time when the client/server connect is established, and the marshaling code can be generated In addition protocol

the appropriate imposed

per se adds to the latency.

to provide achieve

to include

to the overhead

a natural

high

The primary

set of semantics

performance,

header.

by transport-level reminiscent

our protocol

on the critical fast-path of the code. Under normal error-free operation,

protocols,

purpose of simple

was implemented the server’s

of RPC

the RPC protocols

procedure

is

call.

To

so as not to intrude

response

to a call from

the

client acknowledges it at the client end. Similarly the next call from the client acknowledges the previous response. The state of an RPC is maintained by the client and server using a “call/response” record. Call records are preallocated at the client at bind time. These contain a header, most of whose fields do not change the latency

with

side and contain In

order

packets

each call.

at call time. to

header recover

on the client

These

Similarly,

are therefore

response

information from

from

dropped

and the server

prefilled

records a previous

packets,

call that

our

RPC

side. On the client

blocked for the duration of the call, retransmission the client address space, as with the original call.

so as to minimize

are retained

at the server can be reused.

transport

buffers

side, since the client

is

proceeds from the data in Thus no latency is added

due to buffering when the call is first transmitted. On the server side, the reply is nonblocking; hence a copy of the data has to be made before returning from the kernel. The copy is overlapped with the transmission on the network. Once again no latency is added to the reply path. One alternative to this would be to use Copy-on-Write, which would not affect latency but can potentially 3.2

save buffer

RPC Performance

space for large

multipacket

RPCS.

Measurements

This section examines the performance of our RPC implementation. Our goal is to show that structuring techniques, such as those we have used, yield an RPC software system so effective that the hardware costs shown in Table I are significant. We gather the data to support our analysis of controller design in the next section. Table II shows the time in microseconds for RPC calls on the various platforms.

Two

procedures

takes two integer two arguments—an an integer ACM

result.

Transactions

called

Minus

and

MaxArg

were

timed.

Minus

arguments and returns an integer result. MaxArg takes integer parameter and a variable-length array, returning The exact

on Computer

sizes of the packets

Systems,

Vol. 11, No, 2, May

exchanged 1993

varies

depending

Low-Latency Communication Table II

II.

Allocation

on High-Speed

of RPC Time

.

193

in Microseconds

I Activity

Networks

(sccont

fltherr h4inus

Fim7cr

Ethcrrrl

(Spwc

MaxArg

Minus

Maxar

Minus

DEC) Maxarg

Fl)r

All

DEC)

Minus

=MaxArg

Chen[ Call Controller Latency

28

145

59

137

46

173

25

159

26

27

45

54

48

82

8

44

Time

58

1221

58

1221

4

122

3

88

25

25

27

27

56

70

17

20

Server Packet Receipt

39

470

59

169

42

42

29

347

Server Reply

27

2-1

46

46

36

36

25

25

Controller

25

25

44

44

~ 49

82

8

44

57

57

57

57

5

5

3

3

26

27

56 ’29

17

17

49

27 49

56

29

26 29

340

2052

471

1831

371

340

2070

496

1997

380

on the WUC for Call

Intcrntpt

Time

Handhng

Latency

on the Wue

Interrupt Chcnt

on Server

for Reply

Handling Reply

hcelpl

Total Attributed Measured

on CIIcn!

T]mc

Time

on the network 48 bytes

type.

on FDDI,

Minus

causes

and 53 bytes

60 bytes

single-user

rows

which

showed

are explained

below:

—Client Call. This is the total time out a call packet. It consists of five up the argument time to validate dure,

—Time

the packet on the

based

for setting

Latency.

getting

MaxArg

Wire

This for

Call.

major

parts:

locate

the cost

2

675

on the Ethernet,

transmits

to the server

on the ATM. on the three

The reply networks.

required on the client side for sending major components: the time for setting

and

of this

server

to do a transmit.

represents

the wire.

the

This

An estimate

controller’s

is computed

of the packet

latency

in

as in Section

2.

transmission

time

after

This is a sum of the times to vector the handler in the device driver, the time in the interrupt

is dismissed.

This is the time spent on the server machine before it is handed to user code. It consists

the cost of examining correct

unmarshaling

170

of the network.

and to return

—Server Call Receipt. the call packet arrives

The

figure

Handling on Server. interrupt to the interrupt

the handler,

693

less variance.

up the controller

to and from

on the bandwidth

—Interrupt network

29 776

to the system call, the time to perform a kernel entry, the the arguments, the time spent in the marshaling proce-

and the time

—Controller

29

164

both in single-user and in multiuser modes. The different; the times reported in the tables are the

measurements

The various

29

697

to be exchanged

on the ATM.

1514 bytes on Ethernet and FDDI, and 1537 bytes from the server is 60, 48, and 53 bytes respectively Measurements were made times were not significantly

s +

and

the packet

dispatch

copying/mapping component

varies

the data

and kernel packet,

into

depending

the

data

and

the

server’s

on the

when of two

structures time

spent

address

capabilities

to in

space. of the

controller. —Server needed

Reply. This is quite similar to Client Call. This includes the time to call the correct server procedure and to set up the results for the

system call, the time spent in the kernel in checking the time to set up the controller for transmit. ACM

Transactions

on Computer

Systems,

the

arguments,

and

Vol. 11, No. 2, May 1993.

194

.

C. A. Thekkathand

H. M. Levy

—Client ceipt

Reply Receipt. This is the counterpart and takes place on the client side.

—Total

Attributed

we have been measurements A few points

Time.

This

is the total

able to attribute or by estimating about

Table

of the

of the components.

to the various from functional

II are in order

Server

Call

It is the time

activities, either specifications.

here.

First,

Re-

though

by direct

the SparcSta-

tion and the DECstationv 5000 have similar CPU performance and use identical Ethernet controllers, the software cost for doing comparable tasks varies. In general, our experience has been that SunOS extracts a higher penalty than Ultrix. We believe this is primarily due to the richer SunOS virtual-memory architecture rather than to the SparcStation’s architectural peculiarities ATM

such

network,

packet

into

Similarly,

as register

windows.

Second,

29 cells are sent to the server.

29 cells at the sender

is included

the cost of reassembling

in the case of MaxArg

The cost of segmenting in the row entitled

the cells is included

on the the user

Client

in the Server

Call. Packet

Receipt row. In the case of single-cell transmission, the segmentation and reassembly code is completely bypassed. Finally, we exploited the flexible interrupt structure of the ATM controller to ensure that in the normal case only the last cell in a multicell the Total Attributed Time Section

2, in all cases except

in the range

one, the difference

of 1–8Y0. The only

on the ATM,

where

tables,

exception

for the reasons

overestimated the total Overall, as we can Ethernet

packet caused a host interrupt. In comparing with the Measured Time we note that as in

the

experimental

error,

is in the case of a multicell

packet

mentioned

time required. see by comparing

latency

for

is within

the

earlier

the

small

in Section

FDDI

Minus

and

RPC

the

2, we have DECstation

request

on FDDI

is

12% higher than for Ethernet. On the other hand, as expected, the much higher bandwidth on FDDI becomes evident for the larger packet sizes; the MaxArg RPC request takes nearly three times longer on Ethernet than on FDDI. Comparing the performance on the ATM network with the others, we see how effective a simple controller design is for reducing latency for small packets. is a factor

However,

for larger

in reducing

tion

and reassembly

are

being

built

[12,

packets,

performance. either 32].

the software ATM

in hardware It

would

fragmentation/reassembly

controllers

that

be interesting

characteristics of these controllers with FORE controller we use. As noted earlier in Section 2, for small

perform

or by dedicated the

to

simpler

packets,

fragmenta-

on-board

processors

compare approach

the taken

the cost of “doing

latency in the

business”

(i.e., controller latency) on the network has increased with FDDI. However, compared to the low-level software latency imposed by the packet exchange, the higher-level RPC functions (stubs, context switching, and protocol overhead) add only 87 pseconds for Ethernet, pseconds for ATM. Higher-level RPC functions cost more relating to the DMA interface ACM TransactIons

controller/host the high-level on Computer

117 ~seconds on

interface. First, cost of dispatching

Systems,

Vol

11, No. 2, May

FDDI

for

for two

FDDI, reasons,

and

97

both

because of the more complex the server process and data 1993.

Low-Latency management is greater Ethernet, the code path

Communication

for the required

on High-Speed

Networks

DEC FDDI. Second, for both for RPC for these controllers

different inherent

from that used for a simple packet exchange. overhead added by RPC. However, the overhead

of FDDI

than

for the

controller/host While

Ethernet

because

of some of the

the absolute

increase

traps and context However, the bulk

contributed

switching that of the overhead

main

factors

that

use of preallocation, exploiting

simple

Our

experience

that

the

ments

and

with

also determine in Tables

Our experiments, ordinary

network

latency-impacting

AND NETWORK

RPC

on a variety

has

latency

latency.

experienced the

fact

In this

and the network

that

influence

included

controllers

and no DMA,

tradeoffs

features the

make

minimum

Transfer. the

between

allow

by an RPC.

Our

a faster

achievable.

posed by the lower the communication

access

measure-

network

we discuss

that

are capable

us to examine

these

The

the data

details

capabilities

it difficult

efficient

network

does not

the

effects

of

I/ O

(2) the overhead of transferring in more detail below.

on

indicates on

on latency.

different

sor moves the data to and from the bus, transfer primitive). These issues are (1) the

4.1.1 Data

the

DESIGN

controller

to restrict Excessive

to the user.

and

number

on the Certain

of data

unnecessary

each of these

system

movements data

movement

on Computer

Systems,

of

to one, im-

performance of between con-

troller types and copying costs. Table III summarizes the number of copies that the controller, and the user need to perform, respectively, for each type of device: Transactions

bus vary

combinations

levels of the system can lead to bad overall system. We discuss below the interaction

ACM

There

using a block the device and

We discuss

data

controller.

the

types.

controllers that (i.e., the proces-

usually without cost of servicing

of moving

of the

of scatter-gather

some of the essential

are basically two issues to be considered in choosing between provide DMA and those that support programmed 1/0 (PIO)

depending

for

different

section,

over-

are (1) the

of controllers

a significant Similarly,

the

transmission,

optimizing

reinforce

which

performance

and network

processors.

II

DMA,

is to keep

controller,

a lower

4.1 DMA versus Programmed

reduction

system’s

overheads.

the

I and

imply

the controller

DMA,

the

controller

for

to the cost of features like

(4) the speed of the host

low-latency

of the

this

computation

of

case,

communication

necessarily both

of the

is comparable

RPC adds about 130% is due to architectural

to the

FOR CONTROLLER

design

cross-machine

contribute

peculiarities

common

4. IMPLICATIONS

protocols

characteristics

may not scale with processor speeds. can be reduced with increasing CPU

(2) overlapping

the

and

Thus, there is an is more in the case

by RPC functions

as a percentage, of this overhead

speeds. A significant factor in achieving heads of memory copying to a minimum.

(3)

FDDI and is slightly

interface.

both ATM and Ethernet, low-level messages. Part

The

195

.

the kernel, PIO, DMA,

Vol. 11, No. 2, May 1993.

196

.

C. A. Thekkathand Table

III.

H. M, Levy

Number

Number

of Copies

Controller

Types

of Copies (Device/KemelAJser)

PIO (KM)

Fragment

for Various

I PIO

I DMA

I DMA (S-G)

Send Header Data

otlm 0/1/0

Ollp

I/lp

lPIO

0/1/1

1/0/1

lPIO

OIIP OIIP

0/1/0

1/1/0

1/0/0

0/1/1

Ip/1

lPP

Receive Header Data

and DMA with scatter-gather. The some form of kernel-level marshaling FRPC

running

represents

on the ATM

scatter-gather

Certain

implicit

controller

assumptions

made

the cost of copies

the

internal

either by using that with PIO, into

multiple

copied)

user

use a contiguous DMA,

the

data

over

address

spaces,

the

while

with

and

that

on

tiguous

locations

of allowable (or

a few)

in memory.

location(s),

Thus, and

cost

can

user

scatter

Most

data

DMA

S-G

be made

data

and

negligible

with

(instead

of

the DMA

can

scatter-gather

small

controller

amounts can

goes to the correct

of

perform

destination.

the number of times data is moved on and the host memory. With PIO and to keep

the

number

scatter-gather

the user

of copies

to one

DMA this would not be possible, in could be located in multiple nonconcontrollers

for each segment

then

so that that

the

First,

Second, we assume be reliably mapped

arbitrarily

DMA,

below.

the network

is mapped

memory,

we assume

With data

size requirements segments.

this

DMA,

it is possible

compromising protection. because application-level

minimum

between

incoming

without general, have

are clarified

occur

of transferring

so that

marshaling

table

might

in kernel

one would like to minimize bus between the network

kernel-level

in the

Finally,

is capable

bus

The column

or some similar technique. memory, or FIFO, cannot

set of addresses.

controller

its FIFOS.

Typically

to the header

demultiplexing

Ideally, the host

that

buffer.

video RAMs the on-board

to be adjacent

with

DMA,

we are ignoring controller’s

column PIO (KM) represents PIO with as described in Section 3, for instance,

will

have

have

the

to marshal

controller

(e.g.,

and a maximum the data

move

it.

LANCE) number into

While

one it

is

possible to build controllers to overcome this restriction. Using several small segments to gather data comes at a price, because the controller has to set up multiple DMA transfers. A similar situation is true on the receiving side as well. As shown in Table III, using PIO with kernel-level marshaling allows the data-copying cost to be kept to the minimum possible. However, one aspect of copying that is not captured in the table is the different rate at which data is

z the Autonet Sept. ACM

controller

being

built

at DEC

SRC;

personal

communication,

1991. Transactions

on Computer

Systems,

Vol. 11, No

2, May

1993

Charles

P. Thacker,

Low-Latency Communication

on High-Speed

Networks

.

197

moved over the bus for PIO and DMA. Typically, word-at-a-time PIO accesses over the bus are slower than block DMA accesses; this is the case on both the DECstation TURBOChannel bus and the SparcStation SBUS. While PIO versus

DMA

point

is of limited

beyond

which

is required

concern

PIO will

to touch

each byte

across the bus (for instance, more

efficient

4.1.2 and

Cache

and

with DMA

current might

TLB

the total

bookkeeping

cache lines

corresponding

IV,

of cache

flushes

flush

Newer

it

adequate

support

data

transfers

If the cache does not snoop

by the

to the data are

that

a serious

the situation

for

DMA

and

The cache flush required

processor/cache

coherence

without

controllers.

costs

designs In

penalty

interruptcontroller-

the time

architecture

taken

to flush

memory

the table

from

architec-

suggests,

because

cache flush instructions, by destroying locality. this

absence

the

As is evident

on current

than

recognize the

The total essential

was transferred.

is even worse

[15].

the

cost is simply host

in addition to the costs of executing additional flushes have a negative impact on performance ory

moving it is usually

subsystem with This subsection

controller-initiated

due to cache effects.

instructions

cache

above,

subsystem,

overhead.

the

However,

in software),

size.

cost of our DMA

sum

to execute

tures.

the processor than

blocks could be left incoherent as a result of the requiring cache flushes. If the cache is write-back,

interrupt-handling

Table

other

need to be purged before a DMA operation from memory. the contribution of the cache flush cost as a percentage of

cost is the

related

a certain

detail. costs outlined

of overhead

on 1/0 operations, cache DMA operation to memory,

handling

for reasons

a checksum

is a break-even

unless

interaction of the memory overall penalty than PIO.

and processor

dirty entries may Table IV shows

data

there

Thus,

While Table 111 seems to indicate that PIO can perform the same number of copies, very often

architectures the extract a heavier

be a source

packets, DMA.

Effects.

DMA

the memory

could

of the

beyond

describes this effect in more In addition to the copying from

short than

to generate

to use DMA

scatter-gather

for

be slower

problem

and

of snooping

cache

provide

caches,

mem-

the

usual

“solution” to this problem is to allocate buffers temporarily in uncached memory before they are copied to user space. This approach, used in the stock SunOS

Ethernet

driver,

either

incurs

an extra

benefit of cached accesses to frequently used Another cost of DMA is the manipulation necessary. On packet arrival, the controller however, there is generally no way to guarantee the

correct

option this

destination

of either remapping

address

remapping

space.

Thus

or performing

can require

an explicit

copy

overhead

data. of page

tables

or loses that

the

is often

stores the data on a page; that the page is mapped into the

an extra

kernel

is faced

with

the

copy. On a multiprocessor,

and expensive

TLB

coherency

opera-

tion [6]. To summarize our experience with the various controllers, we believe that DMA capabilities without adequate support from the cache and memory subsystem can be bad for performance in modern RISC processors. We also believe

that

controllers

that

have

ACM

simpler

Transactions

interfaces

on Computer

to the

Systems,

host

have

the

Vol. 11, No. 2, May 1993.

198

C. A. Thekkathand

.

H. M Levy Table

IV.

Cache

Flush

Cost

Received Packet Size in bytes FDDI (DEC) Ethernet (Spare) 1514 60 60 1514 Total Interrupt Cache

Percentage

,

potential simple 4.2

Overhead

for reducing

overheads.

controller/host

are two

controller.

common buffer

in memory.

other

ways

and

14

48

15

43

next

subsection

argues

data

a range

between

(possibly

the

data

to the

overhead

FIFO

user.

We have

shall

therefore

and

costs of different controller types. with the two types of controllers

interface

such

component.

as that

found

processors

Our in the

objective ATM

shows

the

cost

and

already

of servicing

through a copy or a remapping here is to compare the interfaces

the to be

descriptors

for transmits

and

discussed

in an the

restrict

ourselves

mentioned

above

to indi-

the

and (2) the controllerresearch has studied

[3, 26], and so we shall is with

interface such as that found in the Ethernet We reproduce some of the measurements table

host

overhead can be significantly reduced cost is composed of two components:

costs on RISC

second

the

and the host share

(1) the CPU-dependent cost of vectoring the interrupt dependent cost of servicing the interrupt. Previous only

use of

all) of host memory

is to use a simple

cate that the interrupt-handling by using a FIFO. Interrupt-handling

interrupt-vectoring

for the

host access it directly. A significant overhead to the processor system is the cost of servicing the

movement

interrupt-handling Our experiments

70 30

and have the controller

getting

of data

7

of transferring

alternative

receives and have the interfacing the controller interrupt

46

10

Memory Interface

used as a packet

role

21

3

The

One way is to designate The

21

interfaces.

Host / Controller

There

Time

Flush Time

‘

to

compare

a more

elaborate

or the FDDI from Section

interrupt

and

examine

a simple

controllers. 2 in Table

transferring

FIFO

descriptor V. The the

data

operation, as appropriate. Since our intent and not the network, we have ignored the

reassembly overhead on the ATM controller. As the table indicates, for transferring small

amounts

of data

to the user,

the overhead of the FIFO-based controller (10 ~seconds) is less than half that of the best-case descriptor-based controller (24 ~seconds for the DEC Ethernet). For larger packets, the balance is in favor of controllers that allow the kernel to perform page-mapping operations, because the copying cost dominates the interrupt-handling cost. The ability to map data into user spaces at low cost is one alternative to kernel-level marshaling with PIO, which retains the benefits of protection and reduced data movement without the need to synthesize code. A typical limitation with FIFO-based controllers, such as the FORE ATM, is that there is no easy way to map the memory in a protected manner simultaneously into multiple user spaces. ACM

TransactIons

on Computer

Systems,

Vol.

11, No

2, May

1993

Low-Latency Communication on High-Speed Networks Table r

V.

Interrupt-Handling

(DEC) 1514

Ethernet

60

199

.

Cost

Ethernet (Spare) 60 1514

FDDI (DEC) 60 1514

Intenupt Time

13

13

21

21

46

10

11

201

9

39

10

70 10

5

Copy/Mapin

5

138

24

214

30

60

56

80

10

148

Time

Toral

In contrast, to support

with

descriptor-based

address

mapping

packet

irrespective

memory,

of DMA

it is possible

support.

“

ATM (DEC) 1537 53

in principle

However,

typically,

on descriptor-based PIO controllers, the existence of a small amount of on-board buffer memory makes it difficult to provide address-mapping support, because that memory is a scarce resource that must be managed sparingly. For instance, the DECstation’s Ethernet controller has only 128 Kbytes of buffer memory to be used for both send and receive buffers. Mapping pages of the buffer memory into user space would be costly, because the

smallest

Reducing dropped tively

units

that

the number packets

during

managing

the

can

be

individually

of available

buffers

periods

scarce

of high

buffer

mapped

this

way could

load.

On the

resource

results

are

4-Kbyte

pages.

lead to delays other

in the

hand, kernel

due to

conservamaking

extra copy of the data from user space into the packet buffer. With a trivial amount of controller hardware support, it is possible

an

to solve

the protection granularity problem, providing a larger number of individually protected buffers in controller memory. The basic idea is to populate only a fraction of each virtual page that refers to controller memory. As a concrete example, Figure 2-Kbyte

we consider an alternative design to a DEC Ethernet 1 shows the sketch of the design. Controller memory buffers,

us to have write troller

each of which

64 buffers

directly

in our

to controller

ignores

will

hold

128-Kbyte memory

the high-order

the physical page number the TLB. This has the

bit

controller. without

of the

Our larger based

experience simple

host/controller

FIFO-based

packets packet

with

would buffer.

same controllers.

packet; To allow

sacrificing

page-offset

this

would

user

processes

protection,

and

as

allow

the

concatenates

to con-

it with

(PFN) field of each physical address presented by effect of causing each 2-Kbyte physical page of

controller memory to be doubly mapped half of a 4-Kbyte process virtual page. while

an Ethernet

controller. is organized

are

served

it might

To our knowledge,

interfaces on board is the VMP-NAB in the absence of memory system

both

the top half

interfaces

controllers

be better Thus,

into

leads

ideally

with

a more

be beneficial the only

suited

us to believe for

small

conventional

to support

controller

[19]. Our support,

and the bottom

that

experiments DMA may

both

that

packets, descriptor-

forms

has multiple

on the host

also suggest that incur the cost of

additional copies and/or cache-flushing overheads. In such cases, it would be advantageous to use PIO with a descriptor-based packet memory that the host CPU can either copy or map into user space. This could be done either by providing enough buffers or with hardware support (e.g., as mentioned above). The decision to copy or map in will depend on whether the processor is required to touch every byte of the packet or not. For instance, if it is ACM

Transactions

on Computer

Systems,

Vol. 11, No. 2, May 1993

200

.

C. A. Thekkath and H, M, Levy

. : :.

. Processor Virtual

: .

Address Space

: :

: :. : :

Processor

Fig.

1.

Address

Controller

mappmgfor

Memory

Controller

Ethernet

buffers.

necessary to calculate a software checksum, then an integrated copy and checksum loop (as proposed in [9] and [ 10]) would suggest that mapping is of limited benefit. This consideration will be even more important with future memory transfers. initiated adequate 4.3

subsystem designs If the processor

that will provide support for I/O-initiated does need to mediate the transfer, then

data moves would be of benefit support for cache consistency.

Network

if

the

memory

system

data 1/’0-

provides

Types

Compared to an Ethernet, a high-speed token ring like FDDI offers greater bandwidth. However, token rings have a latency that increases with the number of stations on the network. Consider a network where the offered load is quite small. That is, on the average, only a few nodes have data to transmit at a given time. As stations are added to the network, the Ethernet latency remains practically constant while the latency of a token ring will increase linearly due to the delay introduced by each node in reinserting tokens. This implies that even on a lightly loaded, moderately-sized token network, achieving low-latency cross-machine communication is difficult. As an example, if each station introduces a one-microsecond token rotation delay, a network of 100 stations would make it infeasible to provide lowlatency communication. As load is increased, both the Ethernet and the FDDI token ring will experience greater Iatencies, with the FDDI reaching an asymptotic value [32]. Thus, on balance, it appears that low-latency communications are not well served by a token ring, despite high bandwidth. We ACM

TransactIons

on Computer

Systems,

Vol

11, No

2, May

1993

Low-Latency

Communication

on High-Speed

Networks

201

.

should point out that in our experiments with RPC, token rotation latency was not a problem because we used a private ring with two nodes on it. However, if we added more nodes to the ring, we would expect to see a degradation

in the latency

ATM-style the

networks

delay

instance,

of RPC.

that

fragment

and

of

fragmenting

and

reassembly

our

experience

indicates

that

interleave of

with

packets

have

medium-sized software

to incur

packets.

reassembly,

For

latency

begins to be impacted with packet sizes in the range of 1500 bytes. It is not immediately clear how adding fragmentation and reassembly support in hardware overall

or in an on-board latency,

5. SUMMARY Modern

processor,

even though

as provided

in [12] and [31],

fragmentation/reassembly

will

affect

is fast.

AND CONCLUSIONS

distributed

systems

require

both

high

throughput

and

low

latency.

While faster processors help to improve both throughput and latency, it is high throughput, and not low latency, that has been the target of most newer networks In this cations

and controllers. paper

we have explored

on new-generation

implemented designs

a low-latency

in

addition

avenues

networks RPC

to our

own.

for achieving

(specifically, system

Using

using

newer

low-latency

FDDI

techniques RISC

communi-

and ATM).

We have

from

processors

previous

and

perfor-

mance-oriented software structures, our system achieves small-packet, round-trip, user-to-user RPC times of 170 pseconds on ATM, 340–496 ~seconds on Ethernet, and 380 ~seconds on FDDI. Our RPC system demonstrates that it is possible to build an RPC system whose overhead is only 1.5 times the bare

hardware

cost of communication

for small

packets.

Our experiments indicate that controllers play an increasingly crucial role in determining the overall latency in cross-machine communications and can often

be the

bottleneck.

However,

we believe

controller design that can provide lowered niques that achieve excellent performance. us to believe

that

FIFO-based

packets

and

that

DMA-

hidden trollers

costs that

and

network

that

there

are alternatives

latency, facilitating software Specifically, our experience interfaces

descriptor-based

are well controllers

suited may

depending on the memory system architecture. provide multiple host interfaces appear to be

alternative to current factor for performance.

for have

to techleads small many

Hybrid conan attractive

designs. Of course, the network itself is an important For instance, both of our high-throughput networks

have some peculiarities that could affect latency of packets: e.g., the token rotation latency in FDDI network and the fragmentation and reassembly in the ATM Finally,

network. we believe

that

with

careful

design

at all levels

tion system, communications latencies can be substantially entirely new approaches and applications for distributed

of the communicareduced, systems.

enabling

ACKNOWLEDGMENTS

The authors thank Ed Lazowska for his numerous patient readings of earlier drafts of this paper and his comments, which added greatly to its clarity of ACM

Transactions

on Computer

Systems,

Vol. 11, No. 2, May 1993

202

.

C. A. Thekkathand

presentation.

Thanks

suggestions of the

are also

for improving

FDDI

helping

H.M.

Levy

due

to Tom

the paper.

controller

from

us understand

DEC,

many

Anderson

Finally, who

for

we wish

gave

his

comments

to thank

so willingly

and

the designers

of their

time

in

of its details.

REFERENCES

1.ADVANCED

MICRO

Advanced 2. Ross,

F. E.

Devices,

FDDI—A

3. ANDERSON, T. E., architecture

Am7990

DEVICES.

Micro

Sunnyvale, tutorial.

LEVY,

H.

and operating

on Architectural

Support

Local

Area

Calif.,

1986.

IEEE

Network

Cornmun.

M.,

BERSHAD,

system

design.

Mug.

B. N.,

Controller

24, 5 (May

1986),

10-17.

AND LAZOWSBA,

E. D.

The

In Proceedings

for Programming

Languages

Trans.

5. BIRRELL,

Comput.

A.

Comput.

D.,

Syst.

6. BLACK,

D.,

tural

B.

39-59.

1984),

R.,

GOLUB,

J.

D.,

and

Implementing HILL,

approach.

In

for Programming

interaction

Operating

Systems

Lightweight

of

Conference (Apr.

remote

1991).

procedure

call.

1990).

1 (Feb.

A software

Support

8, 1 (Feb.

AND NELSON,

2,

RASHID,

consistency: York,

Syst.

(LANCE).

of the 4th International

4. BERSHAD, B., ANDERSON, T., LAZOWSKA, E., AND LEVY, H. ACM

for Ethernet

remote

C., AND BARON,

Proceedings

Languages

procedure R.

Translation

of the 3rd

and

calls.

ACM

Operating

Systems

LEVY,

M.,

ACM

Trams

lookaside

buffer

Conference (Apr.

on Architec-

1989).

ACM,

New

113–122.

7. CHASE, J. S., AMADOR, Amber

system:

ACM

12th

LAZOWSKA,E. D.,

F. G.,

Parallel

programming

Symposium

on

on a network

Operatzng

Systems

H.

AND LITTLEFIELD,

of multiprocessors. Principles

(Dec.

In 1989).

R. J.

The

Proceedings

of the

ACM,

York,

New

147-158. 8

CHERITON, (Apr.

D. R

1984),

The

V kernel:

A software

9. CLARK, D. D., AND TENNENHOUSE, protocols. tures

In

and

10. CLARK,

Protocols IEEE

11. COMER, model.

1990).

Commun.

B. S.

SIGCOMM New

(Sept.

Proceedings

ACM,

New

ROMKEY,

Msg.

27,

J.

York,

6 (June

A new

A host-network

for

distributed

design 1990

interface

USENIX

An

1, 2

for a new generation

for

ATM.

Architectures

analysis

of

Architec-

of TCP

systems:

Conference

architecture

on Communications

In

and

The

(June

remote

1990),

memory

127–135.

Proceedings

Protocols

processing

of the

(Sept.

1991).

1991 ACM,

307-315. TURBOChannel

CORPORATION.

PMADD-AA

Specification,

Rev.

1.2. Workstation

EQUIPMENT SYSTEMS. C. A. R.

Systems

CORPORATION.

TCA-1OO

Pittsburgh,

HOARE,

Softw.

on Communications

H.

23–36.

CORPORATION.

17

IEEE

200–208.

1989),

EQUIPMENT

Systems,

considerations

J., AND SALWEN,

14. DIGITAL DIGITAL

systems.

Symposium

EQUIPMENT

16, FORE

distributed

Architectural

13. DIGITAL

15

for

SIGCOMM

of the Summer

Symposium

York,

D. L.

of the 1990

D., AND GRIFFIOEN, In

12. DAVIE,

Proceedings

D. D., JACOBSON, V.,

overhead.

base

19-42.

Alpha

Hardware

Engineering, ATM

Ethernet

1991.

Module

Functional

1990.

Architecture

TURBOchannel

Speczficatzon.

TurboChannel Reference

Computer

Man ual,

Interface,

1992.

User’s

Manual.

FORE

Pa. 1992 Communicating

sequential

processes.

Comm un

ACM

21,

8 (Aug.

1978),

666-677. 18, JOHNSON, D. B., AND ZWAENEPOEL, Rep. 19

COMP

KANAKIA, mance

TR91-152,

H.,

mm

network

Symposmm

Dept.

CHERITON,

W.

The

of Computer D. R.

communications

The

Peregrine Science,

VMP

high

network

for multiprocessors.

on Comm unzcatLons

Architectures

performance

Rice Univ., adapter

In

and

board

Proceedings

Protocols

RPC

system.

Tech,

1991. (NAB):

of the 1988

(Aug.

1988).

ACM,

High-perforSIGCOMM New

York,

175-187. 20

KEPPEL, ings

guages 21

and

A portable 4th

Operating

Syst.

Transactions

interface

International

LI, K., AND HUDAK, Comput.

ACM

D.

of the

Systems P.

Memory

7, 4 (Nov. on Computer

for

on-the-fly

Conference

1989),

(Apr.

1991).

coherence

instruction

on Architectural ACM,

New

in shared

York,

virtual

321-359.

Systems,

Vol.

11, No. 2, May

space

modification.

Support

1993

for

In

Programming

ProceedLan-

86-95. memory

systems.

ACM

Trans.

Low-Latency Communication 22. MASSALIN,

H., AND Pu,

ings

of the 12th

York,

191-201.

23. METCALFE, computer

R.

M.,

J.

C.,

Proceedings Languages

AND BORG,

and

Compzd.

Proceedings

J. K.

input/output

R.

Ethernet:

effect (Apr.

Newsletter

31. BRENDAN,

aren’t 1990

IEEE C.,

INC.

operating

systems

USENIX

Conference

New

switching

for

local

on

performance.

In

New

cache

Support

York,

for

Programming

77-84.

of a capability-based getting

Performance

results.

Systems

SBUS Specification

tures

Protocols

and J. N.

Proceedings

Trans.

TRAW,

In

(June

operating

faster

as fast

1990),

247-256.

of Firefly

RPC.

Performance

B.O. Sun

S.,

Comput.

37, 8 (Aug.

AND SMITH,

Proceedings

of the 1991

(Sept.

1991).

A timed-token

ring

of the 7th IEEE

ACM, local

Conference

J.

M.

1988), A

system.

as hardware?

ACM

Evaluation

Microsystems,

Trans.

In

Comput.

Cooperative, Inc.,

1990.

Mountain

View,

New area

E. H.,

JR.

Firefly:

Sympos~um

York,

317–325.

network

and

on Local

A multiprocessor

909–920.

high-performance

SIGCOMM

host

interface

on Communications

its

Computer

performance

Networks

for

ATM

Architec-

characteristics.

(Feb.

1982).

IEEE,

In New

50-56.

33. VAN RENESSE, R., VAN STAVEREN, H., AND TANENBAUM, distributed 34. MINZER,

Received

Proceed-

ACM,

1-17.

benchmark

networks.

Mug.

In

1989).

1990.

workstation.

York,

switches

ACM,

30. THACISSR, C. P., STEWART, L. C., AND SATTERTHWAITE,

32. ULM,

packet

on Architectural

The design

kernel.

(Dec.

395–404.

context

1991).

A. S.

Synthesis

203

.

289-299.

Why

1990),

of

Networks

Principles

Distributed 1976),

Conference

D., AND BURROWS, M.

1 (Feb.

in the

Systems

19, 7 (July

The

Systems

of the Summer

29. SUN MICROSYSTEMS, Calif.,

A.

Operating

27. SCHROEDER, M. 28. SPEC.

D.

ACM

International

J. 29, 4 (1986),

8,

and

on Operating

S. J., AND TANENBAUM,

26. OUSTERHOUT,

Syst.

Commun.

of the 4th

25. MULLENDER,

Threads

Symposium

AND BOGGS,

networks.

24. MOGUL,

C.

ACM

on High-Speed

operating S. E.

27, 9 (Sept.

July

1991;

system.

Broadband 1989),

revised

Sofku—Pratt. ISDN

17-24,

and

Exp.

asynchronous

A. S.

The performance

19, 3 (Mar. transfer

1989), mode

of the Amoeba

223-234. (ATM).

IEEE

Commun.

57.

November

ACM

1992;

accepted

Transactions

January

on Computer

1993

Systems,

Vol. 11, No. 2, May 1993.