Gigabit switch using free-space and parallel optical data links for a PCI ...

10:45am-i 1:OQam WL2

Gigabit Switch Using b'rec-Space and Parallel Optical Data Links for a PCI-Bascd Workstation Cluster J .Ekman, P.(:liandrarnarii, l'.Gni, X.Wang, F.Kiainilev, ECE Dept., University ofl)eliiwarc,l40 Ilvans IIall, Newark, DE 197 16, ph (302) 831-3278, fax (302) X3 1-3321 M.Christenseti, M.Haney, l'.Milojkovic IiCIi Dept. George Mason University, Fairfax, VA 22030 K.l)riscoll, RVanvoorst, Y.Lin, J.Nohava, J.A.Cox Iloneywcll Technology Center, Minneapolis, MN 554 I 8 Istroductiuii: Coinniunic;itiun requirements in high-pcrhirlnaliot~, I i x d l e l computirig system continue io iilcreiisc as the processing notles within these system gain Iirncessint: c;qiahility. To support these growing comrnunication requirements, system atcliitectiirc changes arc necclud. 'Ihe nse o i switched neiworks rather than bus-based systems ;uid the incorporation of optical iiitci-connects x c among pioposetl solutions to increase overall systeitl pcrhmancc. IJndcl. the VIVACE progl'arn, we combine both of thcse approaches to demoiislrate i: switclied nctwork of 12-Gbis raw data hmdwidth irsiiig a 4 'l'bitis biscctioii h;indwidtli Froe-Spaco Optically liiterctnmccted (1:SOI) switch. Optical Network 1nterZ:ice Card: The optical-intei connect IJased VIVACE network is ;icccssed by coiqitite nodes ihrougli the w e of' iui electrical netwnrk interface card (NIC) which provides cusiotii VIVACE iirotiiciil conversion irk addition to tlic necessary electrical aild optical collversions. This interlace ha~.tlwarois dosigncd based o i i re-coniiguiable, liigli-lieiibrnlarico Xilinx Virtox E'l'(;As coupled with high-speed scdizer/rlcscr ializel' c~mpotiaritsa t i d 12-channel parallcl fibcrribbon i n t e r k c devices. Our approach dlows uiiniuial ircchiteotural chinigos to target new I/O technologios iis they cmcrge. Inte~facec;ii-tls ai'e currcntly being developcd under tlic VIVACE prognini using this iirchitectu1.e wliicli are cornpatilile with the latest 64-bitlh6MIlz P(:l stiilidad. '1'0 re~liicerisk and iiicI.case final system l~erlbiiiiaiice,the hardware is tieing dcvclopcd incrementally. As sncli, it stand-aloiie version cif tlic inierliice card has heen built and tested as ~.epottedin [2]. Figurc I shows ii photo of tliis h o a d and a scope triicc 01 a b o d - t d i o ~ ~ rlink d through the Iibcr ribbon (not shown) operating ai I GHz.

lcigure 1. a) Photograph of stand-alone version of interlace card b) Oscilloscopc trace of data link at 1 GHz

The custom protocol Ibr the VIVACE system is designed to liilly utilize the 1iartlw:ire while niaintaitliiig the ability of the hosts to comtnnnica.te using ii Message Passing lriterliice (MPI). Built into the interhce card is the functionality to fully exercise and monitor the switch. l'liis is inipottant for the clemonstration of the scalability cif the VIVACE hardware since the PCI interkm itself can not p v i d e data fitst enough to fully utilize the handwidth of the switch. Thc interfbcc card perforins R U M A reads from host memory across the PCI bus atter receiving a transmission request ;id,iri accordance with an eager or :I non-cage1 transmission protocol, fnnnats the data and transmits it over the fiber ribbon to thc swiicli. Similarly, incoming messagcs tire re-forniatted for transmission across the PCI bus and wl.iltcn into host ineiory. Using the oil-chip incinoiy of the Vittex FPCiA, data being transmitted to the switch can be huffered for enough time to ensure that an enor in transmission requiring data to bo re-sent will not require data to be taken out of'host incnioiy ii second time.

0-7803-5947-W00/$10.0002000 IEEE

494

I -

il)

Transmit to Switch

I

IJ)

Figtire 2. a) Trarisfer of data irom host applicatiim to switch b) Layout of I'CI interfacc card Tlie PCI interface card develolmieiit is curretitly underway and test results will be presented at the conference. Some of the design aspects for this c a d being addressed iire tcstiiig and analysis of data transmissions across the 64hit/66MIlz P(:I bus and OS driver ilevelopmeiit. A VMETRO PCl bus analyzer card is being used in conjutictioii with a re-configurable PCI card from Nallatech with Xiliiix's RcalPCI 64-hit66Mllz I'CI core to do iiiitial testing and analysis of' PCI bus transactions. Windows N I ' drivcrs arc hcing tleveloped for the 1'CI intcrface card using NuMega software. Tlie teslbed fix this work and future hardware demonstrations is a IIELL Precision 620 workstation which has two 64-bit/66MIIz 1'CI slots. 'l'he VIVACF, hardware and application tlernonstratiori is based on ii multi-node compute system in close'physical proxiniity, hut the use of i i parallcl fiber interconnect lietweeii host NLCs itnd the swilch MCM does not make this ii constraint. This interface consists or a 12-cllannel electrical-to-optic;il VCSEL-hased drivcr morlule md a 12.. channel opticiil-to-electrical photodetector iliotlule developcd by I-loneywell Technology Center. These components liave been tested and vcrificd at gleater thw I Gbps p c r link. Link-clror tcstiiig of these modules has also been carried out on pi-ototype NIC hardware yielding better tliati 10 - I 6 bit-omor rates. Free-Space Optical Switch: At the center of the VIVACE system is a switch MCM consisting of sixteen CMOS ASlCs each with 21 GaAs optoelectronic (OE) iiiI.;iy solder-buiiip hontletl to it. The GaAs OD components are 36 x 36 rnonolithically integraled VCSEL atid MSM-photodetector arrays [3]. Sixty-four hit bi-directiortal tlat;i paths fully intercotiiiect the sixteen composite switch chips oii the MCM with free-space optical links estahlislied with micro- atid rnacro-optics. l h c CMOS ASlCs to which thcsc 01; chips are bonded are deepsubrnicron ICs which contain drivers, reccivcrs, switching logic, iind protocol-specific circuitiy. l'he switch chip logic is divided into receive logic inid traiismit logic which comespoiid to incoming and outgoing data, respectively. The reccivc logic is responsibk fix formatting the data coming froin the parallel fiber interfice to the intenial switch-data forrnat oi'3Z-bit words. It also lonvards data to the appropriatc MCM chip lor transtnissioii to the destination host NI(:. 'l'he trmsmit logic is responsible for reformatting the data before it is scnt through the parallel fiber link a i d for implementing congestion colitcol. A sinal1 amount of IIFO l~ufferirigis used in the switch to align data coruiug from tnultiple sources. The combination 0 1 froe-space and fiber-ribhon optical comnmiiication with a switched network architecture will enable multi-gigabit ititer-host coiniminication io a distributed-processing workstation cluster. Rcfercncc: I . VIVACE (VCSEL-based Ititercotinccts in VLSI Architectures for Computational Enhailcement) Program website: http://www.litc.ho~icywell.coii~pliotoiiics/vlsi.liti~l 2. P.Chaodramani et. al., "Iligh-Speed Free-Space Scalable Switchiiig Network for Parallel Computing", Proceedings: Optics in Computing Conference 2000, June 18 - 23, pg 578-582, Quebec City, Quebec Canada 3. M. K. Hibbs-Ui-eniicr, Y. Liu, R. Morgan and J. Lelii~~iti, "VCSEI./MSM Detector Sinart Pixel Arrays" IEEE/LEOS,I 998.

495