an 800mbps system interconnect modeling and ...

1

An 800Mbps System Interconnect Modeling and Simulation for High Speed Computing Mohammad S. Sharawi and Daniel N. Aloi Electrical and System Engineering, Oakland University Rochester, MI 48309 USA [email protected] , [email protected] Abstract— System interconnect modeling for high speed systems is a vital bottleneck for high speed data transfer. We demonstrate the modeling process on a high speed computer differential net running at 400MHz (800Mpbs) with IBM I/O cells. The modeling of the traces on the boards was done using a field solver. The transmission line matrices were used in a SPICE model, and 3-simulation scenarios were tested for this model. The obtained EYE opening of the modeled interconnect simulation was 705mV while the measured EYE opening for the same net topology in the Lab was 710mV . This shows a close match between the actual behavior and the model generated. Careful modeling can be very beneficial to get a design running at first time operation. Keywords—High Speed, PCB, Interconnect, Signal Integrity.

transmission. Special attention and care should be taken when designing high speed boards, or investigating high speed interconnects. This is why we touch upon this important area in electronic design, and investigate the stages that a designer should follow to get an accurate and reliable interconnect model to be used in his high speed designs. The paper is structured as follows: Section II considers modeling of high speed interconnects like PCB traces, and connectors. Section III presents an example of a high speed data net that travels from one machine through a number of boards to another via an external cable. Section IV illustrates the design results and models. The paper is concluded in section V.

I. INTRODUCTION Today’s high speed electronic devices and gadgets are considered a vital element in the life of professionals. Ranging from Personal Computers (PC), to Pagers, Cell phones, hand held devices, Personal Digital Assistants (PDA), and ending with Pocket PCs. The necessity to process and transfer large amounts of data in relatively short periods of time, is the driving factor in high speed industry, where the competition is based on how much data can you transfer and how fast? In order to accomplish this task, special care should be taken when designing high speed circuits and interconnects that combine these circuits and the functionality of a system. As the bit rate of the data increases, and the frequency of switching increases, the interconnect behavior should be looked at carefully. If the design or model of the interconnect is not done properly, the data transfer between various devices might fail, and your product will be doomed. High speed interconnect modeling, design and analysis is not an easy task. As a matter of fact, a new discipline in electrical engineering has evolved that specifically studies, analysis and models such behavior to ensure proper high speed data transfer and operation. This field is called Signal Integrity (SI). SI combines Analog and Digital circuit design methodologies with Electromagnetic field theory to analyze and characterize the behavior of various interconnects ranging from Printed Circuit Board (PCB) traces (wires), to connector models that connect boards or devices together, to board and cable irregularities that has an effect that cannot be ignored at high speeds on signal

0-7803-8834-8/05/$20.00 ©2005 IEEE.

II. Modeling of High Speed Interconnects As the rise time of the signal use gets lower, the transition speed of the signal becomes higher, and the wave is considered as a TEM (Transverse Electromagnetic) wave travelling from driver to receiver. The wire or PCB trace (or etch) connecting the driver to receiver circuits will no longer be treated as a lumped Resistive-Capacitive network (RC), but it will be considered as a transmission line (distributed network, Fig. 1) if the length of the wire/etch is greater than 1/6 of the ratio between the Rise time Tr and Tr Propagation delay D, i.e. l > 6D , or if Tr < 6T D, where T D is the time delay of the interconnect. In this case the PCB trace will be treated and modelled as a TL, and its characteristics will be determined based on the geometry, dielectric constant, frequency of operation, and the stackup of the board.

i(z,t)

Rdz

Ldz

v(z+dz,t) + i(z+dz,t)

Cdz

Gdz

+ v(z,t) −

− dz

Fig. 1. Transmission Line Distributed Network.

Maxwell’s equations for the TEM waves on multiconductor transmission lines reduce to the telegraphers equations. The general form of the telegrapher’s equations in the fre-

6198

Authorized licensed use limited to: King Fahd University of Petroleum and Minerals. Downloaded on March 02,2010 at 12:16:34 EST from IEEE Xplore. Restrictions apply.

2

quency domain are given by [3]: ∂ − v(z, ω) ∂z ∂ − i(z, ω) ∂z

Plane H

S

=

[R(ω) + jωL] i(z, ω)

T

(1)

H

W

=

[G(ω) + jωC] v(z, ω)

(2)

Fig. 2. A three conductor PCB Cross section.

where, boldface lower-case and upper-case symbols denote vectors and matrices, respectively. v is the voltage vector across the lines and i is the current vector along the lines. There are various rules of thumb to evaluate and calculate the parameters of the PCB trace, but usually if we are looking for an accurate model at a certain bandwidth of operation, we rely on a field solver (2D or 3D) to give us the parameters of the PCB trace structure for a certain geometry and stack-up. One of the most widely used field solvers is Maxwell (from Ansoft Inc.) [10]. Lossy Transmission lines that we usually incorporate in our PCB trace designs are usually characterized by the following, • Capacitive Matrix: that includes self and mutual capacitance of the trace with adjacent traces, (C) in Farads/meter. • Inductive Matrix: that includes self and mutual inductance of the trace with adjacent traces, (L) in Henrys/meter. • DC Resistance Matrix, (R) in Ohms/meter. • DC Conductance Matrix, (G) in Siemens/meter. • AC Resistance Matrix (due to Skin effect), (Rs ) in √ Ohms/meter Hz. • Conductance losses Matrix (due to dielectric losses), (Gd ) in Siemens/meter Hz.     L11 C11   L =  L21 L22 C =  C21 C22 C31 C32 C33 L31 L32 L33     R11 G11   R22 R= 0 G =  G21 G22 0 0 R33 G31 G32 G33   Rs11  Rs22 Rs =  0 0 0 Rs33   Gd11  Gd =  Gd21 Gd22 (3) Gd31 Gd32 Gd33 These matrices include information about the value of the parameter for the trace itself, and the effect of adjacent traces on it. The matrices are lower triangular (3 × 3 since we have a three conductor stripline trace, see Fig.2) since the effect of a trace 1 on trace 2 is same as the effect of 2 on 1 (symmetric). Note that the R and Rs do not have any mutual effect (the DC resistance and skin effect are properties of traces themselves and there is no mutual effect). The frequency dependent relationships of these matrices are [3], [4], [6], R(f ) = Ro +

er

Plane

√ f Rs

G(f ) = Go + f Gd

Ω m S m

(4) (5)

C(f )

= Co

L(f ) = Lo

F m H m

(6) (7)

where Lo , Co , Ro and Go are the DC valued matrices characterizing the TL. The characteristic impedance of every conductor can be found using its respective self inductance and capacitance using the equation  L Zo = (8) C  L11 i.e. for conductor 1, its Zo = C Ω. The Propagation 11 delay of such a stripline is given by, √ T D = LC (9) √ i.e. for conductor 2, its T D = L22 C22 inches/sec. After creating the geometry and stackup of the layers, we run the field solver on the traces drawn to get these matrices. After getting the value of these matrices, which is usually saved in a file, we pass this generated file to the transistor level simulator, which is HSPICE is our case. A file for the high speed system should be specified in SPICE, and we have to incorporate as close to real life models in the design as possible, to get close to lab measurements values. This spice-deck should have the correct Driver/Reciever models taken from (generated by) the vendor of the IC (Integrated Circuit) used, the correct interconnect models, i.e. PCB traces, via models, and connector models, and any other parameters that might affect the operation of the system. Then, once we gather this file, we simulate to see the behavior of the circuit and net with such models. The driver/receiver models, and connector models, usually are obtained from the vendor of the device. But PCB and via modelling are done by the designer. And this is not an easy job, since it needs alot of experience, and patience in running long runs of simulation scenarios. In HSPICE, the PCB trace is modelled via the W − line TL model, because it has faster convergence times than the older U − line model, no spurious ringing, frequency dependent loss is well and accurately modeled, and no limit on coupled conductor numbers [4], [6]. HSPICE based modeling is considered the most accurate in industry, unfortunately it is also time consuming, and consumes long simulation times (i.e. this net took about an hour to simulate for a couple of 100’s of nsec on a multiprocessor machine). IBIS modeling on the other hand is not as accurate, but can get relatively good results with 0.1 times the simulation time of an HSPICE based one [9]. This will not be discussed further in this paper.

6199 Authorized licensed use limited to: King Fahd University of Petroleum and Minerals. Downloaded on March 02,2010 at 12:16:34 EST from IEEE Xplore. Restrictions apply.

3

11"

IBM/Driver PRG IN

pkg IBM/Driver

pkg

12" Conn

Meg Board 1

Board 2

11"

12" Meg

Machine 1

12"

2m

Cable

Conn

Cable External Cable

Board 1

12"

2m Conn

11"

Meg

IBM Receiver pkg

Meg

Board 2

Conn

11"

IBM Receiver pkg

Machine 2

Fig. 3. The net topology simulated. Copper etch width is 4mils, thickness is 0.5Oz., and separation is 6mils.

III. A design example In this section we will present a design example to demonstrate the modelling/simulation of various interconnects in a driver receiver path. We will consider a 2 conductor stripline connecting a driver chip (IBM 400MHz driver) on one computer machine, through a high speed low latency cable, to another machine that has the receiver (IBM 400MHz receiver), as shown in Fig. 3. The driver/reciever are differential components (operating with a 2.5V supply, and differential common mode voltage of 1.25V), i.e. the signal is driven with its complement on two conductor lines. Such a signal suffers less noise than single ended signals, because the generated mutual effects are opposite to each other, and their effect is minimized (almost cancelled) relative each other. The traces are being modeled using Maxwell (Fig. 4), with width of 4mils, separation of 6mils, distance from reference planes was 5mils, and thickness of 0.5oz. (0.7mils). The IC package model was taken from a typical 400MHz component1 , so is the Meg-Array connector model, that connects the two boards in each machine together, and the Cable connector model, that connects the two machines together. Such connector models are a combination of an RLGC values that represent the behavior of the connector to frequency changes and DC and AC losses2 . The cable connecting the two machines is 2 meters long and has a DC loss of about 1Ω. Its a low loss, low latency, high speed cable. The etch length on machine 1 was 23inches (board 1 has 11inches, and board 2 has 12inches), and on machine 2 was the same (a scenario that might be encountered while designing high speed boards for high performance computers.) The PCB, connector and

Fig. 4. Analyzing the PCB traces via Maxwell, mesh sections. 1 such models are either generated by companies for their custom designed chips, or obtained from the IC vendor who sells the chip. 2 The BGA (Ball Grid Array) Meg-Array model can be obtained from http://www.fciconnect.com. [11]

driver models are then combined in a single HSPICE file for simulation. The simulation was performed by applying a Pseudo Random bit generator (PRG) with a frequency of 400MHz (800 Mbps) at the driver inputs, and the EYE pattern was then observed at the receiver end. The EYE pattern is very important in Signal Integrity, and is very helpful in identifying problems, like Intersymbol Inteference (ISI), low Swing Levels, timing problems, and noise issues. We always need to see an EYE wide open to get good signal timing (setup and hold times), and have a good sampling margin. Also, the levels should be sufficient for the technology used. Finally, the crossing point should be as thin as possible to have less ISI. IV. Simulation and modelling results The simulation results of our differential net shown in Fig. 3 are presented in this section. The matrices obtained from the Maxwell simulation for the two conductor stripline operating at 800MHz3 , with T = 0.7mils, H = 5mils, w = 4mils, s = 6mils, r = 4.3, where,   1.118267e − 10 C= F/m −1.266590e − 11 1.117687e − 10   3.830068e − 07 L= H/m 4.340322e − 08 3.832060e − 07   9.880000e + 00 R= Ω/m 0.000000e + 00 9.880000e + 00   0.000000e + 00 G= S/m 0.000000e + 00 0.000000e + 00   Ω 1.187494e − 03 √ Rs = 5.245024e − 05 1.161468e − 03 m Hz   S 1.755680e − 11 Gd = −1.988546e − 12 1.754769e − 11 mHz The single ended Zo1 = 56.9Ω, Zo2 = 57.1Ω , and the differential impedence ZoDiff = 113.9Ω. The propagation delay for each trace T D1 = 6.544nsec/m, and T D2 = 6.545nsec/m. The simulation of the net was performed using a PRG with a bit width of 1.25nsec, rising edge of Tr = 200psec, falling edge of Tf = 200psec. The resulted EYE pattern at the receiver end looks like Fig. 5. To see the effect of etch length and width on the observed EYE parameters, we recorded some simulation scenarios in Table I. In Table I, the simulation of the net in Fig. 3 shows a differential level of 743mV with no discontinuity in the path, represented by the via, while the existence of such a via degrades the voltage swing by about 40mV , which is a drop of about 5%. In the third scenario, we split the 12 inches of trace of board 2 in each machine into two parts, one of 4mils width and 3inches of length, and then a via is used to have this trace routed on another layer, and have a thickness of 8mils, twice as much, and length of 7inches. This will reduce the signal loss since the thickness of the long trace is 8mils instead of 4mils, not to mention that the total trace was reduced by 2inches on each machine, 3 We simulated the traces at 800MHz because the data rate was 800 Mbps.

6200 Authorized licensed use limited to: King Fahd University of Petroleum and Minerals. Downloaded on March 02,2010 at 12:16:34 EST from IEEE Xplore. Restrictions apply.

4

TABLE I Simulation results Table.

Trace width 4 mils 4 mils

Trace thickness 0.5 Oz. 0.5 Oz.

4 mils 8 mils

0.5 Oz.

Trace length/machine 23 in. 23 in. with via (3+11) in. + via 7 in + via.a

Differential Eye Opening 743 mV 705 mV 842 mV

a

the trace on board 2 in machines 1 and 2 (12 in.) was redesigned and split into two traces with different lengths to reduce signal loss and increase the EYE width.

with a total of 4inches in the signal path. The signal level increased by about 100mV or about 14%. In high speed signaling and with this amount of loss (signal at driver is 2.5V) on a long interconnect path (as a worse case, specially in huge systems and supercomputers that incorporate the interconnection of many boards, and then connecting subsystems together) such an improvement in signal levels and swing is highly appreciated by system and circuit designers. Fig. 5 shows the EYE pattern for the second case in

Fig. 6. Measured EYE on the receiving end at machine 2.

operation of such interconnects affects the quality and speed of the signal. The best way to model such interconnects is by using electromagnetic field solvers that can give us accurate high speed modeling. We have modeled a high speed differential net operating at 400MHz (800Mbps) using a transistor level simulator (SPICE), and a field solver called Maxwell (Ansoft Inc.), and compared the behavior of the net with that of a net with the same topology that was measured in the Lab. The interconnect model had a behavior as the one used in the real system. We have obtained an eye opening of 705mV using the simulated net, and we measured an eye opening of 710mV for the same net topology in the Lab. This means that by accurate and careful modeling of such system interconnects, we can get a very close to real life behavior of such critical nets. Acknowledgements

Fig. 5. Eye Pattern simulation. Trace width is 4mils, thickness 0.5Oz., length 23inches, and a via is present in the signal path.

Table I. As you can see the EYE voltage opening is about 705mV taken differentially, and has about 180psec of ISI. The width between the crossings of the smallest EYE is about 950psec. Fig. 6 shows the measure EYE pattern of the topology assumed in the design example4 . The EYE opening was 710mV for the second case in Table I. This shows that the simulation and modelling of such interconnects, and the high speed circuit does actually give a very good estimation about real life signal behavior, given the modeling was conducted in the correct fashion. V. Conclusions

The authors would like to thank Silicon Graphics Inc. (SGI) and Brad Juskiewicz for their help in accessing their labs and help in testing the modules. References [1] [2] [3] [4] [5] [6] [7]

[8]

As the speed of data transfer is increased, careful modeling and design of system interconnects should be performed. The transmission line behavior of this high speed 4 This test and measurement was performed in one of Silicon Graphics Inc. Lab, CA-USA.

[9]

[10] [11]

H. Johnson and Martin Graham , High Speed Digital Design: A Handbook of Black Magic , Prentice Hall, 2nd Edition, 1996. H. B. Bakoglu , Circuits, Interconnections, and Packaging for VLSI , Addison-Wesley, 1990. “Star HSPICE, User’s Manual, Volume III”, Avanti Inc., 1996. “Star HSPICE, User’s Manual Supplement”, Avanti Inc., 1997. S. Hall, J. Hall, and J. McCall, High Speed Digital Design: A Handbook of interconnect theory and Design Practices, John Wiley & Sons, Reading, 2000. “Star HSPICE, User’s Manual ”, Avanti Inc. 2000. Dmitri Borisovich and Jose Schutt-Aine, “Optimal Transient Simulation of Transmission Lines”,IEEE Transactions on Circuits and Systems-I: Fundamental Theory and Applications, vol. 43, no.2, pp. 110-121, February 1996. Nannapaneni Narayana Rao , Elements of Engineering Electromagnetics , Prentice Hall, 4th Edition, 1994. Mohammad S. Sharawi,“Modeling and Simulation of High Speed Digital Circuits and Interconnects ”,Middle East Conference on Simulation and Modeling (MESM), Amman-Jordan, Sep. 14-16, 2004. http://www.ansoft.com http://www.fciconnect.com

6201 Authorized licensed use limited to: King Fahd University of Petroleum and Minerals. Downloaded on March 02,2010 at 12:16:34 EST from IEEE Xplore. Restrictions apply.