Telecommunications Infrastructure Standard for Data Centers ANSI ...

Telecommunications Infrastructure Standard for Data Centers ANSI/TIA-942

Chris DiMinico [email protected] MC Communications

IEEE 802.3 HSSG

Data Center Standards

ANSI/TIA-942 Telecommunications Infrastructure Standard for Data Centers Co-chairs: Chris DiMinico & Jonathan Jew Published 2005 – available through TIA at www.tiaonline.org ANSI/NECA/BICSI-002 Data Center Design and Implementation Best Practices co-chairs: Jonathan Jew & John Kacperski best practices – complements TIA-942 – 2007 target IEEE 802.3 HSSG

Who Developed TIA-942?

Developed by the TIA TR-42.1.1 Network Distribution Nodes subcommittee as Project No. 3-0092 Participants included: Architecture & Engineering Firms Consultants End Users Manufacturers

IEEE 802.3 HSSG

Purpose of TIA-942

Encourage early participation of telecom designers in data center design process Fill a void by providing standards for planning of data centers, computer rooms, server rooms, and similar spaces. The standard encompasses much more than just telecommunications infrastructure. Close to half of the technical content deals with facility specifications.

IEEE 802.3 HSSG

Purpose of TIA-942

Define a standard telecommunications infrastructure for data centers Structured

cabling system for data centers using standardized architecture and media Accommodates a wide range of applications (LAN, WAN, SAN, channels, consoles) Accommodates current and future protocols (e.g., 10+ GbE ) Replaces unstructured point-to-point cabling that uses different cabling for different applications

IEEE 802.3 HSSG

Purpose of TIA-942

Specifications for data center telecommunications pathways and spaces Recommendations on media and distance for applications over structured cabling Establish a standard for data center tiers to replace several proprietary standards. The TIA data center tier standard is: • A tool to evaluate existing data centers • A tool to communicate design requirements IEEE 802.3 HSSG

Unstructured Cabling C

D

E

F

G

H

I

J

K

L

M

N

O

P

Q

R

S

T

U

Heater - A/C

1

TABLE 247-7589

AB

AC

AD

AE

AF

TABLE

FI L E

F IL E

CABINET

CABI NET

AI

AJ

AK

AL

AM

AN

AO

AP

AQ

F IL E CABINET

FIL E CABINET

POMONA A/C UNIT #6

AR

AS

HP PRINTER

C D ROM SCANNER

UNIT

N ETWORK CABINET E7

NETWOR K

CABI NET I1

NE TWORK CABI NET I2

NE TWORK

CABI NET I3

I5

NE TWORK C ABINET I4

NETWORK CAB INE T

NETWORK

CABINET I7

NETWOR K C ABINET I6

CAB INE T I8

DESK

NETWOR K

H8 N ETW OR K CABINET H7

N ETW OR K CABINET H6

NETWORK CABINET G7

IBM CAB

CABINET F6

N ETWOR K CABINET G6

N ETW ORK C ABINET F5

NETWORK CABINET G5

N ETW ORK C ABINET

NETWORK C ABINET E6

IBM 3745-A11

IBM 3746-900

NETWORK C ABINET

IBM

TAPE RACK

POMONA A/C UNIT #8

TAPE RACK

NETWORK CABIN ET H4

NETWORK CABIN ET G4


NETWORK

NETWORK CABIN ET


CABIN ET H3

NETWORK CABIN ET H2

IBM DASD

IBM S-390 R34

IBM DASD

F4

NETWORK CABIN ET H1

G3 NETWORK CABIN ET G2

LAN CABINET F1

NETWORK C ABINET E4 NETWORK C ABINET E3

NETWORK C ABINET E2

NETWORK C ABINET E1

3745-41D

IBM 3745-17A

IBM 3745-L13 SAA CABIN ET D2

N ETW OR K CABINET A6

NETW OR K CABINET

NETWORK CABINET

C6

B7

N ETW OR K

N ETWOR K

CABINET A5

N ETW ORK

E5

11

AH

Heater - A/C

POMONA A/C UNIT #7

N ETW OR K CABINET

NETWORK CABIN ET H5

8

AG

AT &T

7

10

AA

DEFINIT Y

6

9

Z

TABLE

He ater - A/C

5

Y

NETW OR K CABINET C5

CABINET B6

HALON PANEL

NETWORK CABINET A4


N ETWOR K CABINET B5



N ETW OR K CABINET

NETW OR K

N ETWOR K

A2

CABINET C2

CABINET B3



N ETW OR K CABINET A3

NETWORK

SAA CABIN ET

CI SCO ROU TE R

4

X

CONS OLE

3

W

Heater - A/C

STORAGE CABINET

DESK

2

V

F/X 247-7543 HALON ABORT HALON RELEASE

C ABI NE T A1

B

PA C BELL

A

D1 N ETWOR K CABINET B1

NETWORK CABIN ET G1

12 247-7538

13 14 STORAGE C ABINE T

TAPE RACK

15

IBM AS -400

IBM AS-400

INSTALL A CABLE WHEN YOU NEED IT (SINGLE-USE, UNORGANIZED CABLING) TABLE

16

IEEE 802.3 HSSG

LOCKER

STORAGE CABINET

LIEBERT PDU -C

3174 CABINET STORAGE CABINET

3174 CABINET

3174 CABINET

3174 CABINET

3174 CABINET

3174 CABINET

Structured Cabling B

C

D

E

F

G

H

I

J

K

L

M

N

O

P

Q

R

S

T

U

V

Heater - A/C

1

TABLE 247-7589

NE TWORK C ABINET I1

NETWORK CABINET I2

CAB IN ET I4

NETWORK CABINET I3

NETWORK

NE TWORK

CABIN ET I6

NETWORK C ABI NET I5

NETWOR K CAB INE T I7

NE TWORK CABI NET I8

IDF

NETWORK C ABINET H6

IDF

N ETWOR K C ABINET H5

T APE RACK

TABLE

AE

AF

AG

FIL E

FILE CABINET

FIBER MDF COPPER & CORE MDF

POMONA A/C UNIT #8

NETWORK C ABINET E7

NETWORK CABINET G7 NETWORK CABIN ET G6 N ETWOR K C ABINET G5 NETWORK CABIN ET

NETWOR K C ABINET F6 NETWOR K C ABINET F5

NETWORK CABIN ET F4

G4

NETWORK CABIN ET F3


NETWORK CABIN ET G3

NETWORK CABIN ET F2


NETWORK CABIN ET G2

LAN CABINET F1


NETWORK CABIN ET G1

C ABINET H4

TAPE RACK

N ETWORK CABINET E6

AH

AI

AJ

AK

AL

AM

AN

AO

CABINET

FIL E CABINET

FIL E CABINET

POMONA A/C UNIT #6

CD ROM SCANN ER

AP

AQ

AR

AS

HP PRINTER

IBM S-390 R34

IBM DASD

IDF

IDF

IBM 3745s IBM CAB

NETWOR K NETWORK

IBM 3745-A11

IBM 3746-900

C ABINET C6

NETWORK CABINET B7

NETWORK C ABINET C5

NETWORK CABINET B6

NETWORK C ABINET

NETWORK CABINET

C ABINET A6

CABINET E5 N ETWORK CABINET E4 N ETWORK CABINET E3 N ETWORK CABINET

N ETWORK CABINET E1

IBM

NETWOR K C ABINET A4

C4

B5

NETWOR K C ABINET

NETWORK

A3

C ABINET C3

NETWORK CABINET B4

NETWORK C ABINET C2

NETWORK CABINET B3

3745-41D

IBM 3745-17A

IBM 3745-L13 SAA CABINET D2

NETWORK

IDF C ABINET C1

SAA C ABINET D1

IDF

IDF

F/X 247-7543 H ALON ABORT


NETWORK CABINET B2 NETWORK CABINET B1

H ALON PANEL


N ETWORK

E2

Mainframe IBM DASD

11

AD

Heater - A/C

POMONA A/C UNIT #7

NETWORK CABINET H8 NETWORK C ABINET H7

N ETWOR K

10

AC

UNIT

7

9

AB

AT&T

6

8

AA

DEFINITY

DESK

5

Z

TABLE

Heater - A/C

4

Y

CISCO ROUTE R CON SOLE

3

X

Heater - A/C

STORAGE CABINET

DESK

2

W

H ALON RELEASE

PAC B ELL C A BIN ET A1

A

IDF

IDF IDF

12 247-7538

13 14 15

S TOR AGE C AB IN ET

TAPE RACK

IBM AS-400

IBM AS-400

STRUCTURED CABLING SYSTEM (ORGANIZED, REUSABLE, FLEXIBLE CABLING) T ABLE

16

IEEE 802.3 HSSG

STORAGE CABINET

LIEBERT PDU- C

LOC KER

3174 CABINET STORAGE CABINET

3174 CABINET

3174 CABINET

3174 CABINET

3174 CABINET

3174 CABINET

Design Elements

Cabling Design Facility Design Network Design

Informative annex’s: Provide best practices Annex A - Cabling Design Considerations Annex B- Telecommunications infrastructure administration Annex C-Access provider information Annex D- Coordination of equipment plans with other engineers Annex E- Data center space considerations Annex F- Site selection IEEE 802.3 HSSG

Design Elements

Cabling Design:

Copper and fiber cabling performance Connectors, cables, distribution hardware Cabling distances Space management

Facility Design: Data center sizing Power distribution methodologies Pathways and spaces HVAC, security, operations, and administration. Flexibility, scalability, reliability and space management

IEEE 802.3 HSSG

Design Elements

Network Design: Support

of legacy systems Enable rapid deployment of new and emerging technologies such as 10 GbE and 10+ GbE copper and fiber applications.

IEEE 802.3 HSSG

Relationship of Spaces BUILDING SITE BUILDING SHELL GENERAL OFFICE SPACE

TELECOM ROOMS & EQUIPMENT ROOMS for spaces outside data center

OFFICE BUILDING SUPPORT SPACE

DATA CENTER SUPPORT STAFF OFFICES

OPERATIONS CENTER

ENTRANCE ROOM(S)

TELECOM ROOM(S) for data center support spaces

COMPUTER ROOM

IEEE 802.3 HSSG

DATA CENTER ELECTRICAL & MECHANICAL ROOMS

STORAGE ROOMS & LOADING DOCKS

Data Center Topology Entrance Room

Entrance Facility

Carriers

Carriers

Active Equip interconnect

Offices, Operations Center, Support Rooms Secondary Entrance Room Active Equip interconnect

Active Equip interconnect

MC MC

Carriers Carriers

Back bone Cabling HC IC

HC IC

HC IC Horizontal Cabling

interconnect Active Equip Storage Area Devices

interconnect Active Equip

ICHC Zone Distribution interconnect

interconnect Active Equip Active Equip

Data Center IEEE 802.3 HSSG

TIA-942 Spaces

Entrance Room (ER) - location of interface with campus and carrier entrance facilities Main Distribution Area (MDA) – location of main cross-connect (MC) Horizontal Distribution Area (HDA) – location of horizontal cross-connect (HC) Zone Distribution Area (ZDA) – location of zone outlet (ZO) or consolidation point (CP) Equipment Distribution Area (EDA) – location of equipment cabinets and racks IEEE 802.3 HSSG

Data Center Cabling Main Distribution Area

MC MC

Backbone Cabling

Horizontal Cabling

HC IC



ICHC

Horizontal Distribution Area

interconnect

Zone Distribution Area

Active Equip

Equipment Distribution Area

Horizontal cabling is the cabling from the horizontal cross-connect (in the main distribution area or horizontal distribution area) to the outlet in the equipment distribution area or zone distribution area. IEEE 802.3 HSSG

Horizontal and Backbone Cabling

Recognized Cables: a) 100-ohm twisted-pair cable (ANSI/TIA/EIA-568B.2), category 6 recommended (ANSI/TIA/EIA568-B.2-1) b)

multimode optical fiber cable, either 62.5/125 micron or 50/125 micron (ANSI/TIA/EIA-568-B.3), 50/125 micron 850 nm laser optimized multimode fiber is recommended (ANSI/TIA-568-3-1)

c)

singlemode optical fiber cable (ANSI/TIA/EIA568-B.3)

d)

75-ohm (734 and 735 type) coaxial cable (Telcordia Technologies GR-139-CORE) IEEE 802.3 HSSG

Horizontal cabling distances

The maximum horizontal distance is 90 m independent of media type. The maximum channel distance including equipment cords is 100 m. The maximum cabling distance in a data center not containing a horizontal distribution area is:

300 m for an optical fiber channel including equipment cords. 100 m for copper cabling including equipment cords.

IEEE 802.3 HSSG

Backbone Cabling

Includes cabling from MDA to ER, HDA Optional cabling between HDAs allowed Maximum backbone cable lengths depend on applications to be supported Centralized optical fiber cabling supported with interconnect, splice, or pull-through at the HDA Star topology with no intermediate crossconnects Various topologies permit redundancy and flexibility to support various data center sizes IEEE 802.3 HSSG

Computer Room & Entrance Room Requirements

Min clear height of 2.6m/8.5 ft Min door size 1m/3ft wide 2.13/7ft high Min dist floor loading 7.2 kPA/150lbf/ft2, recommended min 12 kPA/250 lbf/ft2 20 degrees C to 25 degrees C 40% to 55% relative humidity (reduces ESD) Any sprinkler systems must be pre-action system Common bonding network (CBN) – equipotential ground reference

IEEE 802.3 HSSG

Main Distribution Area

Location of Main Cross-Connect (MC), the central point of distribution for data center structured cabling system Centrally located to avoid exceeding maximum distance restrictions (typically for E1s, E-3s, T-1s and T-3s) Install separate racks for Fiber, UTP, and coaxial cable distribution

IEEE 802.3 HSSG


Location of Horizontal Cross-Connect (HC), the distribution point for cabling to equipment distribution area Distribution LAN, SAN, KVM switches and console servers located in HDA MDA may also include an HC for nearby equipment distribution area Number of HDAs depends on the density of cabling and the size of the data center

IEEE 802.3 HSSG


The capacity of the cable tray system and the size of the cross-connect creates practical limits on the size of the HC Guideline is maximum of 2,000 4-pair UTP or coax cable terminations per HDA Arrange patch bays to minimize patch cable lengths and to simplify cable management Separate racks for fiber, UTP, and coax Locate switches and patch panels to minimize patch cord lengths

IEEE 802.3 HSSG

Zone Distribution Area

Rack, cabinet, or under floor enclosure that houses a zone outlet (ZO) or consolidation point (CP) ZO - structured cabling termination for floor-standing equipment that cannot accept patch panels (e.g. mainframes and large servers). CP - intermediate termination point (e.g. cabling to areas where floor plan is uncertain or dynamic) No cross-connects within the ZDA No active equipment shall be located in the ZDA Maximum of 144 connections in a ZDA Maximum of one ZDA within a horizontal cable run

IEEE 802.3 HSSG

Equipment Cabinets

IEEE 802.3 HSSG

Front rails of cabinets must be recessed to provide adequate room for patch cables and wire managers Recommend 1-to-1 ratio of patching to cable management Arrange switches and patch panels to minimize patching between cabinets & racks Perforated tiles at front of cabinets One edge of cabinets placed at edge of tile

Raised Floor

Better appearance than overhead cabling. Allows higher power densities, better control of cooling, and more flexibility in location of cooling equipment Most stand-alone computer systems are designed for cabling from below Coordinate under floor cabling with mechanical & electrical engineers Recommend wire basket cable trays in hot aisles for telecom cabling

IEEE 802.3 HSSG

Example of Wire Basket Cable Trays For Cabling Under Raised Floor 24" 1.375"

RAISED FLOOR TILE

DIVIDER 18" 9"

5" 4"

FIBER

COPPER CABLING

SUPPORT STRUT

36"

12"

18" 5" 4"

FIBER

COPPER CABLING

SUPPORT STRUT

Open Area 13.5"

24" On Center

IEEE 802.3 HSSG

1"

1"

Overhead Cable Trays

Less expensive than raised floor systems Cable trays can be attached to the top of racks and cabinets (if they are uniform in height) Cable trays suspended from the ceiling provides more flexibility for supporting cabinets/racks of various heights and for adding and removing cabinets/racks. Cable trays can be installed with several layers Coordinate location with lighting, ducts, overhead conduits, overhead power IEEE 802.3 HSSG distribution

Overhead Cable Tray Example 3 Layer cable tray system: Bottom layer – signal Middle layer – power Top layer – fiber Signal Reference Grid in brackets attached to lower layer of trays Fiber patch cables may be in fiber duct attached to threaded rods

IEEE 802.3 HSSG

Infrastructure Administration

Informative annex with TIA-606-A standards compliant labeling scheme for all components. Labeling scheme extended for use in data centers Cabinets and racks labeled by location using tile grid or row/position identifiers All cabinets, racks, patch panels, cables, and patch cords should be labeled IEEE 802.3 HSSG

Labeling Example AA

AB AC AD

AE

AF

AG AH

AI

AJ

AK AL AM AN AO AP AQ AR AS

AT

AU AV AW AX AY

"X" COORDINATE AZ BA

BB BC BD

01 02

Cat 6 x 24

03 04 05

RIGHT FRONT CORNER

06 07

CABINET AQ03

CABINET AJ05

08 09 10 11 12 13

"Y" COORDINATE

Patch Panel for 24 Cat 6 from 1st Panel in Cab AJ05 to 2nd Panel in Cab AQ03 AJ05-A to AQ03-B Ports 01-24 01

02

03

04

05

06

07

08

09

10

11

12

13

14

A

IEEE 802.3 HSSG

15

16

17

18

19

20

21

22

23

24

A

Site Selection

Informative annex with guidelines for selection of a site for a data center Architectural Electrical Mechanical Telecommunications Security Other

IEEE 802.3 HSSG

Facilities Specifications & Tiers

Informative annex with general architectural, structural, electrical, mechanical, and telecommunications recommendations requirements Annex includes detailed architectural, security, electrical, mechanical, and telecommunications recommendations for each Tier Recommended specifications by tier are a uniform way to rate aspects of a data center design and are a starting point for initiating design requirements with qualified architects and engineers

IEEE 802.3 HSSG

Data Center Tiers

Tier 1 – basic data center no redundancy Tier 2 – redundant components Single distribution path with redundant components Tier 3 – concurrently maintainable Multiple distribution paths with only one active Tier 4 – fault tolerant Multiple active distribution paths IEEE 802.3 HSSG

Redundant Topologies Carriers

Carriers

Carriers

Secondary Customer Maintenance Hole (Tier 2 and higher)

Primary Customer Maintenance Hole (Tier 1 and higher) r3

Tier

2

Primary Entrance Room (Tier 1 and higher) Tie

Secondary Entrance Room (Tier 3 and higher)

r4

Tie

Main Dist Area (Tier 1 and higher)

Telecom Room

Tier 3

Tier 4

Tier 1

Offices, Operations Center, Support Rooms

Tier 3

Tier 1

Data Center

Tie

Carriers

r3

Tier 4

Secondary Dist Area (Optional for Tier 4)

Computer Room Horiz Dist Area Optional Cabling

Horiz Dist Area

Horiz Dist Area

Equip Dist Area

Equip Dist Area

Zone Dist Area

Equip Dist Area

IEEE 802.3 HSSG

Data Center Tiers

Higher tiers correspond to higher availability, but also have higher construction costs Data Center can have different tier ratings for different portions of its infrastructure (architectural, security, mechanical, electrical, telecommunications) The overall rating for the data center is equal to the lowest tier rating Capacity of systems may need to be upgraded to maintain tier rating as data center load increases Human error and operating procedures have a major impact on availability

IEEE 802.3 HSSG

Conclusion

TIA-942 is the first standard to specifically addresses data center infrastructure. Primarily a telecom infrastructure standard, but about half of the content deals with facility requirements. Provides a flexible and manageable structured cabling system using standard media. Builds on existing standards, when applicable Guidelines on a wide range of subjects useful to someone designing or managing a data center. An official tiering standard for determining the quality of a center. A way to objectively compare one center with another.

IEEE 802.3 HSSG