Designing a Fiber Structured Cabling System for the Data Center

Designing a Fiber Structured Cabling System for the Data Center A Web Conference Presented by the TIA Fiber Optics LAN Section John M. Struhar, Director Fiber SCS Solutions Ortronics/Legrand [email protected]

1

11/18/05 JS Copyright © 2005 Ortronics/Legrand. All rights reserved.

TIA Fiber Optics LAN Section (FOLS) Founded in 1993 as a Section of the TIA’s Fiber Optics Division Mission: Educate system designers, architects, consultants, engineers, contractors, end users & the media about the technical advantages that optical transmission brings to customer-owned networks Stimulates development of new fiber standards and promotes optical-based applications in customerowned networks

2


Today’s Agenda

Designing a Fiber SCS for the Data Center

Introduction to new standards-based data

3

center & storage area network design Selecting the optimal fiber structured cabling system for your data center & storage area network


Introduction to new standards-based data center & storage area network design – Information generation & storage trends – Data center & storage area network growth – Introduction to the new TIA-942 data center standard

4


New Information Generation Significant Annual Increases

2002 new information production: 5 exabytes – 1 exabyte – 1,000,000,000,000,000,000 bytes – New digital information = 1 Library of Congress every 15 minutes

Four primary physical media – – – –

Print Film Magnetic Optical

– – – –

Telephone Radio Television Internet

New information doubled in last 3 years

350% more information communicated than stored (2002: 18 exabytes) Four electronic channels

Source: “How Much Information 2003?”, School of Information Management & Systems at University of California at Berkeley 5


Why is Network Traffic Growing? A Global Bandwidth “Binge”

applications growing

62% new capacity added in 2003 42% increase in bandwidth demand in 2004

Demand for video could strain networks 2002

Source: TeleGeography, April 2005


2003

Worldwide Available Bandwidth

New bandwidth intensive

Why Are Data Center & SANs Growing? U.S. Legislation & Recommendations

7

Sarbanes-Oxley Act Health Insurance Portability & Accountability Act (HIPAA) Graham-Leach-Bliley Financial Services Modernization Act U.S Federal Reserve Securities & Exchange Commission – Rule 17a SB 1386 - California



8


Definitions

Data Center & Storage Area Network Data Center – “The factory floor of the information age” – ISP: Specialized facility that houses web sites & provides data serving & other services for other companies – Enterprise: Central data processing facility and/or the group of people who manage the enterprise’s data processing & networks Source: http://www.whatis.com 9


Storage Area Network (SAN) – High-speed special purpose network (or subnetwork) that interconnects different kinds of data storage devices with associated data servers on behalf of a larger network of users – Usually located in Data Center

The Integrated Enterprise Network LAN, Data Center & SAN

Physical layer – Copper & optical fiber cabling subsystems

Interconnect devices – Hubs, switches & directors

Translation devices – – – –

Host bus adapters Routers Gateways Bridges

10 11/18/05 JS Copyright © 2005 Ortronics/Legrand. All rights reserved.

Data Center Growth Rapid & Significant

Large enterprise 50% yearly data growth Undergoing major technological shifts $7.4 billion market by 2009

Sources: Yankee Group & IDC reports, 2004 & 2005


Networked Storage Growth

Outpaces Overall Storage Market Growth

Network Attached Storage (NAS) & Storage Area Networks (SANs) – 2004: 50% of overall storage market – 12% CAGR vs. 5% – 2005: 38% Fibre Channel port shipment growth

Dollars invested – 18% of total I.T. budget – 60% of hardware budget Source: Dell ‘Oro Group 2005, iSuppli Corporation, 2004 11/18/05 JS Copyright © 2005 Ortronics/Legrand. All rights reserved.

140.0 120.0 100.0 80.0 60.0 40.0 20.0 0.0

2004

2005

Fibre Channel Port Shipments

Data Center/SAN Media Mix Estimated Percentages Copper & Fiber

Data Center Media Mix

SAN Media Mix

20%

10%

80% Copper Fiber


90% Copper Fiber

Fibre Channel Technology in SANs Short Wavelength VCSELs the Dominant Device

20,000 15,000 8 Gbps 4 Gbps 2 Gbps 1 Gbps

10,000 5,000 0

2001 2002 2003 2004 2005 2006 2007 2008

Fibre Channel Units in Thousands Source: High Speed Optical Data Link Modules, Market Review & Forecast, Strategies Unlimited, 2002 14 11/18/05 JS Copyright © 2005 Ortronics/Legrand. All rights reserved.

Remote Data Centers & SANs Cost-effective DWDM/CWDM Technology

Servers

Servers

Fiber Service Platform

Data Center

FSP Management Suite

Channel director

Fiber Service Platform

CWDM or DWDM over single-mode fiber

Fiber Service Platform Storage Source: Lightwave, January 2004, Todd Bundy, ADVA Optical Networking 15 11/18/05 JS Copyright © 2005 Ortronics/Legrand. All rights reserved.

Channel director

Backup Center

Fiber Service Platform Storage

Data Center Upgrade Example For 50% Annual Storage Capacity Growth

10,000 ft2 data center 50% annual capacity increase typical Doubling of floor space required every 3-5 years Data center floor space cost: $700-1200/ft2 Upgrade cost: $8-12 million over 3 year period

Source: The Meta Group, “Room at the Data Center?” 8-01 16 11/18/05 JS Copyright © 2005 Ortronics/Legrand. All rights reserved.

Insufficient DC/SAN Infrastructure Investment The Costs are Staggering

Ramifications – Minimized customer transactions, interactions & sales volumes – Decreased revenues

Network downtime estimates: – Pay-per-view TV operator: $125,000 per hour – Credit card authorization company: $2,600,000 per hour – Retail brokerage: $6,400,000 per hour Source: Lightwave, January 2004 Todd Bundy, ADVA Optical Networking 17 11/18/05 JS Copyright © 2005 Ortronics/Legrand. All rights reserved.



Premises Structured Cabling System TIA/EIA-568-B Defines 7 Subsystems

Work Area Horizontal Telecom Room

Backbone Entrance Facilities Equipment Room Administration


Do We Really Need Another Standard? Don’t We Already Have Too Many?


Data Center Cabling Standards

Structured Cabling Systems for the Data Centers & SANs

TIA-942 – “Telecommunications Infrastructure Standard for Data Centers” – Published: April 2005 – Order from Global Engineering Documents (www.global.ihs.com)

CENELEC – EN 50173-5

• “Information technology - Generic cabling systems – Part 5: Data Centres” • Expected publication: Early 2006 – EN 50174-2 Amendment

• Adds Annex on Data Center planning & installation

ISO/IEC – “Generic Cabling for Data Centres – Proposed” – ISO/IEC JTC-1/SC 25/WG 3 21 11/18/05 JS Copyright © 2005 Ortronics/Legrand. All rights reserved.

TIA-942 Data Center Standard Objective

Requirements & guidelines for the design & installation of a data center or computer room

Intended for use by designers needing thorough understanding of data center design

Comprehensive document Cabling Network Design Location Access

Architectural design Fire protection Environmental design Water intrusion Electrical design Redundancy


Data Center Structured Cabling System 9 Elements Comprise TIA-942

1. 2. 3. 4. 5. 6. 7. 8. 9.

Computer room Telecommunications room Entrance room

Spaces

Main distribution area Horizontal distribution area Zone distribution area Equipment distribution area Backbone cabling Horizontal cabling


Cabling subsystems

Data Center Cabling

For the Backbone & Horizontal Cabling Subsystems

Backbone subsystem (fiber) – – – – –

Backbone cables Main cross-connects Horizontal cross-connects Mechanical terminations Patch cords

Horizontal subsystem (fiber or copper) – – – –

Horizontal cables Mechanical terminations Patch cords Zone outlet or consolidation point (optional)


TIA-942 Data Center Standard Supported Architectures

Basic data center topology Distributed data center topology Reduced data center topology Centralized fiber optic cabling topology


Basic Data Center Topology And TIA/EIA-568-B Counterparts

Entrance Room – Analogy: “Entrance Facility”

Main Distribution Area (MDA) – Analogy: “Equipment Room”

Horizontal Distribution Area (HDA) – Analogy: “Telecom Room”

Zone Distribution Area (ZDA) – Analogy: “Consolidation Point”

Equipment Distribution Area (EDA) – Analogy: “Work Area”


Distributed Data Center Topology With Multiple Entrance Rooms

May be required for large


data centers Circuit distance limitations may require multiple entrance rooms Primary entrance room has no direct connections to HDA Secondary entrance room may be directly connected to HDA conditionally

Reduced Data Center Topology For Many Enterprise Installations

HDA combined with MDA Telecom room can also be consolidated into MDA

Copper or fiber in the horizontal


Data Center Centralized Fiber Cabling Significant Cost Savings Possible

Alternative to optical cross

connection in the horizontal distribution area No electronics in horizontal distribution area (HDA) Centralized electronics Cost reduction factors – – – – –

Visit TIA Fiber Optics LAN Section web site for information on centralized fiber cabling: www.fols.org 29 11/18/05 JS Copyright © 2005 Ortronics/Legrand. All rights reserved.

Smaller, simpler HDA Faster & easier installation & testing Fewer idle ports Centralized administration Simplified moves, adds & changes

TIA-942 & Draft EN 50173-5 Compared Similarities & Differences

TIA/EIA-942 (2005)

EN 50173-5 (2006)

Horizontal Distribution Area

Main Distribution Area

Zone Distribution Area

EO ZD Equipment Distribution Area

LDP

EO EO

MD EO ZD ENI

Entrance Room

EO EO

ENI

Connection point to the outside world

Equipment Network Interface (ENI)

Functional distribution element within the MDA

Main Distributor (MD)

Functional distribution element within the HDA

Zone Distributor (ZD)

Connection point within the ZDA

Local Distribution Point (LDP)

Connection point within the EDA

Equipment Outlet (EO)

Courtesy: Mike Gilmore, e-Ready Building Limited (2004) 30 11/18/05 JS Copyright © 2005 Ortronics/Legrand. All rights reserved.

Media Selection

Design Considerations per TIA-942

Flexibility with respect to supported services Required useful life of cabling Facility site/size & occupant population Channel capacity within the cabling system Equipment vendor recommendations or specifications Same facility architecture if different media types used


Data Center Cabling Recommendations Transmission Media – Normative “Required”

100-ohm twisted-pair copper cable – Category 3 or 5e allowed – Category 6 recommended

Multimode fiber optic cable – 62.5/125 µm or 50/125 µm allowed – 50/125 µm 850 nm laser optimized multimode fiber recommended

Singlemode optical fiber cable 75-ohm coaxial cable – Type 734 & 735 cable – Type T1.404 coaxial connector


Per TIA-942

Data Center Cabling

Design Recommendations (Informative) - “Optional”

Copper design (informative) – Adequate spacing for labeling on each patch panel – Label each port per Annex B and ANSI/TIA/EIA-606-A

Fiber design (informative) – Installation time reductions – Multi-fiber increments & multi-fiber connectors – Pre-calculated, pre-terminated multi-fiber ribbon assemblies – Consider performance effects of additional connections

Per TIA-942 33 11/18/05 JS Copyright © 2005 Ortronics/Legrand. All rights reserved.

Data Center Standard

Multiple Benefits to Designers & Managers

Consistency in design Predictable level of performance More choice in the marketplace Interoperability between different vendors’ products Economies of scale


Introduction to new standards-based data

center & storage area network design Selecting the optimal fiber structured cabling system for your data center & storage area network


Agenda

Selecting the Optimal Fiber SCS for your Data Center & SAN

Why is the choice of a fiber structured cabling system so important in data centers & SANs?

Guidelines for selecting the fiber termination method Connecting the system elements together


Higher Speed Networks More Demanding Fiber, Cable & Connectivity Choices Critical

– Fiber cable plant loss budgets continue to decrease – Widely perceived 2.6 dB budget for 10 Gbps Ethernet & Fibre Channel – Installation techniques more challenging – Advanced fiber SCS technology provides new options


Multimode Fibers Industry Standard Types

Fiber type

Wavelength (nm)

Max Loss

Min Bandwidth (MHzxkm)

(dB/km)

OFL

EMB

1 Gb/s Reach

10 Gb/s Reach

(meters)

(meters)

62.5 µm (OM1)

850 1300

3.5 1.5

200 500

n.s.* n.s.

275 550

33 300

50 µm (OM2)

850 1300

3.5 1.5

500 500

n.s. n.s.

550 550

82 300

850-nm 10G LaserOptimized 50 µm (OM3)

850 1300

3.5 1.5

1500 500

2000 n.s.

1000 600

300 300

OM1, OM2, OM3 designations Per ISO/IEC 11801, 2nd Edition *n.s. = Not specified 38 11/18/05 JS Copyright © 2005 Ortronics/Legrand. All rights reserved.

OFL = Overfilled launch EMB = Effective modal (laser) bandwidth

Multimode Fiber Types Bandwidth Comparison

62.5/125 µm 200/500 MHz-km

50/125 µm 500/500 Hz-km

OM3 (50/125 µm) 2,000/500 MHz-km


Ethernet Fiber Loss Budgets Decreasing Due to Ever Increasing Speeds

Year

Application

Data Rate

Designation

Standard

Cable Plant Loss Budget (db)

Early 80’s

Ethernet

10 Mbps

10BASE-FL

IEEE 802.3

12.5

Early 90’s

Fast Ethernet

100 Mbps

100BASE-FX

IEEE 802.3

11.0

Late 90’s

Short Wavelength Fast Ethernet

10/100 Mbps

100BASE-SX

TIA/EIA-785

4.0

2000

1 Gigabit Ethernet

1,000 Mbps

1000BASE-SX

IEEE 802.3z

3.56

2004

10 Gigabit Ethernet

10,000 Mbps

10GBASE-SR*

IEEE 802.3ae

2.60

Insertion loss values are for maximum distance specified in the standard & can vary based on the distance & number of connections


Fibre Channel Loss Budgets Also Decreasing The Predominant Protocol in Storage Area Networks

Application

Data Rate

Designation

Supported Cable Plant Loss Distance (m)* Budget (db)

Fibre Channel

1 Gbps

100-M5-SN-I

0.5 – 860*

4.62

Fibre Channel

2 Gbps

200-M5-SN-I

0.5 – 500*

3.31

Fibre Channel

4 Gbps

400-M5-SN-I

0.5 – 270*

2.48

Fibre Channel

10 Gbps

1200-M5-SN-I

0.5 – 300*

2.6

Insertion loss values are for maximum distance specified in the standard & can vary based on the distance & number of connections *Supported distances using 2,000 MHz-km 850 nm laser optimized 50 µm multimode fiber 41 11/18/05 JS Copyright © 2005 Ortronics/Legrand. All rights reserved.

Key Elements of a Robust DC/SAN To Support Multiple Generations of Electronics

Reliable, high bandwidth networks required The optimal solution: systems engineered, manufactured & independently verified to meet & exceed worldwide standards requirements

Performance of individual fiber network elements critical 1. Electronics: Fully qualified devices with high laser coupling efficiency 2. Fiber: Low DMD or high EMBc 3. Cable: Low attenuation 4. Apparatus: Reduced insertion loss per mated pair


Ethernet & Fibre Channel Transceivers Look For High Laser Coupling Efficiency

850 nm operating wavelength more cost effective

Small Form Factor Pluggable (SFP) modules dominant

Broad manufacturer availability Fully qualified devices recommended

Most power is inside 9-38 µm “donut”


850 nm laser spot projected on 50 µm fiber core

Cable Design & Manufacturing Can Affect 10 Gbps Performance

Cables with low attenuation tested & verified not to degrade fiber performance in 10 Gbps networks

Tight control over buffer uniformity & concentricity for highest connector performance


Small-Form-Factor Fiber Connectors High Density Critical for Space-Limited Data Centers

OptiJack

VF-45

MT-RJ


LX.5

LC

Optimum Connector Performance Creates a “Lens” at the Tip of the Connector

Look for polishing techniques yielding ideal tip geometry End-face quality a key factor for maximum link performance Objectives: minimum insertion loss; maximum return loss

Poor polish: fiber depressed into ferrule, causing poor performance

Ideal polish: connector ferrule/fiber end-face scan showing ideal contour

Maximum 10 Gbps system performance 46 11/18/05 JS Copyright © 2005 Ortronics/Legrand. All rights reserved.

10 Gbps Multimode Cabling System

IEEE Link Model 850 nm Serial, 2,000 MHz-km MMF

Power budget consumed by – Cross noise – Receiver eye opening – Relative intensity noise – Mode partition noise – Inter-symbol interference (ISI) – Channel insertion loss

Robust fibre solutions – Low insertion loss – Low Differential Mode Delay (DMD) – “Borrow” budget from other areas for channel insertion loss 47 11/18/05 JS Copyright © 2005 Ortronics/Legrand. All rights reserved.

Total Available Power in dB

various impairments

Cross noise Receiver eye opening Relative intensity noise Mode partition noise

Inter-symbol interference (ISI)

Channel insertion loss (ChIL)

75% of total penalty

Reallocating ISI Loss Penalty Using State-of-the-Art Fiber Technology

Ideal application to MTP/MPO-based systems Ideal for data centers & SANs Exchange ISI for channel insertion loss


Easy to Configure Data Center Systems

Fibre Channel Example, 2 MTP®/MPOs & 3 LC Connections

Example SAN or data center fiber link design

LC

FC-AL Hub MTP/MPO

Standard OM-3 fiber may

Modular pre-terminated optical cassette systems

not support number of connections

State-of-the-art LOMF

Ribbon backbone cable or distribution cable MTP/MPO

fiber & low insertion loss connectors

LC

LC

Patch panel FC Switch

(interconnect) MT


Agenda

Selecting the Optimal Fiber SCS for your Data Center & SAN




Available Fiber Termination Methods Multiple Solutions for Data Centers & SANs

Multimode Cassette-based Pre-terminated Field-terminated


Single-mode Cassette-based Pre-terminated Field-terminated

Cassette-Based DC/SAN Solution Ribbon Backbone or Ribbonized Fiber Cable

Definition: Ribbon backbone or reduced diameter loose tube cable terminated with MTP/MPO connectors designed to interface with optical cassette system

Ideal for use in the Zone Distribution Area (ZDA)


Cassette-Based DC/SAN Solution Selection Criteria

Design Element

Advantage

Highly reliable

Ideal for data centers & SANs

Factory terminated solution

Guaranteed optical performance

Integrated system

Designed for interoperability

Easy, fast, error-free installation

Significant cost savings

Distributes optical signals to common LC & SC interfaces

Integration with existing systems

Cassette supports multiple fibers

Greatly simplified connectivity

Compliant with TIA SP-3-4424-AD7*

Standards-based system

*to become TIA/EIA-568-B.3, Addendum 7 53 11/18/05 JS Copyright © 2005 Ortronics/Legrand. All rights reserved.

Pre-Terminated DC/SAN Solution

Time & Labor Saving Backbone Cable Designs

Definition: Backbone cable with factory installed connectors extending from rear of adapter panel to mating end of another adapter panel in another rack


Pre-Terminated DC/SAN Solution Selection Criteria

Design Element

Advantage

Factory terminated solution

Guaranteed optical performance

Multiple optical links contained in one sheath

Facilitates cable routing & dressing

Fast & easy installation

Reduced on-site time & labor costs

Smaller overall cable diameter & cross-sectional areas

Better air flow & less congestion

Distribution, armored, or reduced diameter plenum cable

Solution for every application


Field-Terminated DC/SAN Solution Time & Labor Saving Backbone Cable Designs

Definition: Field-installable fiber optic connectors installed on-site with local installation crews

LC

SC Fiber connector field termination kit


Adhesive/polish Termination Popular Field-Installable Connector

Cordage secured in connector by adhesive method

Adhesive holds the fiber firmly in the connector ferrule Excess fiber is scored & removed The remaining fiber and adhesive is polished down to the end-face of the ferrule Fiber end polished to same radius curve as ferrule end-face


No-polish Termination Reduced On-Site Labor Costs

Mechanical splice joins factory-installed fiber stub and fiber being terminated

Adhesive holds the factory-installed fiber stub firmly in the connector ferrule

Drop of index-matching gel provides optical interface for cleaved fibers


Fiber end polished and tested at the factory to same radius curve as ferrule end-face

Field-Terminated DC/SAN Solution Selection Criteria

Design Element

Advantage

Wide range of popular connector types available

Integration with existing systems

Connector choices may unique to specific data center/SAN

Minimize hybrid patch cords

No polish connector requires less consumables

Reduced on-site labor costs

Anaerobic adhesive connector requires no heating oven

Reduced on-site labor costs

State-of-the-art fiber connector designs

Good choice for skilled installers

Look for complete, easy to understand instructions

Fewer installation errors


Agenda

Structured Cabling Systems in Data Centers & SANs




Cassette-Based Data Center Solution Channel Components for 10 Gbps Multimode System


Pre-Terminated Data Center Solution Channel Components for Single-mode System

Cable Management Rack

Cable Management Rack SC Adapter Panel 12 Fiber

LC Adapter Panel 24 Fiber

Rack Mount Fiber Patch Cabinet

Single Mode Duplex Patch Cord



Single Mode Duplex Patch Cord

Field-Terminated Data Center Solution Channel Components for 10 Gbps Multimode System




SC Adapter Panel 12 Fiber

LOMF Fiber Patch Cord

LOMF Fiber Patch Cord LC Adapter Panel 24 Fiber



Summary

Data Centers & SANs: Rapidly Growing SCS Applications

Vast amounts of new information being created, communicated & stored

Legislation & other business priorities impacting data center & storage area network growth

The TIA-942 Data Center Standard applies structured cabling principles as TIA/EIA-568 did for commercial buildings

Careful choice of structured cabling system products should span multiple of generations of data center systems


Designing a Fiber Structured Cabling System for the Data Center


Designing a Fiber Structured Cabling System for the Data Center

Designing a Fiber Structured Cabling System for the Data Center

Suggest Documents

Structured Cabling System (SCS)

Key Considerations for Selecting Data Center Fiber Cabling - Plexxi

Structured Cabling

Using the Structured Cabling System for Security Applications

INTRODUCTION TO STRUCTURED CABLING

Structured Cabling Guide

nCompass OM3/OM4 Fiber Cabling System - Anixter

Structured Cabling Systems

Structured Cabling Systems - bimel

A distributed optical fiber sensing system for data center thermal ...

Structured Cabling Systems - bimel [PDF]

structured cabling pdf - Google Drive

Structured Cabling Systems - Sion Net

the vital role of cabling in the data center - Anixter

the vital role of cabling in the data center - Anixter

Designing a System for Structured Assessment of Compliance Risk

Leviton Voice & Data Division T-700: Structured Cabling Systems ...

Designing a Digital Preservation System for a NASA Center for

Designing a Reliable Cabling Infrastructure for ... - Accu-Tech

structured cabling tutorial pdf - Google Drive

Structured Cabling Supplement - Faculty Web Sites

Data Center Cabling Standards Keeping Pace with ... - Bicsi

Bigtable: A Distributed Storage System for Structured Data

Category 6 UTP - Cabling System