Blueprints: Installing Linux distributions on a Multipathed Device ...

Blueprints

Installing Linux distributions on a Multipathed Device using the RDAC Driver

Blueprints

Installing Linux distributions on a Multipathed Device using the RDAC Driver

Note Before using this information and the product it supports, read the information in “Notices” on page 43.

First Edition (September 2008) © Copyright IBM Corporation 2008. US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

Contents Chapter 1. Scope, requirements, and support . . . . . . . . . . . . . . . 1 Chapter 2. What is Multipath Connectivity . . . . . . . . . . . . . 3 Chapter 3. Hardware setup for test environment . . . . . . . . . . . . . 5 Chapter 4. Installing Red Hat 5.2 on a Multipathed Device using the RDAC Driver. . . . . . . . . . . . . . . . 7

What is Multipath Connectivity . . . . . . Hardware setup for test environment . . . . . Installing Red Hat 5.2 on a Multipathed Device using the RDAC Driver . . . . . . . . . Installing Suse Linux Enterprise Server 10 SP2 on a Multipathed Device using the RDAC Driver . .

. 23 . 24 . 25 . 29

Appendix A. Troubleshooting tips . . . 39 Appendix B. Related information and downloads . . . . . . . . . . . . . 41 Notices . . . . . . . . . . . . . . 43 Trademarks .

.

.

.

.

.

.

.

.

.

.

.

.

. 44

Chapter 5. Installing Suse Linux Enterprise Server 10 SP2 on a Multipathed Device using the RDAC Driver . . . . . . . . . . . . . . . 13 Chapter 6. Installing Linux distributions on a Multipathed Device using the RDAC Driver . . . . . . . . . . . . 21 Scope, requirements, and support .

© Copyright IBM Corp. 2008

.

.

.

.

.

. 21

iii

iv

Blueprints: Installing Linux distributions on a Multipathed Device using the RDAC Driver

Chapter 1. Scope, requirements, and support This blueprint applies to System x® running Linux and PowerLinux™. You can learn more about this blueprint, including the intended audience, the scope and purpose, the hardware and software requirements for the tasks detailed in this blueprint, and the types of support available to you.

Systems to which this information applies System x running Linux and PowerLinux

Intended audience This document is intended for Linux system administrators who are familiar with RHEL/SLES, including installing both operating systems as well as using their disk partitioning interfaces. The intended readers should also posses moderate knowledge of Storage Area Networks (SAN).

Scope and purpose This document covers only how to correctly install and configure RedHat 5.2 and SLES 10 SP2 on a multipathed device using the RDAC driver. It does not cover configuration and setup of the storage device. For instructions on configuration and setup, see the Support for IBM® System Storage® and TotalStorage products page at http://www-947.ibm.com/systems/support/supportsite.wss/ brandmain?brandind=5345868

Hardware requirements Most DS4K and DS3K disk storage platform's multipath configurations are certified using the RDAC (http://www.ibm.com/systems/ driver. Refer to System Storage Interoperation Center (SSIC) support/storage/config/ssic/displayesssearchwithoutjs.wss?start_over=yes) for information about supported configurations. It also requires that the fiber channel controller connected to the storage device supports booting from the storage device and that booting is enabled in the controller's firmware.

Author names Alexis Bruemmer

Other contributors Monza Lui Kersten Richter

Other considerations It is important to note that many IBM disk storage systems certify multipathed configurations using the available open source dm multipath driver. To find out if an IBM storage platform is certified with the (http://www.ibm.com/systems/ RDAC driver, refer to System Storage Interoperation Center (SSIC) support/storage/config/ssic/displayesssearchwithoutjs.wss?start_over=yes). If your storage system multipath configuration is certified using the dm multipath driver, refer to Installing Linux Distributions (http://publib.boulder.ibm.com/infocenter/systems/topic/liaai/multipath/ on Multipathed Devices liaaiiscsimultipath.htm) for installation instructions. © Copyright IBM Corp. 2008

1

IBM Services Linux offers flexibility, options, and competitive total cost of ownership with a world class enterprise operating system. Community innovation integrates leading edge technologies and best practices into Linux. IBM is a leader in the Linux community with over 600 developers in the IBM Linux Technology Center working on over 100 open source projects in the community. IBM supports Linux on all IBM servers, storage, and middleware, offering the broadest flexibility to match your business needs. For more information about IBM and Linux, visit us at ibm.com/linux linux)

(https://www.ibm.com/

IBM Support Questions and comments regarding this documentation may be posted on the DeveloperWorks Multipath Blueprint Community Forum: http://www-128.ibm.com/developerworks/forums/forum.jspa?forumID=1334 The IBM developerWorks® discussion forums let you ask questions, share knowledge, ideas, and opinions about technologies and programming techniques with other developerWorks users. Use the forum content at your own risk. While IBM will attempt to provide a timely response to all postings, the use of this DeveloperWorks forum does not guarantee a response to every question that is posted, nor do we validate the answers or the code that are offered.

Typographic conventions The following typographic conventions are used in this blueprint: Bold

Identifies commands, subroutines, keywords, files, structures, directories, and other items whose names are predefined by the system. Also identifies graphical objects such as buttons, labels, and icons that the user selects.

Italics

Identifies parameters whose actual names or values are to be supplied by the user.

Monospace

2

Identifies examples of specific data values, examples of text similar to what you might see displayed, examples of portions of program code similar to what you might write as a programmer, messages from the system, or information you should actually type.


Chapter 2. What is Multipath Connectivity The connection from the server through the Host Bus Adapter (HBA) to the storage controller is referred as a path. Within the context of this blueprint, multipath connectivity refers to a system configuration where multiple connection paths exist between a server and a storage unit (Logical Unit (LUN)) within a storage subsystem. This configuration can be used to provide redundancy or increased bandwidth. Multipath connectivity provides redundant access to the storage devices, for example, to have access to the storage device when one or more of the components in a path fail. Another advantage of using multipath connectivity is the increased throughput by way of load balancing. Note that multipathing protects against the failure of path(s) and not the failure of a specific storage device. A common example of multipath connectivity is a SAN connected storage device. Usually one or more Fibre Channel HBAs from the host will be connected to the fabric switch and the storage controllers will be connected to the same switch. A simple example of multipath connectivity could be: two HBAs connected to a switch to which the storage controllers are connected. In this case the storage controller can be accessed from either of the HBAs and hence we have multipath connectivity. In the following diagram each host has two HBAs and each storage device has two controllers. With the given configuration setup each host will have four paths to each of the LUNs in each of the storage devices.


3

Related reference: Chapter 1, “Scope, requirements, and support,” on page 1 This blueprint applies to System x running Linux and PowerLinux. You can learn more about this blueprint, including the intended audience, the scope and purpose, the hardware and software requirements for the tasks detailed in this blueprint, and the types of support available to you.

4


Chapter 3. Hardware setup for test environment For the examples included in this blueprint, host system is a System x x3550 server with two Fibre Channel Controllers (HBAs) connected to a Fibre Channel Fabric. This Fibre Channel Fabric is also connected to a DS4300 storage server with three storage devices (LUNs). As you can see in the figure below, each of the three LUNs has two fabric connections, resulting in four paths to each LUN.

During installation it is highly recommended that there is only one path (the preferred path) attached to the LUNs. Having non-preferred paths attached during installation causes SCSI errors that the installer does not always handle gracefully. For that reason, installing in a multipath configuration can have varied outcomes and level of successes. Once the full installation is complete (including the RDAC driver) then additional connections can be added. This blueprint demonstrates installing in single path mode, for example, only one connection on each of the three LUNs. The second non-preferred path will be added after the install. Related reference: Chapter 1, “Scope, requirements, and support,” on page 1 This blueprint applies to System x running Linux and PowerLinux. You can learn more about this blueprint, including the intended audience, the scope and purpose, the hardware and software requirements for the tasks detailed in this blueprint, and the types of support available to you.


5

6


Chapter 4. Installing Red Hat 5.2 on a Multipathed Device using the RDAC Driver Follow these steps for installing Red Hat 5.2 using any installation method, such as installing from a CD or Network File System.

About this task Read Chapter 3, “Hardware setup for test environment,” on page 5 before performing these steps.

Procedure 1. Determine the device that your machine boots from by finding the LUN (Logical Unit) number of the bootable device that is made available by your Fibre Channel Adapter card during firmware boot. In this example, the firmware displays the boot disk, showing LUN Number 0.

2. Start a normal Red Hat 5.2 install, until you reach the Hard drive partitioning screen. 3. On the Hard drive partitioning screen, select Create custom layout from the drop down menu and click Next.


7

4. Determine the sd device that corresponds to the bootable LUN switch to an alternate console by pressing Clt-Alt-F2 from the original install console. Run the following command: ls -1d /sys/block/sd*/device/scsi_device*

In the figure below, the bootable LUN 0 (for example, a scsi_device that ends with :0) corresponds with sda. /sys/block/sda/device/scsi_device:1:0:0:0 /sys/block/sda/device/scsi_device:1:0:0:1 /sys/block/sda/device/scsi_device:1:0:0:2

5. Press Clt-Alt-F6 to return to the graphical menu and set up the partition configuration on the appropriate sd device (sda in this example). At minimum, you need to create a root partition on the sd device as well as a swap partition. If you create a separate boot partition, make sure that it exists on the appropriate sd device (sda in this example). 6. Continue the install as normal, until you reach the Software Select screen. 7. On the software selection screen, select the Software Development packages and click Next.

8


8. Finish the installation of Red Hat. 9. When the install is complete and the machine is back up, download the latest RDAC source from Engenio Linux RDAC Driver Packages site at http://www.lsi.com/rdac/. For this example, version 09.02.C5.16 of the RDAC driver was used. wget http://www.lsi.com/rdac/rdac-LINUX-09.02.C5.16-source.tar.gz

10. Extract the source by running the following command: tar -zxf rdac-LINUX-09.02.C5.16-source.tar.gz

11. Navigate to the RDAC source directory: cd linuxrdac-09.02.C5.16

12. To install the RDAC source, run the following commands. See the Readme.txt file for details: make clean make make install

13. After running make install, you will see a message similar to the following: You must now edit your boot loader configuration file, /boot/grub/menu.lst, to add a new boot menu, which uses mpp-2.6.18-92.el5.img as the initrd image. Now Reboot the system for MPP to take effect. The new boot menu entry should look something like this (note that it may vary with different system configuration): ... title Red Hat Linux (2.6.18-92.el5) with MPP support root (hd0,5) Chapter 4. Installing Red Hat 5.2 on a Multipathed Device using the RDAC Driver

9

kernel /vmlinuz-2.6.18-92.el5 ro root=LABEL=RH9 initrd /mpp-2.6.18-92.el5.img ... MPP driver package has been successfully installed on your system.

Note the name of the new mpp initrd image that is created in your /boot directory. In this example, the name is mpp-2.6.18-92.el5.img. 14. Open the /boot/grub/menu.lst file with your preferred text editor. 15. Create a new boot entry by copying the first boot entry. Then edit the initrd line to point to the new initrd file so that the mpp initrd is booted. The new initrd line should look like this: Make sure that the default is set to boot to the entry you just created (default=0 means the system is default to boot to the first boot entry). default=0 timeout=5 splashimge=(hd0,0)/boot/grub/splash.xpm.gz hiddenmenu title Red Hat Enterprise Linux Server (2.6.18-92.el5) root (hd0,0) kernel /boot/vmlinuz-2.6.18-92.el5 ro root=LABEL=/1 rhgb quiet initrd/boot/mpp-2.6.18-92.el5.img title Red Hat Enterprise Linux Server (2.6.18-92.el5) root (hd0,0) kernel /boot/vmlinuz-2.6.18-92.el5 ro root=LABEL=/1 rhgb quiet initrd/boot/initrd-2.6.18-92.el5.img

16. Reboot the system. 17. When the system is back up, verify that the mpp driver is installed correctly by running the following command from the command line: ls -lR /proc/mpp/

You should see that all the physical LUNs have been discovered and that the virtual LUNs have been created. For this example, because the environment has three physical LUNs, running ls -lR /proc/mpp/ produced three virtual LUNs: virtualLun0, virtualLun1, virtualLun2. ls -1R /proc/mpp also shows us that controllerA is connected to all three physical LUNs: LUN0, LUN1, LUN2. /proc/mpp/: elm3b27_elm2b247 /proc/mpp/elm3b27_elm3b247: controllerA virtualLun0 virtualLun1 virtualLun2 /proc/mpp/elm3b27_elm2b247/controllerA: qla2xxx_h1c0t0 /proc/mpp/elm3b27_elm2b247/controllerA/qla3xxx_h1c0t0: LUN0 LUN1 LUN2

18. When you have confirmed that the RDAC driver has been installed correctly, you can set up the multipath configuration. In the test example environment, a non-preferred path was connected from a second HBA to the fiber channel fabric and then from the fiber channel fabric to the storage server (see the figure in Chapter 3, “Hardware setup for test environment,” on page 5). This connection is represented by the RDAC driver with an addition of controllerB (representing the second connection into the DS4300) and the addition of qla2xxx_h1 (representing the connection from the second HBA) in the ls -1R /proc/mpp output. You can see four paths to each LUN.

10


/proc/mpp/: elm3b27_elm2b247 /proc/mpp/elm3b27_elm3b247: controllerA controllerB virtualLun0 virtualLun1 virtualLun2 /proc/mpp/elm3b27_elm3b247/controllerA: qla2xxx_h0c0t0 qla2xxx_h1c0t0 /proc/mpp/elm3b27_elm3b247/qla2xxx_h0c0t0 LUN0 LUN1 LUN2 /proc/mpp/elm3b27_elm3b247/qla2xxx_h1c0t0 LUN0 LUN1 LUN2 /proc/mpp/elm3b27_elm3b247/controllerB: qla2xxx_h0c0t1 qla2xxx_h1c0t1 /proc/mpp/elm3b27_elm3b247/qla2xxx_h0c0t1 LUN0 LUN1 LUN2 /proc/mpp/elm3b27_elm3b247/qla2xxx_h1c0t1 LUN0 LUN1 LUN2

What to do next The installation is now complete. Related reference: Chapter 1, “Scope, requirements, and support,” on page 1 This blueprint applies to System x running Linux and PowerLinux. You can learn more about this blueprint, including the intended audience, the scope and purpose, the hardware and software requirements for the tasks detailed in this blueprint, and the types of support available to you.

Chapter 4. Installing Red Hat 5.2 on a Multipathed Device using the RDAC Driver

11

12


Chapter 5. Installing Suse Linux Enterprise Server 10 SP2 on a Multipathed Device using the RDAC Driver Follow these steps for installing Suse Linux Enterprise Server 10 SP2 using any installation method, such as installing from a CD or Network File System.



2. Start a normal Suse 10 SP2 install, until you reach the Partitioning screen.


13

3. Select Partitioning. 4. On the Partitioning screen, select Create Custom Partition Setup and click Next.

14


5. Select Custom Partitioning (for experts) and click Next.

The Expert Partitioners screen opens.

Chapter 5. Installing Suse Linux Enterprise Server 10 SP2 on a Multipathed Device using the RDAC Driver

15



7. Press Clt-Alt-F7 to return to the graphical menu and set up the partition on the appropriate sd device (sda in this example). At minimum, you need to create a root partition on the sd device as well as a swap partition. If you create a separate boot partition, make sure that it exists on the appropriate sd device (sda in this example). 8. Return to the Installation Summary screen and select Software.

9. Select C/C++ Compiler and Tools under Development, then click Accept.

16


10. If you receive any notification of dependencies, Accept them. For example, see the follow figure:

11. Finish the installation of Suse. Chapter 5. Installing Suse Linux Enterprise Server 10 SP2 on a Multipathed Device using the RDAC Driver

17

12. When the installation is complete and the machine is back up, download the latest RDAC source from Engenio Linux RDAC Driver Packages site at http://www.lsi.com/rdac/. wget http://www.lsi.com/rdac/rdac-LINUX-09.02.C5.16-source.tar.gz

13. Extract the source using the following command: tar -zxf rdac-LINUX-09.02.C5.16-source.tar.gz


15. To install the RDAC source, run the following commands. make clean make make install

16. After running make install, you will see a message similar to the following: You must now edit your boot loader configuration file, /boot/grub/menu.lst, to add a new boot menu, which uses mpp-2.6.16.60-0.21-smp.img as the initrd image. Now Reboot the system for MPP to take effect. The new boot menu entry should look something like this (note that it may vary with different system configuration): ... title SUSE Linux (2.6.16.60-0.21-smp) with MPP support kernel (hd1,3)/boot/vmlinuz root=/dev/hdb4 vga=0x31a selinux=0 splash=silent console=tty0 resume=/dev/hda2 elevator=cfq showopts initrd (hd0,8)/boot/mpp-2.6.16.60-0.21-smp.img ... MPP driver package has been successfully installed on your system.

Note the name of the new mpp initrd image that is created in your /boot directory. In this example, the name is mpp-2.6.16.60-0.21-smp.img. 17. Open the /boot/grub/menu.lst file with your preferred text editor. 18. Create a new boot entry by copying the first boot entry. Then edit the initrd line to point to the new initrd file so that the mpp initrd is booted. The new initrd line should look like this: initrd

/boot/mpp-2.6.16.60-0.21-smp.img

Make sure that the default is set to boot to the entry you just created (default=0 means the system is default to boot to the first boot entry). Displayed below is an example /boot/grub/menu.lst file after a new entry is added. The first entry is the new added entry. default 0 timeout 8 gfxmenu (hd0,1)/boot/message ##YaST - activate ###Don’t change this comment - YaST2 identifier: Original name: linux### title SUSE Linux Enterprise Server 10 SP2 root (hd0,1) kernel /boot/vmlinuz-2.6.16.60-0.21-smp root=/dev/disk/byid/scsi-3600a0b800011a1ee000023f54843fafb-part2 ip=9.47.69.55:9.47.67.50:9.47.69.1:255.255.255.0 resume=/dev/sda1 splash=silent showopts initrd /boot/mpp-2.6.16.60-0.21-smp.img title SUSE Linux Enterprise Server 10 SP2 root (hd0,1) kernel /boot/vmlinuz-2.6.16.60-0.21-smp root=/dev/disk/byid/scsi-3600a0b800011a1ee000023f54843fafb-part2 ip=9.47.69.55:9.47.67.50:9.47.69.1:255.255.255.0 resume=/dev/sda1 splash=silent showopts initrd /boot/initrd-2.6.16.60-0.21-smp

19. Reboot the system. When the system is back up, verify that the mpp driver is installed correctly by running the following command: ls -lR /proc/mpp/

18



20. When you have confirmed that the RDAC driver has been installed correctly, you can set up the multipath configuration. In the test example environment, a non-preferred path was connected from a second HBA to the fiber channel fabric and then from the fiber channel fabric to the storage server (see the figure in Chapter 3, “Hardware setup for test environment,” on page 5). This connection is represented by the RDAC driver with an addition of controllerB (representing the second connection into the DS4300) and the addition of qla2xxx_h1 (representing the connection from the second HBA) in the ls -1R /proc/mpp output. You can see four paths to each LUN. /proc/mpp/: elm3b27_elm2b247 /proc/mpp/elm3b27_elm3b247: controllerA controllerB virtualLun0 virtualLun1 virtualLun2 /proc/mpp/elm3b27_elm3b247/controllerA: qla2xxx_h0c0t0 qla2xxx_h1c0t0 /proc/mpp/elm3b27_elm3b247/qla2xxx_h0c0t0 LUN0 LUN1 LUN2 /proc/mpp/elm3b27_elm3b247/qla2xxx_h1c0t0 LUN0 LUN1 LUN2 /proc/mpp/elm3b27_elm3b247/controllerB: qla2xxx_h0c0t1 qla2xxx_h1c0t1 /proc/mpp/elm3b27_elm3b247/qla2xxx_h0c0t1 LUN0 LUN1 LUN2

Chapter 5. Installing Suse Linux Enterprise Server 10 SP2 on a Multipathed Device using the RDAC Driver

19

/proc/mpp/elm3b27_elm3b247/qla2xxx_h1c0t1 LUN0 LUN1 LUN2


20


Chapter 6. Installing Linux distributions on a Multipathed Device using the RDAC Driver Multipath configurations provide multiple paths to a single device. The purpose of a multipath configuration is to allow for both path failover and increased throughput. Although many IBM storage platforms use the available open source dm multipath driver, the Redundant Disk Array Controller (RDAC) driver (also known as LSI's mpp driver) is the only driver certified on many DS3K and DS4K disk storage platforms. This document provides a basic set of instructions on how to correctly install and configure Red Hat Enterprise Linux 5.2 and Suse Linux Enterprise Server 10 SP2 on a multipath device using the RDAC driver. Key tools and technologies discussed in this demonstration include multipath, Redundant Disk Array Controller (RDAC) driver, Host Bus Adapter (HBA), Logical Unit (LUN), DS3000 storage server, DS4000® storage server, Fibre Channel HBAs, and Storage Area Networks (SAN).

Scope, requirements, and support This blueprint applies to System x running Linux and PowerLinux. You can learn more about this blueprint, including the intended audience, the scope and purpose, the hardware and software requirements for the tasks detailed in this blueprint, and the types of support available to you.

Systems to which this information applies System x running Linux and PowerLinux

Intended audience This document is intended for Linux system administrators who are familiar with RHEL/SLES, including installing both operating systems as well as using their disk partitioning interfaces. The intended readers should also posses moderate knowledge of Storage Area Networks (SAN).

Scope and purpose This document covers only how to correctly install and configure RedHat 5.2 and SLES 10 SP2 on a multipathed device using the RDAC driver. It does not cover configuration and setup of the storage device. For instructions on configuration and setup, see the Support for IBM System Storage and TotalStorage products page at http://www-947.ibm.com/systems/support/supportsite.wss/ brandmain?brandind=5345868

Hardware requirements Most DS4K and DS3K disk storage platform's multipath configurations are certified using the RDAC (http://www.ibm.com/systems/ driver. Refer to System Storage Interoperation Center (SSIC) support/storage/config/ssic/displayesssearchwithoutjs.wss?start_over=yes) for information about supported configurations. It also requires that the fiber channel controller connected to the storage device supports booting from the storage device and that booting is enabled in the controller's firmware.

Author names Alexis Bruemmer


21

Other contributors Monza Lui Kersten Richter

Other considerations It is important to note that many IBM disk storage systems certify multipathed configurations using the available open source dm multipath driver. To find out if an IBM storage platform is certified with the (http://www.ibm.com/systems/ RDAC driver, refer to System Storage Interoperation Center (SSIC) support/storage/config/ssic/displayesssearchwithoutjs.wss?start_over=yes). If your storage system multipath configuration is certified using the dm multipath driver, refer to Installing Linux Distributions (http://publib.boulder.ibm.com/infocenter/systems/topic/liaai/multipath/ on Multipathed Devices liaaiiscsimultipath.htm) for installation instructions.

IBM Services Linux offers flexibility, options, and competitive total cost of ownership with a world class enterprise operating system. Community innovation integrates leading edge technologies and best practices into Linux. IBM is a leader in the Linux community with over 600 developers in the IBM Linux Technology Center working on over 100 open source projects in the community. IBM supports Linux on all IBM servers, storage, and middleware, offering the broadest flexibility to match your business needs. For more information about IBM and Linux, visit us at ibm.com/linux linux)

(https://www.ibm.com/

IBM Support Questions and comments regarding this documentation may be posted on the DeveloperWorks Multipath Blueprint Community Forum: http://www-128.ibm.com/developerworks/forums/forum.jspa?forumID=1334 The IBM developerWorks discussion forums let you ask questions, share knowledge, ideas, and opinions about technologies and programming techniques with other developerWorks users. Use the forum content at your own risk. While IBM will attempt to provide a timely response to all postings, the use of this DeveloperWorks forum does not guarantee a response to every question that is posted, nor do we validate the answers or the code that are offered.

Typographic conventions The following typographic conventions are used in this blueprint: Bold

Identifies commands, subroutines, keywords, files, structures, directories, and other items whose names are predefined by the system. Also identifies graphical objects such as buttons, labels, and icons that the user selects.

Italics

Identifies parameters whose actual names or values are to be supplied by the user.

Monospace

22

Identifies examples of specific data values, examples of text similar to what you might see displayed, examples of portions of program code similar to what you might write as a programmer, messages from the system, or information you should actually type.


What is Multipath Connectivity The connection from the server through the Host Bus Adapter (HBA) to the storage controller is referred as a path. Within the context of this blueprint, multipath connectivity refers to a system configuration where multiple connection paths exist between a server and a storage unit (Logical Unit (LUN)) within a storage subsystem. This configuration can be used to provide redundancy or increased bandwidth. Multipath connectivity provides redundant access to the storage devices, for example, to have access to the storage device when one or more of the components in a path fail. Another advantage of using multipath connectivity is the increased throughput by way of load balancing. Note that multipathing protects against the failure of path(s) and not the failure of a specific storage device. A common example of multipath connectivity is a SAN connected storage device. Usually one or more Fibre Channel HBAs from the host will be connected to the fabric switch and the storage controllers will be connected to the same switch. A simple example of multipath connectivity could be: two HBAs connected to a switch to which the storage controllers are connected. In this case the storage controller can be accessed from either of the HBAs and hence we have multipath connectivity. In the following diagram each host has two HBAs and each storage device has two controllers. With the given configuration setup each host will have four paths to each of the LUNs in each of the storage devices.

Chapter 6. Installing Linux distributions on a Multipathed Device using the RDAC Driver

23


Hardware setup for test environment For the examples included in this blueprint, host system is a System x x3550 server with two Fibre Channel Controllers (HBAs) connected to a Fibre Channel Fabric. This Fibre Channel Fabric is also connected to a DS4300 storage server with three storage devices (LUNs). As you can see in the figure below, each of the three LUNs has two fabric connections, resulting in four paths to each LUN.

During installation it is highly recommended that there is only one path (the preferred path) attached to the LUNs. Having non-preferred paths attached during installation causes SCSI errors that the installer does not always handle gracefully. For that reason, installing in a multipath configuration can have varied outcomes and level of successes. Once the full installation is complete (including the RDAC driver) then additional connections can be added. This blueprint demonstrates installing in single path mode, for example, only one connection on each of the three LUNs. The second non-preferred path will be added after the install.

24



Installing Red Hat 5.2 on a Multipathed Device using the RDAC Driver Follow these steps for installing Red Hat 5.2 using any installation method, such as installing from a CD or Network File System.



2. Start a normal Red Hat 5.2 install, until you reach the Hard drive partitioning screen. 3. On the Hard drive partitioning screen, select Create custom layout from the drop down menu and click Next.


25



5. Press Clt-Alt-F6 to return to the graphical menu and set up the partition configuration on the appropriate sd device (sda in this example). At minimum, you need to create a root partition on the sd device as well as a swap partition. If you create a separate boot partition, make sure that it exists on the appropriate sd device (sda in this example). 6. Continue the install as normal, until you reach the Software Select screen. 7. On the software selection screen, select the Software Development packages and click Next.

26


8. Finish the installation of Red Hat. 9. When the install is complete and the machine is back up, download the latest RDAC source from Engenio Linux RDAC Driver Packages site at http://www.lsi.com/rdac/. For this example, version 09.02.C5.16 of the RDAC driver was used. wget http://www.lsi.com/rdac/rdac-LINUX-09.02.C5.16-source.tar.gz

10. Extract the source by running the following command: tar -zxf rdac-LINUX-09.02.C5.16-source.tar.gz


12. To install the RDAC source, run the following commands. See the Readme.txt file for details: make clean make make install

13. After running make install, you will see a message similar to the following: You must now edit your boot loader configuration file, /boot/grub/menu.lst, to add a new boot menu, which uses mpp-2.6.18-92.el5.img as the initrd image. Now Reboot the system for MPP to take effect. The new boot menu entry should look something like this (note that it may vary with different system configuration): ... title Red Hat Linux (2.6.18-92.el5) with MPP support root (hd0,5) Chapter 6. Installing Linux distributions on a Multipathed Device using the RDAC Driver

27

kernel /vmlinuz-2.6.18-92.el5 ro root=LABEL=RH9 initrd /mpp-2.6.18-92.el5.img ... MPP driver package has been successfully installed on your system.

Note the name of the new mpp initrd image that is created in your /boot directory. In this example, the name is mpp-2.6.18-92.el5.img. 14. Open the /boot/grub/menu.lst file with your preferred text editor. 15. Create a new boot entry by copying the first boot entry. Then edit the initrd line to point to the new initrd file so that the mpp initrd is booted. The new initrd line should look like this: Make sure that the default is set to boot to the entry you just created (default=0 means the system is default to boot to the first boot entry). default=0 timeout=5 splashimge=(hd0,0)/boot/grub/splash.xpm.gz hiddenmenu title Red Hat Enterprise Linux Server (2.6.18-92.el5) root (hd0,0) kernel /boot/vmlinuz-2.6.18-92.el5 ro root=LABEL=/1 rhgb quiet initrd/boot/mpp-2.6.18-92.el5.img title Red Hat Enterprise Linux Server (2.6.18-92.el5) root (hd0,0) kernel /boot/vmlinuz-2.6.18-92.el5 ro root=LABEL=/1 rhgb quiet initrd/boot/initrd-2.6.18-92.el5.img

16. Reboot the system. 17. When the system is back up, verify that the mpp driver is installed correctly by running the following command from the command line: ls -lR /proc/mpp/


18. When you have confirmed that the RDAC driver has been installed correctly, you can set up the multipath configuration. In the test example environment, a non-preferred path was connected from a second HBA to the fiber channel fabric and then from the fiber channel fabric to the storage server (see the figure in Chapter 3, “Hardware setup for test environment,” on page 5). This connection is represented by the RDAC driver with an addition of controllerB (representing the second connection into the DS4300) and the addition of qla2xxx_h1 (representing the connection from the second HBA) in the ls -1R /proc/mpp output. You can see four paths to each LUN.

28


/proc/mpp/: elm3b27_elm2b247 /proc/mpp/elm3b27_elm3b247: controllerA controllerB virtualLun0 virtualLun1 virtualLun2 /proc/mpp/elm3b27_elm3b247/controllerA: qla2xxx_h0c0t0 qla2xxx_h1c0t0 /proc/mpp/elm3b27_elm3b247/qla2xxx_h0c0t0 LUN0 LUN1 LUN2 /proc/mpp/elm3b27_elm3b247/qla2xxx_h1c0t0 LUN0 LUN1 LUN2 /proc/mpp/elm3b27_elm3b247/controllerB: qla2xxx_h0c0t1 qla2xxx_h1c0t1 /proc/mpp/elm3b27_elm3b247/qla2xxx_h0c0t1 LUN0 LUN1 LUN2 /proc/mpp/elm3b27_elm3b247/qla2xxx_h1c0t1 LUN0 LUN1 LUN2


Installing Suse Linux Enterprise Server 10 SP2 on a Multipathed Device using the RDAC Driver Follow these steps for installing Suse Linux Enterprise Server 10 SP2 using any installation method, such as installing from a CD or Network File System.


Procedure 1. Determine the device that your machine boots from by finding the LUN (Logical Unit) number of the bootable device that is made available by your Fibre Channel Adapter card during firmware boot. Chapter 6. Installing Linux distributions on a Multipathed Device using the RDAC Driver

29

In this example, the firmware displays the boot disk, showing LUN Number 0.

2. Start a normal Suse 10 SP2 install, until you reach the Partitioning screen.

30


3. Select Partitioning. 4. On the Partitioning screen, select Create Custom Partition Setup and click Next.

5. Select Custom Partitioning (for experts) and click Next.


31

The Expert Partitioners screen opens.

6. Determine the sd device that corresponds to the bootable LUN switch to an alternate console by pressing Clt-Alt-F2 from the original install console.

32


Run the following command: ls -1d /sys/block/sd*/device/scsi_device*


7. Press Clt-Alt-F7 to return to the graphical menu and set up the partition on the appropriate sd device (sda in this example). At minimum, you need to create a root partition on the sd device as well as a swap partition. If you create a separate boot partition, make sure that it exists on the appropriate sd device (sda in this example). 8. Return to the Installation Summary screen and select Software.

9. Select C/C++ Compiler and Tools under Development, then click Accept.


33

10. If you receive any notification of dependencies, Accept them. For example, see the follow figure:

11. Finish the installation of Suse.

34


12. When the installation is complete and the machine is back up, download the latest RDAC source from Engenio Linux RDAC Driver Packages site at http://www.lsi.com/rdac/. wget http://www.lsi.com/rdac/rdac-LINUX-09.02.C5.16-source.tar.gz

13. Extract the source using the following command: tar -zxf rdac-LINUX-09.02.C5.16-source.tar.gz


15. To install the RDAC source, run the following commands. make clean make make install

16. After running make install, you will see a message similar to the following: You must now edit your boot loader configuration file, /boot/grub/menu.lst, to add a new boot menu, which uses mpp-2.6.16.60-0.21-smp.img as the initrd image. Now Reboot the system for MPP to take effect. The new boot menu entry should look something like this (note that it may vary with different system configuration): ... title SUSE Linux (2.6.16.60-0.21-smp) with MPP support kernel (hd1,3)/boot/vmlinuz root=/dev/hdb4 vga=0x31a selinux=0 splash=silent console=tty0 resume=/dev/hda2 elevator=cfq showopts initrd (hd0,8)/boot/mpp-2.6.16.60-0.21-smp.img ... MPP driver package has been successfully installed on your system.

Note the name of the new mpp initrd image that is created in your /boot directory. In this example, the name is mpp-2.6.16.60-0.21-smp.img. 17. Open the /boot/grub/menu.lst file with your preferred text editor. 18. Create a new boot entry by copying the first boot entry. Then edit the initrd line to point to the new initrd file so that the mpp initrd is booted. The new initrd line should look like this: initrd

/boot/mpp-2.6.16.60-0.21-smp.img

Make sure that the default is set to boot to the entry you just created (default=0 means the system is default to boot to the first boot entry). Displayed below is an example /boot/grub/menu.lst file after a new entry is added. The first entry is the new added entry. default 0 timeout 8 gfxmenu (hd0,1)/boot/message ##YaST - activate ###Don’t change this comment - YaST2 identifier: Original name: linux### title SUSE Linux Enterprise Server 10 SP2 root (hd0,1) kernel /boot/vmlinuz-2.6.16.60-0.21-smp root=/dev/disk/byid/scsi-3600a0b800011a1ee000023f54843fafb-part2 ip=9.47.69.55:9.47.67.50:9.47.69.1:255.255.255.0 resume=/dev/sda1 splash=silent showopts initrd /boot/mpp-2.6.16.60-0.21-smp.img title SUSE Linux Enterprise Server 10 SP2 root (hd0,1) kernel /boot/vmlinuz-2.6.16.60-0.21-smp root=/dev/disk/byid/scsi-3600a0b800011a1ee000023f54843fafb-part2 ip=9.47.69.55:9.47.67.50:9.47.69.1:255.255.255.0 resume=/dev/sda1 splash=silent showopts initrd /boot/initrd-2.6.16.60-0.21-smp

19. Reboot the system. When the system is back up, verify that the mpp driver is installed correctly by running the following command: ls -lR /proc/mpp/ Chapter 6. Installing Linux distributions on a Multipathed Device using the RDAC Driver

35


20. When you have confirmed that the RDAC driver has been installed correctly, you can set up the multipath configuration. In the test example environment, a non-preferred path was connected from a second HBA to the fiber channel fabric and then from the fiber channel fabric to the storage server (see the figure in Chapter 3, “Hardware setup for test environment,” on page 5). This connection is represented by the RDAC driver with an addition of controllerB (representing the second connection into the DS4300) and the addition of qla2xxx_h1 (representing the connection from the second HBA) in the ls -1R /proc/mpp output. You can see four paths to each LUN. /proc/mpp/: elm3b27_elm2b247 /proc/mpp/elm3b27_elm3b247: controllerA controllerB virtualLun0 virtualLun1 virtualLun2 /proc/mpp/elm3b27_elm3b247/controllerA: qla2xxx_h0c0t0 qla2xxx_h1c0t0 /proc/mpp/elm3b27_elm3b247/qla2xxx_h0c0t0 LUN0 LUN1 LUN2 /proc/mpp/elm3b27_elm3b247/qla2xxx_h1c0t0 LUN0 LUN1 LUN2 /proc/mpp/elm3b27_elm3b247/controllerB: qla2xxx_h0c0t1 qla2xxx_h1c0t1 /proc/mpp/elm3b27_elm3b247/qla2xxx_h0c0t1 LUN0 LUN1 LUN2

36


/proc/mpp/elm3b27_elm3b247/qla2xxx_h1c0t1 LUN0 LUN1 LUN2



37

38


Appendix A. Troubleshooting tips This topic discusses troubleshooting tips and caveats. After the installation is complete, if the system does not boot up, verify whether the server supports booting from Fibre Channel connected storage by looking into the firmware boot options. Also check if you are using the appropriate LUN shown as a boot device by your HBA. After installing the RDAC driver, if the system does not boot up, verify that the entry you added to /boot/grub/menu.lst points to a mpp initrd that is in the /boot directory. If you do not have a /proc/mpp directory after performing a successful boot with the RDAC driver installed, verify that you changed the default entry in /boot/grub/menu.lst to point to the boot entry with the mpp initrd. If you are not installing on a true single path environment then you potentially see SCSI errors. These errors are displayed because any I/O request that is sent on a non-preferred path will fail. You may be able to install your system, however, the errors slow down the installer and, in many cases, cause the install to fail. You should perform the install on a single path configuration. Related reference: Chapter 1, “Scope, requirements, and support,” on page 1 This blueprint applies to System x running Linux and PowerLinux. You can learn more about this blueprint, including the intended audience, the scope and purpose, the hardware and software requirements for the tasks detailed in this blueprint, and the types of support available to you.


39

40


Appendix B. Related information and downloads You can find additional information about the processes and tools described in these procedures.

Related information v System Storage Interoperation Center (SSIC) http://www.ibm.com/systems/support/storage/config/ssic/ displayesssearchwithoutjs.wss?start_over=yes v Installing Linux Distributions on Multipathed Devices http://publib.boulder.ibm.com/infocenter/systems/topic/liaai/multipath/liaaiiscsimultipath.htm v Installation and Support Guide for Windows 2000, Windows Server 2003, NetWare, ESX Server, and Linux - IBM TotalStorage DS4000 Storage Manager V10.10 https://www-304.ibm.com/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-57808 &brandind=5000028 v Installation and Support Guide for Microsoft Windows Server 2003, Linux, Novell NetWare, and VMware ESX - IBM System Storage DS3200, DS3300, DS3400, BladeCenter Boot Disk (Type 1726) http://www-304.ibm.com/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-5069917 &brandind=5000028 v IBM Linux Storage Connectivity Forum: iSCSI and Multipath http://www-128.ibm.com/developerworks/forums/forum.jspa?forumID=1334 Related reference: Chapter 1, “Scope, requirements, and support,” on page 1 This blueprint applies to System x running Linux and PowerLinux. You can learn more about this blueprint, including the intended audience, the scope and purpose, the hardware and software requirements for the tasks detailed in this blueprint, and the types of support available to you.


41

42


Notices This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not grant you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY 10504-1785 U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Licensees of this program who wish to have information about it for the purpose of enabling: (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged, should contact: IBM Corporation Dept. LRAS/Bldg. 903 11501 Burnet Road Austin, TX 78758-3400 U.S.A. Such information may be available, subject to appropriate terms and conditions, including in some cases, payment of a fee. The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement, IBM International Program License Agreement or any equivalent agreement between us.


43

For license inquiries regarding double-byte (DBCS) information, contact the IBM Intellectual Property Department in your country or send inquiries, in writing, to: IBM World Trade Asia Corporation Licensing 2-31 Roppongi 3-chome, Minato-ku Tokyo 106-0032, Japan IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.

Trademarks IBM, the IBM logo, and ibm.com® are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® and ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at Copyright and trademark information at www.ibm.com/legal/copytrade.shtml Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. Java™ and all Java-based trademarks and logos are registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Other company, product, or service names may be trademarks or service marks of others.

44


Printed in USA

Blueprints: Installing Linux distributions on a Multipathed Device ...

Blueprints: Installing Linux distributions on a Multipathed Device ...

Suggest Documents

Installing Multiple Linux Distributions on a Single ... - Whil Hentzen

Installing SquirrelMail on EnGarde Secure Linux HOWTO

Installing Ruby on Rails in Linux

Installing Oracle Enterprise Linux On VMware Workstation

Installing PHPMyAdmin on EnGarde Secure Linux HOWTO

Installing Ruby on Rails in Linux

Installing Thinkdesktop on Umbuntu Linux: - mediaserver

A Guide to Installing RTAI Linux

Boosting up Embedded Linux device: experience on Linux-based ...

Linux Device Drivers

1 Installing MySQL under Windows 2 Installing MySQL under Linux ...

Installing Debian GNU/Linux 3.0 For PowerPC

Installing Star Micronics Printers on Red Hat Linux (Ethernet and ...

P03: Installing SAS/IntrNet on Windows and Linux

Building and Installing Geant4 on Linux (Ubuntu 14.04)

Blueprints: Quick Start Guide for installing and running KVM - IBM

Linux Device Drivers, 2nd Edition

Installing Microsoft Office Applications on an Android Device

Installing a USB external hard drive backup (Linux) - Nickalls.org

Installing a 3 node Linux based cluster for scientific ...

Kali Linux CTF Blueprints - Buchanan, Cam.pdf - Google Drive

SANS DFIR Linux Distributions - SANS Forensics

Linux Distributions and Applications Analysis During ... - SYRCoSE

Introduction to Linux Device Drivers - CiteSeer