Back to Contents Page

Fibre Channel Over Ethernet: Broadcom NetXtreme II™ Network Adapter User Guide

Overview

FCoE Boot from SAN

Configuring FCoE

Overview

In today's data center, multiple networks, including network attached storage (NAS), management, IPC, and storage, are used to achieve the desired performance and versatility. In addition to iSCSI for storage solutions, Fibre Channel over Ethernet (FCoE) can now be used with capable Broadcom C-NICs. FCoE is a standard that allows Fibre Channel protocol to be transferred over Ethernet by preserving existing Fibre Channel infrastructures and capital investments by classifying received FCoE and FCoE Initialization Protocol (FIP) frames.

The following FCoE features are supported:

DCB supports storage, management, computing, and communications fabrics onto a single physical fabric that is simpler to deploy, upgrade, and maintain than in standard Ethernet networks. DCB technology allows the capable Broadcom C-NICs to provide lossless data delivery, lower latency, and standards-based bandwidth sharing of data center physical links. The DCB supports FCoE, iSCSI, Network-Attached Storage (NAS), Management, and IPC traffic flows. For more information on DCB, see Using Data Center Bridging (DCB).

Supported Operating Systems

Windows

For information on installing, upgrading, and uninstalling the drivers, see Windows Driver Software in this user guide.

Linux

For information on installing, upgrading, and uninstalling the drivers, see Linux Driver Software in this user guide.

FCoE Boot from SAN

This section describes the install and boot procedures for the Windows and Linux operating systems. The Windows procedure is a direct install to LUN. For Linux, the installation procedure is indirect because the drivers are not inbox and therefore not included in the INITRD image. The following section details the BIOS setup and configuration of the boot environment prior to the OS install.

Preparing System BIOS for FCoE Build and Boot

Modify System Boot Order

The Broadcom initiator must be the first entry in the system boot order. The second entry must be the OS installation media. It is important that the boot order be set correctly or else the installation will not proceed correctly. Either the desired boot LUN will not be discovered or it will be discovered but marked offline.

For Cisco Server, you can configure the boot order through Cisco CIMC or system BIOS configuration. CIMC is recommended.

The Broadcom initiator should be the first entry in Cisco BIOS followed by the OS installation media.

Prepare Broadcom Multiple Boot Agent for FCoE Boot

  1. During POST, press CTRL+S at the Broadcom NetXtreme Ethernet Boot Agent banner to invoke the CCM utility.


  2. Select the device through which boot is to be configured.


  3. Ensure DCB/DCBX is enabled on the device. FCoE boot is only supported on DCBX capable configurations. As such, DCB/DCBX must be enabled, and the directly attached link peer must also be DCBX capable with parameters that allow for full DCBX synchronization.


  4. Set the Boot Protocol field to FCoE in the MBA Configuration Menu through CCM.


  5. Configure the desired boot target and LUN. From the Target Information Menu, select the first available path.


  6. Enable the Connect field. Enter the target WWPN and Boot LUN information for the target to be used for boot.




  7. Press ESC until prompted to exit and save changes.
  8. Proceed to OS installation once storage access has been provisioned in the SAN.

Provisioning Storage Access in the SAN

Storage access consists of zone provisioning and storage selective LUN presentation, each of which is commonly provisioned per initiator WWPN. Two main paths are available for approaching storage access:

Pre-Provisioning

With pre-provisioning, note the initiator WWPN and manually modify fabric zoning and storage selective LUN presentation to allow the appropriate access for the initiator.

The initiator WWPN can be seen at the bottom of the screen in the FCoE boot target configuration window.

The initiator WWPN can also be directly inferred from the FIP MAC address associated with the interface(s) planned for boot. Two MAC addresses are printed on stickers attached to the SFP+ cage on your adapter. The FIP MAC ends in an odd digit. The WWPN is 20:00: + <FIP MAC>. For example, if the FIP MAC is 00:10:18:11:22:33, then the WWPN will be 20:00:00:10:18:11:22:33.

CTRL+R Method

The CTRL+R method allows you to use the boot initiator to bring up the link and login into all available fabrics and targets. Using this method, you can ensure that the initiator is logged into the fabric/target before making provisioning changes, and as such, can provision without manually typing in WWPNs.

  1. Configure at least one boot target through CCM as described above.
  2. Allow the system to attempt to boot through the selected initiator.
  3. Once the initiator boot starts, it will commence with DCBX sync, FIP Discovery, Fabric Login, Target Login, and LUN readiness checks. As each of these phases completes, if the initiator is unable to proceed to the next phase, MBA will present the option to press CTRL+R.
  4. Once CTRL+R has been activated, the boot initiator will maintain a link in whatever phase has most recently succeeded and allow you time to make the necessary provisioning corrections to proceed to the next phase.
  5. If the initiator logs into the fabric, but is unable to log into the target, a CTRL+R will pause the boot process and allow you to configure fabric zoning.
  6. Once zoning is complete, the initiator will automatically log into all visible targets. If the initiator is unable to discover the designated LUN on the designated target as provisioned in step 1, CTRL+R will pause the boot process and allow you to configure selective LUN presentation.
  7. The boot initiator will periodically poll the LUN for readiness, and once the user has provisioned access to the LUN, the boot process will automatically proceed.
  8. NOTE: This does not preclude the need to put the boot initiator into one-time disable mode as described in One-Time Disable.

One-Time Disable

Broadcom's FCoE ROM is implemented as Boot Entry Vector (BEV). In this implementation, the Option ROM only connects to the target once it has been selected by BIOS as the chosen boot device. This is different from other implementations that will connect to the boot device even if another device has been selected by the system BIOS. For OS installation over the FCoE path, it is necessary to instruct the Option ROM to bypass FCoE and skip to CD/DVD installation media. As instructed earlier, the boot order must be configured with Broadcom boot first and installation media second. Furthermore, during OS installation, it is required to bypass the FCoE boot and pass through to the installation media for boot. It is required to do this by disabling the FCoE boot ROM from booting, and not by simply allowing the FCoE ROM to attempt to boot and allowing the BIOS to fail through and boot the installation media. Finally, it is required that the FCoE ROM successfully discover and test the readiness of the desired boot LUN in order for installation to proceed successfully. Failure to allow the boot ROM to discover the LUN and do a coordinated bypass will result in a failure to properly install the O/S to the LUN. In order to affect this coordinated bypass, there are two choices:

Wait through all option ROM banners. Once FCoE Boot is invoked, it will connect to the target, and provide a 4 second window to press CTRL+D to invoke the bypass. Press CTRL+D to proceed to installation.

Windows 2008 R2 and Windows 2008 SP2 Installation

Since Broadcom FCoE drivers are not inbox for Windows 2008 R2 SP1 or Windows 2008 SP2, the EVBD and OFC drivers need to be loaded during installation. Go through the normal procedures for OS installation. When no disk devices are found, Windows will prompt you to load additional drivers. At this point, connect a USB flash drive containing the full contents of the provided EVBD and OFC boot driver folders.

  1. Load the EVBD driver first.


  2. Then load the bxfcoe (OFC) driver.


  3. Select the boot LUN to be installed:


  4. Continue with the rest of the installation. After installation is complete and booted to SAN, execute the provided Windows driver installer and reboot. Installation is now complete.
  5. NOTE: The boot initiator must be configured to point at the desired installation LUN, and the boot initiator must have successfully logged and determined the readiness of the LUN prior to starting installation. If these requirements are not met, the devices will still show up in the drive list above, but upon proceeding with installation Read/Write errors will occur.

Linux Installation

The Linux Boot from SAN procedure is indirect and requires the OS to be first installed locally and then imaged to the target LUN.

Configure the adapter boot parameters and Target Information (press CTRL+S and enter the CCM utility, as detailed in Preparing System BIOS for FCoE Build and Boot.

SLES11 SP1 Installation
  1. Install SLES11 SP1 to a locally attached hard drive.
  2. Use by-id for fstab options during installation.
  3. Once installation is complete, install the Broadcom linux-nx2 driver package and ensure a connection to the desired SAN boot LUN.
  4. For multipathd/device-mapper configurations, enable multipath services by using the following commands:
    1. Set boot.multipath to run level B:
    2. # chkconfig boot.multipath on
      
    3. Set boot.device-mapper to run level B
    4. # chkconfig boot.device-mapper on
      
    5. Set multipathd to run levels 0-6
    6. # chkconfig --level 0123456 multipathd on
      
  5. For PowerPath configurations, no chkconfig changes are required.
  6. Set network on:
    1. Set network to run levels 0-6:
    2. # chkconfg --level 0123456 network on
      
  7. Run fcoe_setup.sh to create FCoE INITRD and copy the created INITRD file to /boot:
  8. #  ./fcoe_setup.sh
    
    #  cp fcoe-initrd.img  /boot
    
  9. Create backups of '/boot/grub/menu.lst' and '/etc/fstab'.
  10. Add a new boot entry to /boot/grub/menu.lst by pointing INITRD to the new INITRD that was created in step 7, /boot/fcoe-initrd.img and modifying the root/resume parameters to point at the correct device names. For multipathd/device-mapper configurations, use the /dev/disk/by-id/dm-uuid-part... device names. For PowerPath configurations, use the /dev/emcpower*:


  11. Modify /etc/fstab to point to the same device paths for the root (/) and swap mount points.


  12. Now that the local drive is reconfigured to support FCoE boot. Copy the entire local hard drive contents to the SAN LUN via dd:
  13. # dd if=<SOURCE> of=<DEST> bs=1MB
    
    where <SOURCE> is the local disk (e.g. /dev/sda) 
    
    and <DEST> is the SAN LUN (e.g. /dev/sdc)
    
  14. Restore '/boot/grub/menu.lst' and '/etc/fstab' files to their original contents by copying the backup files created above to their original filenames.
RHEL6.0 Installation
  1. Install RHEL 6.0 to a locally attached hard drive.
  2. Use Vol-Group option during installation (default).
  3. Once installation is complete, install the Broadcom linux-nx2 driver package and ensure a connection to the desired SAN boot LUN.
  4. For multipathd/device-mapper configurations, enable multipath services by using the following commands:
  5. # chkconfig --level 0123456 multipathd on
    
  6. Set network services to start at all run levels:
  7. # chkconfig --level 0123456 network on
    

    or

    # chkconfig --level 0123456 NetworkManager on
    
  8. Run fcoe_setup.sh to create FCoE INITRD and copy the created INITRD file to /boot:
  9. #  ./fcoe_setup.sh
    
    #  cp fcoe-initrd.img  /boot
    
  10. Create backups of '/boot/grub/menu.lst'
  11. Add a new boot entry to /boot/grub/menu.lst by pointing INITRD to the new INITRD that was created in step 7, /boot/fcoe-initrd.img.
  12. Now that the local drive is reconfigured to support FCoE boot. Copy the entire local hard drive contents to the SAN LUN via dd:
  13. # dd if=<SOURCE> of=<DEST> bs=1MB
    
    where <SOURCE> is the local disk (e.g. /dev/sda) 
    
    and <DEST> is the SAN LUN (e.g. /dev/sdc)
    
  14. Because the LVM drive drive/group/volume signatures are now identical between the local HDD and the SAN LUN, the local HDD must be removed prior to reboot and booting from SAN in order to prevent LVM discovery conflicts.
  15. Restore '/boot/grub/menu.lst' its original contents by copying the backup files created above to their original filenames.

Booting from SAN After Installation

Now that boot configuration and OS installation are complete, you can reboot and test the installation. On this and all future reboots, no other user interactivity is required. Ignore the CTRL+D prompt and allow the system to boot through to the FCoE SAN LUN.

At this time, if additional redundant failover paths are desired, you can configure those paths through CCM, and the MBA will automatically failover to secondary paths if the first path is not available. Further, the redundant boot paths will yield redundant paths visible through host MPIO software allowing for a fault tolerant configuration.

Configuring FCoE

By default, DCB is enabled on Broadcom NetXtreme II FCoE-, DCB-compatible C-NICs. To enable/disable FCoE and DCB and to optimize DCB support for the various fabrics on the network, customize the ETS, PFC, FCoE, and DCBX parameters. For Windows operating systems, use either Broadcom Advanced Control Suite (BACS) or Broadcom's Comprehensive Configuration Management (CCM) utility to configure the DCB parameters. For Linux operating systems, use Broadcom's Comprehensive Configuration Management (CCM) utility. See Using Broadcom Advanced Control Suite for more information on BACS.


Back to Contents Page