In today's data center, multiple networks, including network attached storage (NAS), management, IPC, and storage, are used to achieve the desired performance and versatility. In addition to iSCSI for storage solutions, Fibre Channel over Ethernet (FCoE) can now be used with capable Broadcom C-NICs. FCoE is a standard that allows Fibre Channel protocol to be transferred over Ethernet by preserving existing Fibre Channel infrastructures and capital investments by classifying received FCoE and FCoE Initialization Protocol (FIP) frames.
The following FCoE features are supported:
DCB supports storage, management, computing, and communications fabrics onto a single physical fabric that is simpler to deploy, upgrade, and maintain than in standard Ethernet networks. DCB technology allows the capable Broadcom C-NICs to provide lossless data delivery, lower latency, and standards-based bandwidth sharing of data center physical links. The DCB supports FCoE, iSCSI, Network-Attached Storage (NAS), Management, and IPC traffic flows. For more information on DCB, see Using Data Center Bridging (DCB).
For information on installing, upgrading, and uninstalling the drivers, see Windows Driver Software in this user guide.
For information on installing, upgrading, and uninstalling the drivers, see Linux Driver Software in this user guide.
This section describes the install and boot procedures for the Windows and Linux operating systems. The Windows procedure is a direct install to LUN. For Linux, the installation procedure is indirect because the drivers are not inbox and therefore not included in the INITRD image. The following section details the BIOS setup and configuration of the boot environment prior to the OS install.
The Broadcom initiator must be the first entry in the system boot order. The second entry must be the OS installation media. It is important that the boot order be set correctly or else the installation will not proceed correctly. Either the desired boot LUN will not be discovered or it will be discovered but marked offline.
For Cisco Server, you can configure the boot order through Cisco CIMC or system BIOS configuration. CIMC is recommended.
The Broadcom initiator should be the first entry in Cisco BIOS followed by the OS installation media.
Storage access consists of zone provisioning and storage selective LUN presentation, each of which is commonly provisioned per initiator WWPN. Two main paths are available for approaching storage access:
With pre-provisioning, note the initiator WWPN and manually modify fabric zoning and storage selective LUN presentation to allow the appropriate access for the initiator.
The initiator WWPN can be seen at the bottom of the screen in the FCoE boot target configuration window.
The initiator WWPN can also be directly inferred from the FIP MAC address associated with the interface(s) planned for boot. Two MAC addresses are printed on stickers attached to the SFP+ cage on your adapter. The FIP MAC ends in an odd digit. The WWPN is 20:00: + <FIP MAC>. For example, if the FIP MAC is 00:10:18:11:22:33, then the WWPN will be 20:00:00:10:18:11:22:33.
The CTRL+R method allows you to use the boot initiator to bring up the link and login into all available fabrics and targets. Using this method, you can ensure that the initiator is logged into the fabric/target before making provisioning changes, and as such, can provision without manually typing in WWPNs.
NOTE: This does not preclude the need to put the boot initiator into one-time disable mode as described in One-Time Disable.
Broadcom's FCoE ROM is implemented as Boot Entry Vector (BEV). In this implementation, the Option ROM only connects to the target once it has been selected by BIOS as the chosen boot device. This is different from other implementations that will connect to the boot device even if another device has been selected by the system BIOS. For OS installation over the FCoE path, it is necessary to instruct the Option ROM to bypass FCoE and skip to CD/DVD installation media. As instructed earlier, the boot order must be configured with Broadcom boot first and installation media second. Furthermore, during OS installation, it is required to bypass the FCoE boot and pass through to the installation media for boot. It is required to do this by disabling the FCoE boot ROM from booting, and not by simply allowing the FCoE ROM to attempt to boot and allowing the BIOS to fail through and boot the installation media. Finally, it is required that the FCoE ROM successfully discover and test the readiness of the desired boot LUN in order for installation to proceed successfully. Failure to allow the boot ROM to discover the LUN and do a coordinated bypass will result in a failure to properly install the O/S to the LUN. In order to affect this coordinated bypass, there are two choices:
Wait through all option ROM banners. Once FCoE Boot is invoked, it will connect to the target, and provide a 4 second window to press CTRL+D to invoke the bypass. Press CTRL+D to proceed to installation.
Since Broadcom FCoE drivers are not inbox for Windows 2008 R2 SP1 or Windows 2008 SP2, the EVBD and OFC drivers need to be loaded during installation. Go through the normal procedures for OS installation. When no disk devices are found, Windows will prompt you to load additional drivers. At this point, connect a USB flash drive containing the full contents of the provided EVBD and OFC boot driver folders.
NOTE: The boot initiator must be configured to point at the desired installation LUN, and the boot initiator must have successfully logged and determined the readiness of the LUN prior to starting installation. If these requirements are not met, the devices will still show up in the drive list above, but upon proceeding with installation Read/Write errors will occur.
The Linux Boot from SAN procedure is indirect and requires the OS to be first installed locally and then imaged to the target LUN.
Configure the adapter boot parameters and Target Information (press CTRL+S and enter the CCM utility, as detailed in Preparing System BIOS for FCoE Build and Boot.
# ./fcoe_setup.sh
# cp fcoe-initrd.img /boot
# dd if=<SOURCE> of=<DEST> bs=1MB
where <SOURCE> is the local disk (e.g. /dev/sda)
and <DEST> is the SAN LUN (e.g. /dev/sdc)
# chkconfig --level 0123456 multipathd on
# chkconfig --level 0123456 network on
or
# chkconfig --level 0123456 NetworkManager on
# ./fcoe_setup.sh
# cp fcoe-initrd.img /boot
# dd if=<SOURCE> of=<DEST> bs=1MB
where <SOURCE> is the local disk (e.g. /dev/sda)
and <DEST> is the SAN LUN (e.g. /dev/sdc)
Now that boot configuration and OS installation are complete, you can reboot and test the installation. On this and all future reboots, no other user interactivity is required. Ignore the CTRL+D prompt and allow the system to boot through to the FCoE SAN LUN.
At this time, if additional redundant failover paths are desired, you can configure those paths through CCM, and the MBA will automatically failover to secondary paths if the first path is not available. Further, the redundant boot paths will yield redundant paths visible through host MPIO software allowing for a fault tolerant configuration.
By default, DCB is enabled on Broadcom NetXtreme II FCoE-, DCB-compatible C-NICs. To enable/disable FCoE and DCB and to optimize DCB support for the various fabrics on the network, customize the ETS, PFC, FCoE, and DCBX parameters. For Windows operating systems, use either Broadcom Advanced Control Suite (BACS) or Broadcom's Comprehensive Configuration Management (CCM) utility to configure the DCB parameters. For Linux operating systems, use Broadcom's Comprehensive Configuration Management (CCM) utility. See Using Broadcom Advanced Control Suite for more information on BACS.