Back to Contents Page

VMware Driver Software: Broadcom NetXtreme II™ Network Adapter User Guide

Packaging

Networking Support

Drivers

FCoE Support

Packaging

The VMware driver is released in the packaging formats shown in Table 1.

Table 1: VMware Driver Packaging
Format
Drivers
Compressed tar
bnx2x-version.tar.gz
VMware VIB
vmware-esx-drivers-net-bnx2x-version.x86_64.vib

Networking Support

This section describes the bnx2x VMware ESX driver for the Broadcom NetXtreme II PCIE 10 GbE network adapters.

Drivers

Download, Install, and Update Drivers

To download, install, or update the VMware ESX/ESXi driver for NetXtreme II 10 GbE network adapters, see http://www.vmware.com/support.

Driver Parameters

Several optional parameters can be supplied as a command line argument to the vmkload_mod command. These parameters can also be set via the esxcfg-module command. See the man page for more information.

int_mode

The optional parameter int_mode is used to force using an interrupt mode other than MSI-X. By default, the driver will try to enable MSI-X if it is supported by the kernel. If MSI-X is not attainable, then the driver will try to enable MSI if it is supported by the kernel. If MSI is not attainable, then the driver will use the legacy INTx mode.

Set the int_mode parameter to 1 as shown below to force using the legacy INTx mode on all NetXtreme II network adapters in the system.

vmkload_mod bnx2x int_mode=1

Set the int_mode parameter to 2 as shown below to force using MSI mode on all NetXtreme II network adapters in the system.

vmkload_mod bnx2x int_mode=2

disable_tpa

The optional parameter disable_tpa can be used to disable the Transparent Packet Aggregation (TPA) feature. By default, the driver will aggregate TCP packets, but if you would like to disable this advanced feature, it can be done.

Set the disable_tpa parameter to 1 as shown below to disable the TPA feature on all NetXtreme II network adapters in the system.

vmkload_mod bnx2x.ko disable_tpa=1

Use ethtool to disable TPA (LRO) for a specific network adapter.

num_rx_queues

The optional parameter num_rx_queues may be used to set the number of Rx queues on kernels starting from 2.6.24 when multi_mode is set to 1 and interrupt mode is MSI-X. Number of Rx queues must be equal to or greater than the number of Tx queues (see num_tx_queues parameter). If the interrupt mode is different than MSI-X (see int_mode parameter), then then the number of Rx queues will be set to 1, discarding the value of this parameter.

num_tx_queues

The optional parameter num_tx_queues may be used to set the number of Tx queues on kernels starting from 2.6.27 when multi_mode is set to 1 and interrupt mode is MSI-X. The number of Rx queues must be equal to or greater than the number of Tx queues (see num_rx_queues parameter). If the interrupt mode is different than MSI-X (see int_mode parameter), then the number of Tx queues will be set to 1, discarding the value of this parameter.

pri_map

The optional parameter pri_map is used to map the VLAN PRI value or the IP DSCP value to a different or the same CoS in the hardware. This 32-bit parameter is evaluated by the driver as 8 values of 4 bits each. Each nibble sets the desired hardware queue number for that priority.

For example, set the pri_map parameter to 0x22221100 to map priority 0 and 1 to CoS 0, map priority 2 and 3 to CoS 1, and map priority 4 to 7 to CoS 2. In another example, set the pri_map parameter to 0x11110000 to map priority 0 to 3 to CoS 0, and map priority 4 to 7 to CoS 1.

qs_per_cos

The optional parameter qs_per_cos is used to specify the number of queues that will share the same CoS. This parameter is evaluated by the driver up to 3 values of 8 bits each. Each byte sets the desired number of queues for that CoS. The total number of queues is limited by the hardware limit.

For example, set the qs_per_cos parameter to 0x10101 to create a total of three queues, one per CoS. In another example, set the qs_per_cos parameter to 0x404 to create a total of 8 queues, divided into only 2 CoS, 4 queues in each CoS.

cos_min_rate

The optional parameter cos_min_rate is used to determine the weight of each CoS for Round-robin scheduling in transmission. This parameter is evaluated by the driver up to 3 values of 8 bits each. Each byte sets the desired weight for that CoS. The weight ranges from 0 to 100.

For example, set the cos_min_rate parameter to 0x101 for fair transmission rate between two CoS. In another example, set the cos_min_rate parameter to 0x30201 to give the higher CoS the higher rate of transmission. To avoid using the fairness algorithm, omit setting the optional parameter cos_min_rate or set it to 0.

dropless_fc

The optional parameter dropless_fc can be used to enable a complementary flow control mechanism on Broadcom network adapters. The default flow control mechanism is to send pause frames when the on-chip buffer (BRB) is reaching a certain level of occupancy. This is a performance targeted flow control mechanism. On Broadcom network adapters, you can enable another flow control mechanism to send pause frames if one of the host buffers (when in RSS mode) is exhausted. This is a "zero packet drop" targeted flow control mechanism.

Set the dropless_fc parameter to 1 as shown below to enable the dropless flow control mechanism feature on all Broadcom network adapters in the system.

vmkload_mod bnx2x dropless_fc=1

Driver Defaults

Speed: Autonegotiation with all speeds advertised

Flow Control: Autonegotiation with rx and tx advertised

MTU: 1500 (range 46–9000)

Rx Ring Size: 4078 (range 0–4078)

Tx Ring Size: 4078 (range (MAX_SKB_FRAGS+4) - 4078). MAX_SKB_FRAGS varies on different kernels and different architectures. On a 2.6 kernel for x86, MAX_SKB_FRAGS is 18.

Coalesce RX Microseconds: 25 (range 0–3000)

Coalesce TX Microseconds: 50 (range 0–12288)

MSI-X: Enabled (if supported by 2.6 kernel)

TSO: Enabled

WoL: Disabled

Unloading and Removing Driver

To unload the driver, type the following:

vmkload_mod -u bnx2x

Driver Messages

The following are the most common sample messages that may be logged in the file /var/log/messages. Use dmesg -n <level> to control the level at which messages will appear on the console. Most systems are set to level 6 by default. To see all messages, set the level higher.

Driver Sign On

Broadcom NetXtreme II 5771x 10Gigabit Ethernet Driver 
bnx2x 0.40.15 ($DateTime: 2007/11/22 05:32:40 $)

NIC Detected

eth0: Broadcom NetXtreme II BCM57710 XGb (A1) 
PCI-E x8 2.5GHz found at mem e8800000, IRQ 16, node addr 001018360012

MSI-X Enabled Successfully

bnx2x: eth0: using MSI-X

Link Up and Speed Indication

bnx2x: eth0 NIC Link is Up, 10000 Mbps full duplex, receive & transmit flow control ON

Link Down Indication

bnx2x: eth0 NIC Link is Down

Memory Limitation

If you see messages in the log file that look like the following, then the ESX host is severely strained. To relieve this, disable NetQueue.

Dec  2 18:24:20 ESX4 vmkernel: 0:00:00:32.342 cpu2:4142)WARNING: Heap: 1435: Heap bnx2x already at its maximumSize. Cannot expand.
Dec  2 18:24:20 ESX4 vmkernel: 0:00:00:32.342 cpu2:4142)WARNING: Heap: 1645: Heap_Align(bnx2x, 4096/4096 bytes, 4096 align) failed.  caller: 0x41800187d654
Dec  2 18:24:20 ESX4 vmkernel: 0:00:00:32.342 cpu2:4142)WARNING: vmklinux26: alloc_pages: Out of memory

Disable NetQueue by manually loading the bnx2x vmkernel module via the command.

vmkload_mod bnx2x multi_mode=0

or to persist the settings across reboots via the command

esxcfg-module -s multi_mode=0 bnx2x

Reboot the machine for the settings to take place.

MultiQueue/NetQueue

The optional parameter num_queues may be used to set the number of Rx and Tx queues when multi_mode is set to 1 and interrupt mode is MSI-X. If interrupt mode is different than MSI-X (see int_mode parameter), the number of Rx and Tx queues will be set to 1, discarding the value of this parameter.

If you would like the use of more then 1 queue, force the number of NetQueues to use via the following command:

esxcfg-module -s "multi_mode=1 num_queues=<num of queues>" bnx2x

Otherwise, allow the bnx2x driver to select the number of NetQueues to use via the following command:

esxcfg-module -s "multi_mode=1 num_queues=0" bnx2x

The optimal number is to have the number of NetQueues match the number of CPUs on the machine.

FCoE Support

This section describes the contents and procedures associated with installation of the VMware software package for supporting Broadcom FCoE C-NICs.

Drivers

Table 2: Broadcom NetXtreme II FCoE Drivers
Driver
Description
bnx2x
This driver manages all PCI device resources (registers, host interface queues, etc.) and also acts as the Layer 2 VMware low-level network driver for Broadcom's NetXtreme II 10G device. This driver directly controls the hardware and is responsible for sending and receiving Ethernet packets on behalf of the VMware host networking stack. The bnx2x driver also receives and processes device interrupts, both on behalf of itself (for L2 networking) and on behalf of the bnx2fc (FCoE protocol) and cnic drivers.
bnx2fc
The Broadcom VMware FCoE driver is a kernel mode driver used to provide a translation layer between the VMware SCSI stack and the Broadcom FCoE firmware/hardware. In addition, the driver interfaces with the networking layer to transmit and receive encapsulated FCoE frames on behalf of open-fcoe's libfc/libfcoe for FIP/device discovery.
bnx2i
The bnx2i driver is Broadcom VMware iSCSI HBA driver. Similar to bnx2fc, bnx2i is a kernel mode driver used to provide a translation layer between the VMware SCSI stack and the Broadcom iSCSI firmware/hardware. Bnx2i functions under the open-iscsi framework.

Supported Distributions

The FCoE/DCB feature set is supported on VMware ESXi 5.0 and above.

Enabling FCoE

To enable FCoE hardware offload on the C-NIC

  1. Determine the ports that are FCoE-capable:
  2. # esxcli fcoe nic list
    

Output example:

vmnic4
User Priority: 3
Source MAC: FF:FF:FF:FF:FF:FF
Active: false
Priority Settable: false
Source MAC Settable: false
VLAN Range Settable: false
  1. Enable the FCoE interface:
  2. # esxcli fcoe nic discover -n vmnicX
    

    Where X is the interface number gained from esxcli fcoe nic list.

  3. Verify that the interface is working:
  4. # esxcli fcoe adapter list 
    

Output example:

vmhba34
Source MAC: bc:30:5b:01:82:39
FCF MAC: 00:05:73:cf:2c:ea
VNPort MAC: 0e:fc:00:47:04:04
Physical NIC: vmnic7
User Priority: 3
VLAN id: 2008

The output of this command should show valid:

FCF MAC, VNPort MAC, Priority, and VLAN id for the Fabric that is connected to the C-NIC.

The following command can also be used to verify that the interface is working properly:

#esxcfg-scsidevs -a 

Output example:

vmhba34 bnx2fc     link-up   fcoe.1000<mac address>:2000<mac address>  () Software FCoE
vmhba35 bnx2fc     link-up   fcoe.1000<mac address>:2000<mac address>  () Software FCoE

NOTE: The label "Software FCoE" is a VMware term used to describe initiators that depend on the inbox FCoE libraries and utilities. Broadcom's FCoE solution is a fully state connection-based hardware offload solution designed to significantly reduce the CPU burden encumbered by a non-offload software initiator.

Installation Check

To verify the correct installation of the driver and to ensure that the host port is seen by the switch, follow the procedure below.

To verify the correct installation of the driver

  1. Verify the host port shows up in the switch FLOGI database using the "show flogi database" command for the case of a Cisco FCF and "fcoe -loginshow" command for the case of a Brocade FCF.
  2. If the Host WWPN does not appear in the FLOGI database, then provide driver log messages for review.

Limitations


Back to Contents Page