The VMware driver is released in the packaging formats shown in Table 1.
Format
|
Drivers
|
---|---|
Compressed tar
|
bnx2x-version.tar.gz
|
VMware VIB
|
vmware-esx-drivers-net-bnx2x-version.x86_64.vib
|
This section describes the bnx2x VMware ESX driver for the Broadcom NetXtreme II PCIE 10 GbE network adapters.
To download, install, or update the VMware ESX/ESXi driver for NetXtreme II 10 GbE network adapters, see http://www.vmware.com/support.
Several optional parameters can be supplied as a command line argument to the vmkload_mod command. These parameters can also be set via the esxcfg-module command. See the man page for more information.
int_mode
The optional parameter int_mode is used to force using an interrupt mode other than MSI-X. By default, the driver will try to enable MSI-X if it is supported by the kernel. If MSI-X is not attainable, then the driver will try to enable MSI if it is supported by the kernel. If MSI is not attainable, then the driver will use the legacy INTx mode.
Set the int_mode parameter to 1 as shown below to force using the legacy INTx mode on all NetXtreme II network adapters in the system.
vmkload_mod bnx2x int_mode=1
Set the int_mode parameter to 2 as shown below to force using MSI mode on all NetXtreme II network adapters in the system.
vmkload_mod bnx2x int_mode=2
disable_tpa
The optional parameter disable_tpa can be used to disable the Transparent Packet Aggregation (TPA) feature. By default, the driver will aggregate TCP packets, but if you would like to disable this advanced feature, it can be done.
Set the disable_tpa parameter to 1 as shown below to disable the TPA feature on all NetXtreme II network adapters in the system.
vmkload_mod bnx2x.ko disable_tpa=1
Use ethtool to disable TPA (LRO) for a specific network adapter.
num_rx_queues
The optional parameter num_rx_queues may be used to set the number of Rx queues on kernels starting from 2.6.24 when multi_mode is set to 1 and interrupt mode is MSI-X. Number of Rx queues must be equal to or greater than the number of Tx queues (see num_tx_queues parameter). If the interrupt mode is different than MSI-X (see int_mode parameter), then then the number of Rx queues will be set to 1, discarding the value of this parameter.
num_tx_queues
The optional parameter num_tx_queues may be used to set the number of Tx queues on kernels starting from 2.6.27 when multi_mode is set to 1 and interrupt mode is MSI-X. The number of Rx queues must be equal to or greater than the number of Tx queues (see num_rx_queues parameter). If the interrupt mode is different than MSI-X (see int_mode parameter), then the number of Tx queues will be set to 1, discarding the value of this parameter.
pri_map
The optional parameter pri_map is used to map the VLAN PRI value or the IP DSCP value to a different or the same CoS in the hardware. This 32-bit parameter is evaluated by the driver as 8 values of 4 bits each. Each nibble sets the desired hardware queue number for that priority.
For example, set the pri_map parameter to 0x22221100 to map priority 0 and 1 to CoS 0, map priority 2 and 3 to CoS 1, and map priority 4 to 7 to CoS 2. In another example, set the pri_map parameter to 0x11110000 to map priority 0 to 3 to CoS 0, and map priority 4 to 7 to CoS 1.
qs_per_cos
The optional parameter qs_per_cos is used to specify the number of queues that will share the same CoS. This parameter is evaluated by the driver up to 3 values of 8 bits each. Each byte sets the desired number of queues for that CoS. The total number of queues is limited by the hardware limit.
For example, set the qs_per_cos parameter to 0x10101 to create a total of three queues, one per CoS. In another example, set the qs_per_cos parameter to 0x404 to create a total of 8 queues, divided into only 2 CoS, 4 queues in each CoS.
cos_min_rate
The optional parameter cos_min_rate is used to determine the weight of each CoS for Round-robin scheduling in transmission. This parameter is evaluated by the driver up to 3 values of 8 bits each. Each byte sets the desired weight for that CoS. The weight ranges from 0 to 100.
For example, set the cos_min_rate parameter to 0x101 for fair transmission rate between two CoS. In another example, set the cos_min_rate parameter to 0x30201 to give the higher CoS the higher rate of transmission. To avoid using the fairness algorithm, omit setting the optional parameter cos_min_rate or set it to 0.
dropless_fc
The optional parameter dropless_fc can be used to enable a complementary flow control mechanism on Broadcom network adapters. The default flow control mechanism is to send pause frames when the on-chip buffer (BRB) is reaching a certain level of occupancy. This is a performance targeted flow control mechanism. On Broadcom network adapters, you can enable another flow control mechanism to send pause frames if one of the host buffers (when in RSS mode) is exhausted. This is a "zero packet drop" targeted flow control mechanism.
Set the dropless_fc parameter to 1 as shown below to enable the dropless flow control mechanism feature on all Broadcom network adapters in the system.
vmkload_mod bnx2x dropless_fc=1
Speed: Autonegotiation with all speeds advertised
Flow Control: Autonegotiation with rx and tx advertised
MTU: 1500 (range 469000)
Rx Ring Size: 4078 (range 04078)
Tx Ring Size: 4078 (range (MAX_SKB_FRAGS+4) - 4078). MAX_SKB_FRAGS varies on different kernels and different architectures. On a 2.6 kernel for x86, MAX_SKB_FRAGS is 18.
Coalesce RX Microseconds: 25 (range 03000)
Coalesce TX Microseconds: 50 (range 012288)
MSI-X: Enabled (if supported by 2.6 kernel)
TSO: Enabled
WoL: Disabled
To unload the driver, type the following:
vmkload_mod -u bnx2x
The following are the most common sample messages that may be logged in the file /var/log/messages. Use dmesg -n <level>
to control the level at which messages will appear on the console. Most systems are set to level 6 by default. To see all messages, set the level higher.
Driver Sign On
Broadcom NetXtreme II 5771x 10Gigabit Ethernet Driver
bnx2x 0.40.15 ($DateTime: 2007/11/22 05:32:40 $)
NIC Detected
eth0: Broadcom NetXtreme II BCM57710 XGb (A1)
PCI-E x8 2.5GHz found at mem e8800000, IRQ 16, node addr 001018360012
MSI-X Enabled Successfully
bnx2x: eth0: using MSI-X
Link Up and Speed Indication
bnx2x: eth0 NIC Link is Up, 10000 Mbps full duplex, receive & transmit flow control ON
Link Down Indication
bnx2x: eth0 NIC Link is Down
Memory Limitation
If you see messages in the log file that look like the following, then the ESX host is severely strained. To relieve this, disable NetQueue.
Dec 2 18:24:20 ESX4 vmkernel: 0:00:00:32.342 cpu2:4142)WARNING: Heap: 1435: Heap bnx2x already at its maximumSize. Cannot expand.
Dec 2 18:24:20 ESX4 vmkernel: 0:00:00:32.342 cpu2:4142)WARNING: Heap: 1645: Heap_Align(bnx2x, 4096/4096 bytes, 4096 align) failed. caller: 0x41800187d654
Dec 2 18:24:20 ESX4 vmkernel: 0:00:00:32.342 cpu2:4142)WARNING: vmklinux26: alloc_pages: Out of memory
Disable NetQueue by manually loading the bnx2x vmkernel module via the command.
vmkload_mod bnx2x multi_mode=0
or to persist the settings across reboots via the command
esxcfg-module -s multi_mode=0 bnx2x
Reboot the machine for the settings to take place.
MultiQueue/NetQueue
The optional parameter num_queues may be used to set the number of Rx and Tx queues when multi_mode is set to 1 and interrupt mode is MSI-X. If interrupt mode is different than MSI-X (see int_mode parameter), the number of Rx and Tx queues will be set to 1, discarding the value of this parameter.
If you would like the use of more then 1 queue, force the number of NetQueues to use via the following command:
esxcfg-module -s "multi_mode=1 num_queues=<num of queues>" bnx2x
Otherwise, allow the bnx2x driver to select the number of NetQueues to use via the following command:
esxcfg-module -s "multi_mode=1 num_queues=0" bnx2x
The optimal number is to have the number of NetQueues match the number of CPUs on the machine.
This section describes the contents and procedures associated with installation of the VMware software package for supporting Broadcom FCoE C-NICs.
The FCoE/DCB feature set is supported on VMware ESXi 5.0 and above.
To enable FCoE hardware offload on the C-NIC
Output example:
vmnic4
User Priority: 3
Source MAC: FF:FF:FF:FF:FF:FF
Active: false
Priority Settable: false
Source MAC Settable: false
VLAN Range Settable: false
Output example:
vmhba34
Source MAC: bc:30:5b:01:82:39
FCF MAC: 00:05:73:cf:2c:ea
VNPort MAC: 0e:fc:00:47:04:04
Physical NIC: vmnic7
User Priority: 3
VLAN id: 2008
The output of this command should show valid:
FCF MAC, VNPort MAC, Priority, and VLAN id for the Fabric that is connected to the C-NIC.
The following command can also be used to verify that the interface is working properly:
#esxcfg-scsidevs -a
Output example:
vmhba34 bnx2fc link-up fcoe.1000<mac address>:2000<mac address> () Software FCoE
vmhba35 bnx2fc link-up fcoe.1000<mac address>:2000<mac address> () Software FCoE
NOTE: The label "Software FCoE" is a VMware term used to describe initiators that depend on the inbox FCoE libraries and utilities. Broadcom's FCoE solution is a fully state connection-based hardware offload solution designed to significantly reduce the CPU burden encumbered by a non-offload software initiator.
To verify the correct installation of the driver and to ensure that the host port is seen by the switch, follow the procedure below.
To verify the correct installation of the driver