This file describes how to install the Solaris driver for Broadcom's NetXtreme II 10 Gigabit Ethernet network adapters. Refer to the 'bnxe' manual page for details on how to configure the driver.
The Solaris driver is released in two formats:
NOTE: A DU image does not exist at this time because of driver size limitations. Solaris DU diskettes can be used to install the driver into the system both during system installation and/or after the system has been installed and booted.
This driver only works with the GLDv3 Streams interface as it appears in Solaris 10 (Update 4) and later.
or
To make these changes permanent:
To upgrade the Broadcom driver package to the current version, you must first uninstall the previous driver version from the system. See Uninstalling Driver. Once the previous driver has been removed, you can follow any of the installation methods in this document to install the new driver version.
NOTE: Do not install multiple instances of the driver on a single system.
The bnxe driver can be configured via the bnxe.conf file installed under /kernel/drv. When this config file is modified, the system must be either rebooted or the driver unloaded and reconfigured using the update_drv admin command.
All configuration can be specified per-instance. The format used is as follows and each line must end with a semicolon:
bnxe<#>_<config_item>=X;
So for adv_autoneg_cap, you would use the following:
bnxe0_adv_autoneg_cap=1;
bnxe1_adv_autoneg_cap=0;
bnxe2_adv_autoneg_cap=1;
bnxe3_adv_autoneg_cap=1;
If a configuration item is not specified for a specific instance, then the default value will be used. The default value used by all instances can be overridden using:
default_<config_item>=X;
For boolean values, 1 = TRUE and 0 = FALSE.
The number of RX/TX buffer descriptors specified in the configuration file can have a detrimental affect on memory usage. If the counts are too high, DMA allocations can fail, thereby affecting other drivers loaded on the system. If DMA allocations fail during system initialization and/or boot, then there is a chance the system will not boot. This behavior is an implementation constraint of the Solaris OS. Additionally, it has been seen that the amount of DMA allocation space available on a system running in 32-bit mode is less than when running as 64-bit.
For a single RX descriptor, the following is allocated:
For a single TX descriptor, the following is allocated:
The default number of RX/TX buffer descriptors is 2048 for each. When using a Broadcom BCM57711 network adapter in multifunction mode, the number of configured descriptors is divided by four, ending up at 512. This is to keep the number of DMA allocations at a minimum. After installation, it is suggested these descriptor counts be increased until stability is guaranteed and the desired performance is reached.
For example, using the default setting of 2048 for the number of both RX and TX descriptors, the approximate amount of memory a single interface would consume is:
Single Function Mode
Multifunction Mode (#descs / 4)
If you have a system with many interfaces, it is possible to reach the allocation limit of MSIX interrupts. By default, Solaris limits each driver to 2 MSIX allocations, and there is an issue with the pcplusmp module where only a maximum of 31 MSIX interrupts are available per interrupt priority level.
If your system has four Broadcom BCM57711 network adapter ports, each running in multifunction mode, Solaris will enumerate 16 bnxe interfaces. The last interface attached will fail to allocate its second MSIX interrupt and revert to Fixed. This in turn can eventually expose an issue in the system regarding interrupt management resulting in interrupts never being received on the interface that reverted back to Fixed.
To ensure all interfaces are able to allocate their two MSIX interrupts, the workaround is to change the priority levels of specific interfaces. Network drivers are automatically assigned an interrupt priority level of 6, so changing an interface's priority level to 5 is common.
% grep bnxe /etc/path_to_inst
"/pci@0,0/pci8086,2779@1/pci14e4,1650@0" 0 "bnxe"
"/pci@0,0/pci8086,2779@1/pci14e4,1650@0,1" 1 "bnxe"
% grep bnxe /etc/driver_aliases
bnxe "pci14e4,164e"
bnxe "pci14e4,164f"
bnxe "pci14e4,1650"
bnxe "pciex14e4,164e"
bnxe "pciex14e4,164f"
bnxe "pciex14e4,1650"
name = "pciex14e4,1650" parent = "/pci@0,0/pci8086,2779@1" unit-address = "0" interrupt-priorities = 5;
name = "pciex14e4,1650" parent = "/pci@0,0/pci8086,2779@1" unit-address = "0,1" interrupt-priorities = 5;
% echo "::interrupts -d" | mdb -k
FCoE is supported on Solaris 11 with limited support on Solaris 10, Update 9. The following features are the differences in Solaris 10, Update 9 when compared to Solaris 11:
# modinfo | grep bnxef
249 fffffffff8d63000 486b8 54 1 bnxef (6.4.13)
Then issue the modunload command to unload the module before "create-fcoe-port" is issued to create a new FCoE port.
# modunload -i 249
Any time "create-fcoe-port" needs to be issued, the driver must be unloaded, if it is already loaded. If not, the "create-fcoe-port" will fail indicating the driver is busy. This is true when you have two or more instances of bnxef loaded, in which case, you should first delete all FCoE ports and then unload the driver. Unloading will occur only when all the instances are deleted.
The bnxef Broadcom 10 Gb FCoE driver works with all the major FCoE fabric devices. It is also compatible with all known FC disk devices and tape devices working through the FCoE fabric.
Delete all FCoE ports created across the various bnxe instances.
The procedure should unload the driver. However, if there are many instances of the FCoE ports created, all the FCoE ports must be deleted before the unload can be attempted.
The bnxef driver can be configured via the bnxef.conf file installed under /kernel/drv. When this config file is modified, the system must be either rebooted or use the update_drv(1M) command to update the driver configuration.
The details of the configurations parameters are detailed in the bnxef(7D) man page. The default parameters should work for all conditions.