Back to Contents Page

Solaris Driver Software: Broadcom NetXtreme II™ Network Adapter User Guide

Overview

Installing the Driver

Upgrading the Driver

Uninstalling Driver

Configuring the Driver

Memory Usage

Interrupt Management

FCoE Support

Overview

This file describes how to install the Solaris driver for Broadcom's NetXtreme II 10 Gigabit Ethernet network adapters. Refer to the 'bnxe' manual page for details on how to configure the driver.

The Solaris driver is released in two formats:

This driver only works with the GLDv3 Streams interface as it appears in Solaris 10 (Update 4) and later.

Installing the Driver

  1. Change directory to where BRCMbnxe-version.pkg resides.
  2. pkgadd -d BRCMbnxe-version.pkg

or

  1. Copy BRCMbnxe-X.Y.Z.tar.Z to /tmp.
  2. cd /tmp
  3.    uncompress BRCMbnxe-version.tar.Z
    
       tar -xvf BRCMbnxe-version.tar
    
       pkgadd -d /tmp
    
  4. Execute prtconf to determine instance numbers of the NIC.
  5. ifconfig bnxe[instance_number] plumb
  6. ifconfig bnxe[instance_number] ip_address netmask .... up

To make these changes permanent:

  1. Use your favorite text editor and create a file named hostname.bnxe[instance_number] in the /etc directory. Add the IP address of the interface to this file, save, and exit.
  2. Add a proper subnet mask to the file /etc/netmasks.

Upgrading the Driver

To upgrade the Broadcom driver package to the current version, you must first uninstall the previous driver version from the system. See Uninstalling Driver. Once the previous driver has been removed, you can follow any of the installation methods in this document to install the new driver version.

NOTE: Do not install multiple instances of the driver on a single system.

Uninstalling Driver

  1. ifconfig bnxe[instance_number] down
  2. ifconfig bnxe[instance_number] unplumb
  3. pkgrm BRCMbnxe

Configuring the Driver

The bnxe driver can be configured via the bnxe.conf file installed under /kernel/drv. When this config file is modified, the system must be either rebooted or the driver unloaded and reconfigured using the update_drv admin command.

All configuration can be specified per-instance. The format used is as follows and each line must end with a semicolon:

bnxe<#>_<config_item>=X;

So for adv_autoneg_cap, you would use the following:

bnxe0_adv_autoneg_cap=1;
bnxe1_adv_autoneg_cap=0;
bnxe2_adv_autoneg_cap=1;
bnxe3_adv_autoneg_cap=1;

If a configuration item is not specified for a specific instance, then the default value will be used. The default value used by all instances can be overridden using:

default_<config_item>=X;

For boolean values, 1 = TRUE and 0 = FALSE.

Memory Usage

The number of RX/TX buffer descriptors specified in the configuration file can have a detrimental affect on memory usage. If the counts are too high, DMA allocations can fail, thereby affecting other drivers loaded on the system. If DMA allocations fail during system initialization and/or boot, then there is a chance the system will not boot. This behavior is an implementation constraint of the Solaris OS. Additionally, it has been seen that the amount of DMA allocation space available on a system running in 32-bit mode is less than when running as 64-bit.

For a single RX descriptor, the following is allocated:

For a single TX descriptor, the following is allocated:

The default number of RX/TX buffer descriptors is 2048 for each. When using a Broadcom BCM57711 network adapter in multifunction mode, the number of configured descriptors is divided by four, ending up at 512. This is to keep the number of DMA allocations at a minimum. After installation, it is suggested these descriptor counts be increased until stability is guaranteed and the desired performance is reached.

For example, using the default setting of 2048 for the number of both RX and TX descriptors, the approximate amount of memory a single interface would consume is:

Single Function Mode

Multifunction Mode (#descs / 4)

Interrupt Management

If you have a system with many interfaces, it is possible to reach the allocation limit of MSIX interrupts. By default, Solaris limits each driver to 2 MSIX allocations, and there is an issue with the pcplusmp module where only a maximum of 31 MSIX interrupts are available per interrupt priority level.

If your system has four Broadcom BCM57711 network adapter ports, each running in multifunction mode, Solaris will enumerate 16 bnxe interfaces. The last interface attached will fail to allocate its second MSIX interrupt and revert to Fixed. This in turn can eventually expose an issue in the system regarding interrupt management resulting in interrupts never being received on the interface that reverted back to Fixed.

To ensure all interfaces are able to allocate their two MSIX interrupts, the workaround is to change the priority levels of specific interfaces. Network drivers are automatically assigned an interrupt priority level of 6, so changing an interface's priority level to 5 is common.

  1. First read the driver.conf man page for a background primer.
  2. Find out the driver instance paths assigned on your system.
  3. % grep bnxe /etc/path_to_inst
    
    "/pci@0,0/pci8086,2779@1/pci14e4,1650@0" 0 "bnxe"
    
    "/pci@0,0/pci8086,2779@1/pci14e4,1650@0,1" 1 "bnxe"
    
  4. Normally, the name of the driver is the last portion of the path, but you should use the most appropriate PCI ID found in /etc/driver_aliases. Depending on how the hardware is layered, there are cases where the name identified in path_to_inst will not work. To figure out which name to use, examine the output from prtconf -v and match against the IDs specified in the driver_aliases file.
  5. % grep bnxe /etc/driver_aliases
    
    bnxe "pci14e4,164e"
    
    bnxe "pci14e4,164f"
    
    bnxe "pci14e4,1650"
    
    bnxe "pciex14e4,164e"
    
    bnxe "pciex14e4,164f"
    
    bnxe "pciex14e4,1650"
    
  6. The parent of the driver is the entire path leading up to the name.
  7. The unit-address is located after the final @ in the path.
  8. Therefore, change both of the bnxe interfaces found in path_to_inst to interrupt priority 5 and use the following config lines to bnxe.conf:
  9. name = "pciex14e4,1650" parent = "/pci@0,0/pci8086,2779@1" unit-address = "0" interrupt-priorities = 5;
    
    name = "pciex14e4,1650" parent = "/pci@0,0/pci8086,2779@1" unit-address = "0,1" interrupt-priorities = 5;
    
  10. After modifying the config, either reboot the system or unplumb all interfaces and run the update_drv command.
  11. When the system has been reconfigured and the interfaces plumbed back up, verify the new interrupt priority settings by running the following command as root:
  12. % echo "::interrupts -d" | mdb -k
    

FCoE Support

Overview

FCoE is supported on Solaris 11 with limited support on Solaris 10, Update 9. The following features are the differences in Solaris 10, Update 9 when compared to Solaris 11:

Any time "create-fcoe-port" needs to be issued, the driver must be unloaded, if it is already loaded. If not, the "create-fcoe-port" will fail indicating the driver is busy. This is true when you have two or more instances of bnxef loaded, in which case, you should first delete all FCoE ports and then unload the driver. Unloading will occur only when all the instances are deleted.

Supported FC/FCoE Devices

The bnxef Broadcom 10 Gb FCoE driver works with all the major FCoE fabric devices. It is also compatible with all known FC disk devices and tape devices working through the FCoE fabric.

Unloading FCoE Driver

Delete all FCoE ports created across the various bnxe instances.

  1. Delete all the NPIV ports created before deleting FCoE ports.
  2. brcmfcoeadm delete-fcoe-port bnxe<instance_number>
  3. modinfo | grep bnxef
  4. The first column for the above command will give the module ID for the bnxef driver.

  5. modunload -i <module id>

The procedure should unload the driver. However, if there are many instances of the FCoE ports created, all the FCoE ports must be deleted before the unload can be attempted.

Configuring the FCoE Driver

The bnxef driver can be configured via the bnxef.conf file installed under /kernel/drv. When this config file is modified, the system must be either rebooted or use the update_drv(1M) command to update the driver configuration.

The details of the configurations parameters are detailed in the bnxef(7D) man page. The default parameters should work for all conditions.


Back to Contents Page