Back to Contents Page

Linux Driver Software: Broadcom NetXtreme II™ Network Adapter User Guide

Introduction

Limitations

Packaging

Installing Linux Driver Software

Unloading/Removing the Linux Driver

Patching PCI Files (Optional)

Network Installations

Setting Values for Optional Properties

Driver Defaults

Driver Messages

Teaming with Channel Bonding

Statistics

Linux iSCSI Offload

Introduction

This section discusses the Linux drivers for the Broadcom NetXtreme II network adapters.

Table 1: Broadcom NetXtreme II Linux Drivers
Linux Driver
Description
bnx2
cnic
Linux drivers for the BCM5706/BCM5708/BCM5709 10/100/1000/2500 Mbps network adapters. The bnx2 driver is the networking driver and the cnic driver supports additional features required by the bnx2i iSCSI offload driver.
bnx2x
Linux driver for the BCM57710/BCM57711/BCM57712 10 Gbps network adapters
bnx2i
Linux driver to enable iSCSI offload on the BCM5706/BCM5708/BCM5709/BCM57710/BCM57711/BCM57712 10/100/1000/2500/10000 Mbps network adapters

Limitations

bnx2 Driver

The current version of the driver has been tested on 2.4.x kernels (starting from 2.4.24) and all 2.6.x kernels. The driver may not compile on kernels older than 2.4.24.

NOTE: Support for the 2.4.21 kernels is provided in Red Hat Enterprise Linux 3.

Testing is concentrated on i386 and x86_64 architectures. Only limited testing has been done on other architectures. Minor changes to some source files and Makefile may be needed on some kernels. Additionally, the Makefile will not compile the cnic driver on kernels older than 2.6.16. iSCSI offload is only supported on 2.6.16 and newer kernels.

NOTE: For Broadcom NetXtreme II BCM5708 devices with a silicon revision prior to B2, the open source bnx2 driver does not support the reporting and configuration of NetXtreme II WOL settings via ethtool. For silicon revisions of B2 or later, the bnx2 driver reports support for Magic Packet WOL via ethtool. Enabling support via ethtool is mandatory to successfully wake the system. To determine the silicon revision of your Broadcom NetXtreme II device, use the lspci command, where "10" = revision B0, "11" = revision B1, and "12" = revision B2.

bnx2x Driver

The current version of the driver has been tested on 2.6.x kernels starting from 2.6.9. The driver may not compile on kernels older than 2.6.9. Testing is concentrated on i386 and x86_64 architectures. Only limited testing has been done on some other architectures. Minor changes to some source files and Makefile may be needed on some kernels.

bnx2i Driver

The current version of the driver has been tested on 2.6.x kernels, starting from 2.6.18 kernel. The driver may not compile on older kernels. Testing is concentrated on i386 and x86_64 architectures, Red Hat EL5, and SUSE 11 SP1 distributions.

Packaging

The Linux driver is released in the packaging formats shown in Table 2. The netxtreme2 package contains the bnx2 (1 Gb network adapter) driver, the bnx2x (10 Gb network adapter) driver, and the bnx2i (iSCSI) driver for source RPM and compressed tar.

Table 2: Linux Driver Packaging
Format
Drivers
Source RPM
netxtreme2-version.src.rpm
Compressed tar
netxtreme2-version.tar.gz
Supplemental tar
netxtreme2_sup-version.tar.gz

Identical source files to build the driver are included in both RPM and TAR source packages. The supplemental tar file contains additional utilities such as patches and driver diskette images for network installation.

Updated open-iSCSI components are released in source RPM format.

Installing Linux Driver Software

Installing the Source RPM Package

Building the Driver from the Source TAR File

NOTE: If a bnx2/bnx2x/bnx2i driver is loaded and the Linux kernel is updated, the driver module must be recompiled if the driver module was installed using the source RPM or the TAR package.

Installing the Source RPM Package

NOTE: The examples in this procedure refer to the bnx2 driver, but also apply to the bnx2x and bnx2i drivers.

  1. Install the source RPM package:
  2. rpm -ivh netxtreme2-version.src.rpm
    
  3. Change the directory to the RPM path and build the binary driver for your kernel (the RPM path is different for different Linux distributions):
  4. cd /usr/src/redhat,OpenLinux,turbo,packages,rpm ...
    
    rpm -bb SPECS/bnx2.spec
    

    or

    rpmbuild -bb SPECS/netxtreme2.spec (for RPM version 4.x.x)
    

    NOTE: During your attempt to install a source RPM package, the following message may be displayed:

    error: cannot create %sourcedir /usr/src/redhat/SOURCES

    The most likely cause of the error is that the rpm-build package has not been installed. Locate the rpm-build package on the Linux installation media and install it using the following command:

    rpm -ivh rpm-build-version.arch.rpm

    Complete the installation of the source RPM.

  5. Install the newly built package (driver and man page):
  6. rpm -ivh RPMS/i386/netxtreme2-version.arch.rpm
    

    The --force option is needed if installing over an existing distribution that may already contain an older version of the driver.

    Depending on the kernel, the driver is installed to one of the following paths:

    2.4.x kernels

    /lib/modules/kernel_version/kernel/drivers/net/bnx2.o
    

    2.6.x kernels

    /lib/modules/kernel_version/kernel/drivers/net/bnx2.ko
    

    2.6.16 kernels and newer (bnx2 driver)

    /lib/modules/kernel_version/kernel/drivers/net/bnx2.ko
    
    /lib/modules/kernel_version/kernel/drivers/net/cnic.ko
    

    2.6.16 kernels and newer (bnx2i driver)

    /lib/modules/kernel_version/kernel/drivers/scsi/bnx2i/bnx2i.ko
    
  7. Load the driver:
  8. insmod bnx2
    

    or

    modprobe bnx2 
    

    Verify that your network adapter supports iSCSI by checking the message log. If the message "bnx2i: dev eth0 does not support iSCSI" appears in the message log after loading the bnx2i driver, then iSCSI is not supported. This message may not appear until the interface is opened, as with:

    ifconfig eth0 up
    
  9. Load the cnic driver (if applicable):
  10. insmod cnic.ko
    

    or

    modprobe cnic
    

To configure the network protocol and address, refer to the documentation provided with your operating system.

Installing the KMP Package

NOTE: The examples in this procedure refer to the bnx2x driver, but also apply to the bnx2 and bnx2i drivers.

  1. Install the KMP package:
  2. rpm -ivh <file>
    
    rmmod bnx2x
    
  3. Load the driver:
  4. 
    

Building the Driver from the Source TAR File

NOTE: The examples used in this procedure refer to the bnx2 driver, but also apply to the bnx2x driver.

  1. Create a directory and extract the TAR files to the directory:
  2. tar xvzf netxtreme2-version.tar.gz
    
  3. Build the driver bnx2.ko (or bnx2.o) as a loadable module for the running kernel:
  4. cd netxtreme2-version
    make
  5. Test the driver by loading it (first unload the existing driver, if necessary):
  6. rmmod bnx2
    
    insmod bnx2.o
    
    modprobe crc32 && insmod bnx2.o
    
    or, for Linux 2.6 kernels:
    
    rmmod bnx2
    
    insmod bnx2.ko
    

    Verify that your network adapter supports iSCSI by checking the message log. If the message "bnx2i: dev eth0 does not support iSCSI" appears in the message log after loading the bnx2i driver, then iSCSI is not supported. This message may not appear until the interface is opened, as with:

    ifconfig eth0 up
    
  7. Load the cnic driver (if applicable):
  8. insmod cnic.ko
    
  9. Install the driver and man page:
  10. make install
    

    NOTE: See the RPM instructions above for the location of the installed driver.

  11. Install the user daemon (brcm_iscsiuio).

Refer to ifconfig eth0 upLoad and Run Necessary iSCSI Software Components for instructions on loading the software components required to use the Broadcom iSCSI offload feature.

To configure the network protocol and address after building the driver, refer to the manuals supplied with your operating system.

Verify that your network adapter supports iSCSI by checking the message log. If the message "bnx2i: dev eth0 does not support iSCSI" appears in the message log after loading the bnx2i driver, then iSCSI is not supported. This message may not appear until the interface is opened, as with:

ifconfig eth0 upLoad and Run Necessary iSCSI Software Components

The Broadcom iSCSI Offload software suite consists of three kernel modules and a user daemon. Required software components can be loaded either manually or through system services.

  1. Unload the existing driver, if necessary:
  2. Manual:

    rmmod bnx2i
    
  3. Load the iSCSI driver:
  4. Manual:

    insmod bnx2i.ko
    

    or

    modprobe bnx2i
    

Unloading/Removing the Linux Driver

Unloading/Removing the Driver from an RPM Installation

Removing the Driver from a TAR Installation

Unloading/Removing the Driver from an RPM Installation

NOTES:

To unload the driver, use ifconfig to bring down all eth# interfaces opened by the driver, and then type the following:

rmmod bnx2

NOTE: The above command will also remove bnx2, bnx2x, and cnic modules.

If the driver was installed using RPM, do the following to remove it:

rpm -e netxtreme2

Removing the Driver from a TAR Installation

NOTE: The examples used in this procedure refer to the bnx2 driver, but also apply to the bnx2x and bnx2i drivers.

If the driver was installed using make install from the tar file, the bnx2.o or bnx2.ko driver file has to be manually deleted from the operating system. See Installing the Source RPM Package for the location of the installed driver.

Patching PCI Files (Optional)

NOTE: The examples used in this procedure refer to the bnx2 driver, but also apply to the bnx2x and bnx2i drivers.

For hardware detection utilities such as Red Hat kudzu to properly identify bnx2 supported devices, a number of files containing PCI vendor and device information may need to be updated.

Apply the updates by running the scripts provided in the supplemental tar file. For example, on Red Hat Enterprise Linux, apply the updates by doing the following:

./patch_pcitbl.sh  /usr/share/hwdata/pcitable pci.updates 
/usr/share/hwdata/pcitable.new bnx2
./patch_pciids.sh /usr/share/hwdata/pci.ids pci.updates 
/usr/share/hwdata/pci.ids.new

Next, the old files can be backed up and the new files can be renamed for use.

cp /usr/share/hwdata/pci.ids /usr/share/hwdata/old.pci.ids
cp /usr/share/hwdata/pci.ids.new /usr/share/hwdata/pci.ids
cp /usr/share/hwdata/pcitable /usr/share/hwdata/old.pcitable
cp /usr/share/hwdata/pcitable.new /usr/share/hwdata/pcitable

Network Installations

For network installations through NFS, FTP, or HTTP (using a network boot disk or PXE), a driver disk that contains the bnx2/bnx2x driver may be needed. The driver disk images for the most recent Red Hat and SuSE versions are included. Boot drivers for other Linux versions can be compiled by modifying the Makefile and the make environment. Further information is available from the Red Hat website, http://www.redhat.com.

Setting Values for Optional Properties

Optional properties exist for the different drivers:

bnx2 Driver

disable_msi

The disable_msi optional property can be supplied as a command line argument to the insmod or modprobe command. The property can also be set in modprobe.conf. See the man page for more information. All other driver settings can be queried and changed using the ethtool utility. See the ethtool man page for more information. The ethtool settings do not persist across a reboot or module reload. The ethtool commands can be put in a startup script such as /etc/rc.local to preserve the settings across a reboot.

NOTE: Some combinations of property values may conflict and result in failures. The driver cannot detect all such conflicting combinations.

This property is used to disable Message Signal Interrupts (MSI), and the property is valid only on 2.6 kernels that support MSI. On 2.4 kernels, this property cannot be used. By default, the driver enables MSI if it is supported by the kernel. It runs an interrupt test during initialization to determine if MSI is working. If the test passes, the driver enables MSI. Otherwise, it uses legacy INTx mode.

insmod bnx2.ko disable_msi=1

or

modprobe bnx2 disable_msi=1

bnx2x Driver

disable_tpa

The disable_tpa parameter can be supplied as a command line argument to disable the Transparent Packet Aggregation (TPA) feature. By default, the driver will aggregate TCP packets. Use disable_tpa to disable the advanced TPA feature.

Set the disable_tpa parameter to 1 as shown below to disable the TPA feature on all NetXtreme II network adapters in the system. The parameter can also be set in modprobe.conf. See the man page for more information.

insmod bnx2x.ko disable_tpa=1

or

modprobe bnx2x disable_tpa=1

int_mode

The int_mode parameter is used to force using an interrupt mode.

Set the int_mode parameter to 1 to force using the legacy INTx mode on all NetXtreme II adapters in the system.

insmod bnx2x.ko int_mode=1

or

modprobe bnx2x int_mode=1

Set the int_mode parameter to 2 to force using MSI mode on all NetXtreme II adapters in the system.

insmod bnx2x.ko int_mode=2

or

modprobe bnx2x int_mode=2

Set the int_mode parameter to 3 to force using MSI-X mode on all NetXtreme II adapters in the system.

dropless_fc

The dropless_fc parameter can be used to enable a complementary flow control mechanism on BCM57711/BCM57712 adapters. The default flow control mechanism is to send pause frames when the on-chip buffer (BRB) is reaching a certain level of occupancy. This is a performance targeted flow control mechanism. On BCM57711/BCM57712 adapters, one can enable another flow control mechanism to send pause frames, where one of the host buffers (when in RSS mode) are exhausted.

This is a "zero packet drop" targeted flow control mechanism.

Set the dropless_fc parameter to 1 to enable the dropless flow control mechanism feature on all BCM57711/BCM57712 NetXtreme II adapters in the system.

insmod bnx2x.ko dropless_fc=1

or

modprobe bnx2x dropless_fc=1

disable_iscsi_ooo

The disable_iscsi_ooo parameter is to disable the allocation of the iSCSI TCP Out-of-Order (OOO) reception resources, specifically for VMware for low-memory systems.

multi_mode

The optional parameter multi_mode is for use on systems that support multi-queue networking. Multi-queue networking on the receive side depends only on MSI-X cability of the system, multi-queue networking on the transmit side is supported only on kernels starting from 2.6.27. By default, multi_mode parameter is set to 1. Thus, on kernels up to 2.6.26, the driver will allocate on the receive side one queue per-CPU and on the transmit side only one queue. On kernels starting from 2.6.27, the driver will allocate on both receive and transmit sides, one queue per-CPU. In any case, the number of allocated queues will be limited by number of queues supported by hardware.

The multi_mode optional parameter can also be used to enable SAFC (Service Aware Flow Control) by differentiating the traffic to up to 3 CoS (Class of Service) in the hardware according to the VLAN PRI value or according to the IP DSCP value (least 3 bits).

num_queues

The optional parameter num_queues may be used to set the number of queues when multi_mode is set to 1 and interrupt mode is MSI-X. If interrupt mode is different than MSI-X (see int_mode), the number of queues will be set to 1, discarding the value of this parameter.

pri_map

The optional parameter pri_map is used to map the VLAN PRI value or the IP DSCP value to a different or same CoS in the hardware. This 32-bit parameter is evaluated by the driver as an 8 value of 4 bits each. Each nibble sets the desired hardware queue number for that priority. For example, set pri_map to 0x11110000 to map priority 0 to 3 to CoS 0 and map priority 4 to 7 to CoS 1.

qs_per_cos

The optional parameter qs_per_cos is used to specify how many queues will share the same CoS. This parameter is evaluated by the driver up to 3 values of 8 bits each. Each byte sets the desired number of queues for that CoS. The total number of queues is limited by the hardware limit. For example, set qs_per_cos to 0x10101 to create a total of three queues, one per CoS. In another example, set qs_per_cos to 0x404 to create a total of 8 queues, divided into 2 CoS, 4 queues in each CoS.

cos_min_rate

The optional parameter cos_min_rate is used to determine the weight of each CoS for round-robin scheduling in transmission. This parameter is evaluated by the driver as up to 3 values of 8 bits each. Each byte sets the desired weight for that CoS. The weight ranges from 0 to 100. For example, set cos_min_rate to 0x101 for fair transmission rate between 2 CoS. In another example, set the cos_min_rate to 0x30201 to give CoS the higher rate of transmission. To avoid using the fairness algorithm, omit setting cos_min_rate or set it to 0.

Set the multi_mode parameter to 2 as shown below to differentiate the traffic according to the VLAN PRI value.

insmod bnx2x.ko multi_mode=2 pri_map=0x11110000 qs_per_cos=0x404

or

modprobe bnx2x multi_mode=2 pri_map=0x11110000 qs_per_cos=0x404

Set the multi_mode parameter to 4 as shown below to differentiate the traffic according to the IP DSCP value.

insmod bnx2x.ko multi_mode=4 pri_map=0x22221100 qs_per_cos=0x10101 cos_min_rate=0x30201

or

modprobe bnx2x multi_mode=4 pri_map=0x22221100 qs_per_cos=0x10101 cos_min_rate=0x30201

bnx2i Driver

Optional parameters en_tcp_dack, error_mask1, and error_mask2 can be supplied as command line arguments to the insmod or modprobe command for bnx2i.

error_mask1 and error_mask2

"Config FW iSCSI Error Mask #", use to configure certain iSCSI protocol violation to be treated either as a warning or a fatal error. All fatal iSCSI protocol violations will result in session recovery (ERL 0). These are bit masks.

Defaults: All violations will be treated as errors.

CAUTION! Do not use error_mask if you are not sure about the consequences. These values are to be discussed with Broadcom development team on a case-by-case basis. This is just a mechanism to work around iSCSI implementation issues on the target side and without proper knowledge of iSCSI protocol details, users are advised not to experiment with these parameters.

en_tcp_dack

"Enable TCP Delayed ACK", enables/disables TCP delayed ACK feature on offloaded iSCSI connections.

Defaults: TCP delayed ACK is ENABLED. For example:

insmod bnx2i.ko en_tcp_dack=0

or

modprobe bnx2i en_tcp_dack=0

time_stamps

"Enable TCP TimeStamps", enables/disables TCP time stamp feature on offloaded iSCSI connections.

Defaults: TCP time stamp option is DISABLED. For example:

insmod bnx2i.ko time_stamps=1

or

modprobe bnx2i time_stamps=1

sq_size

"Configure SQ size", used to choose send queue size for offloaded connections and SQ size determines the maximum SCSI commands that can be queued. SQ size also has a bearing on the number of connections that can be offloaded; as QP size increases, the number of connections supported will decrease. With the default values, the BCM5706/BCM5708 adapter can offload 28 connections.

Defaults: 128

Range: 32 to 128

Note that Broadcom validation is limited to a power of 2; for example, 32, 64, 128.

rq_size

"Configure RQ size", used to choose the size of asynchronous buffer queue size per offloaded connections. RQ size is not required greater than 16 as it is used to place iSCSI ASYNC/NOP/REJECT messages and SCSI sense data.

Defaults: 16

Range: 16 to 32

Note that Broadcom validation is limited to a power of 2; for example, 16, 32.

event_coal_div

"Event Coalescing Divide Factor", performance tuning parameter used to moderate the rate of interrupt generation by the iscsi firmware.

Defaults: 1

Valid values: 1, 2, 4, 8

last_active_tcp_port

"Last active TCP port used", status parameter used to indicate the last TCP port number used in the iSCSI offload connection.

Defaults: N/A

Valid values: N/A

Note: This is a read-only parameter.

ooo_enable

"Enable TCP out-of-order feature", enables/disables TCP out-of-order rx handling feature on offloaded iSCSI connections.

Defaults: TCP out-of-order feature is ENABLED. For example:

insmod bnx2i.ko ooo_enable=1

or

modprobe bnx2i ooo_enable=1

Driver Defaults

bnx2 Driver

Speed: Autonegotiation with all speeds advertised

Flow Control: Autonegotiation with RX and TX advertised

MTU: 1500 (range is 46–9000)

RX Ring Size: 255 (range is 0–4080)

RX Jumbo Ring Size: 0 (range 0–16320) adjusted by the driver based on MTU and RX Ring Size

TX Ring Size: 255 (range is (MAX_SKB_FRAGS+1)–255). MAX_SKB_FRAGS varies on different kernels and different architectures. On a 2.6 kernel for x86, MAX_SKB_FRAGS is 18.

Coalesce RX Microseconds: 18 (range is 0–1023)

Coalesce RX Microseconds IRQ: 18 (range is 0–1023)

Coalesce RX Frames: 6 (range is 0–255)

Coalesce RX Frames IRQ: 6 (range is 0–255)

Coalesce TX Microseconds: 80 (range is 0–1023)

Coalesce TX Microseconds IRQ: 80 (range is 0–1023)

Coalesce TX Frames: 20 (range is 0–255)

Coalesce TX Frames IRQ: 20 (range is 0–255)

Coalesce Statistics Microseconds: 999936 (approximately 1 second) (range is 0–16776960 in increments of 256)

MSI: Enabled (if supported by the 2.6 kernel and the interrupt test passes)

TSO: Enabled (on 2.6 kernels)

WoL: Initial setting based on NVRAM's setting

bnx2x Driver

Speed: Autonegotiation with all speeds advertised

Flow control: Autonegotiation with RX and TX advertised

MTU: 1500 (range is 46–9000)

RX Ring Size: 4078 (range is 0–4078)

TX Ring Size: 4078 (range is (MAX_SKB_FRAGS+4)–4078). MAX_SKB_FRAGS varies on different kernels and different architectures. On a 2.6 kernel for x86, MAX_SKB_FRAGS is 18.

Coalesce RX Microseconds: 25 (range is 0–3000)

Coalesce TX Microseconds: 50 (range is 0–12288)

Coalesce Statistics Microseconds: 999936 (approximately 1 second) (range is 0–16776960 in increments of 256)

MSI-X: Enabled (if supported by the 2.6 kernel and the interrupt test passes)

TSO: Enabled

WoL: Disabled

Driver Messages

The following are the most common sample messages that may be logged in the /var/log/messages file. Use dmesg -n <level> to control the level at which messages appear on the console. Most systems are set to level 6 by default. To see all messages, set the level higher.

bnx2 and bnx2x Driver

NOTE: The examples used in this procedure refer to the bnx2 driver, but also apply to the bnx2x driver.

Driver Sign on

Broadcom NetXtreme II Gigabit Ethernet Driver 
bnx2 v1.6.3c (July 23, 2007)

CNIC Driver Sign on (bnx2 only)

Broadcom NetXtreme II cnic v1.1.19 (Sep 25, 2007)

NIC Detected

eth#: Broadcom NetXtreme II BCM5708 1000Base-T (B0) 
PCI-X 64-bit 133MHz found at mem f6000000, IRQ 16, node addr 0010180476ae
cnic: Added CNIC device: eth0

Link Up and Speed Indication

bnx2: eth# NIC Link is Up, 1000 Mbps full duplex

Link Down Indication

bnx2: eth# NIC Link is Down

MSI enabled successfully (bnx2 only)

bnx2: eth0: using MSI

MSI-X enabled successfully (bnx2x only)

bnx2x: eth0: using MSI-X

bnx2i Driver

BNX2I Driver signon

Broadcom NetXtreme II iSCSI Driver bnx2i v2.1.1D (May 12, 2010)

Network port to iSCSI transport name binding

bnx2i: netif=eth2, iscsi=bcm570x-050000
bnx2i: netif=eth1, iscsi=bcm570x-030c00

Driver completes handshake with iSCSI offload-enabled CNIC device

bnx2i [05:00.00]: ISCSI_INIT passed

NOTE: This message is displayed only when the user attempts to make an iSCSI connection.

Driver detects iSCSI offload is not enabled on the CNIC device

bnx2i: iSCSI not supported, dev=eth3
bnx2i: bnx2i: LOM is not enabled to offload iSCSI connections, dev=eth0
bnx2i: dev eth0 does not support iSCSI

Exceeds maximum allowed iSCSI connection offload limit

bnx2i: alloc_ep: unable to allocate iscsi cid
bnx2i: unable to allocate iSCSI context resources

Network route to target node and transport name binding are two different devices

bnx2i: conn bind, ep=0x... ($ROUTE_HBA) does not belong to hba $USER_CHOSEN_HBA

where ROUTE_HBA --> net device on which connection was offloaded based on route information USER_CHOSEN_HBA --> HBA to which target node is bound (using iscsi transport name)

Target cannot be reached on any of the CNIC devices

bnx2i: check route, cannot connect using cnic

Network route is assigned to network interface, which is down

bnx2i: check route, hba not found

Attempting to offload iSCSI connection onto a Jumbo Frame-enabled device

bnx2i: eth# network i/f mtu is set to #mtu
bnx2i: iSCSI HBA can support mtu of 1500

NOTE: Change mtu to 1500 using ifconfig and restart the interface in order to offload iSCSI connections.

SCSI-ML initiated host reset (session recovery)

bnx2i: attempting to reset host, #3

CNIC detects iSCSI protocol violation - Fatal errors

bnx2i: iscsi_error - wrong StatSN rcvd
bnx2i: iscsi_error - hdr digest err
bnx2i: iscsi_error - data digest err
bnx2i: iscsi_error - wrong opcode rcvd
bnx2i: iscsi_error - AHS len > 0 rcvd
bnx2i: iscsi_error - invalid ITT rcvd
bnx2i: iscsi_error - wrong StatSN rcvd
bnx2i: iscsi_error - wrong DataSN rcvd
bnx2i: iscsi_error - pend R2T violation
bnx2i: iscsi_error - ERL0, UO
bnx2i: iscsi_error - ERL0, U1
bnx2i: iscsi_error - ERL0, U2
bnx2i: iscsi_error - ERL0, U3
bnx2i: iscsi_error - ERL0, U4
bnx2i: iscsi_error - ERL0, U5
bnx2i: iscsi_error - ERL0, U
bnx2i: iscsi_error - invalid resi len
bnx2i: iscsi_error - MRDSL violation
bnx2i: iscsi_error - F-bit not set
bnx2i: iscsi_error - invalid TTT
bnx2i: iscsi_error - invalid DataSN
bnx2i: iscsi_error - burst len violation
bnx2i: iscsi_error - buf offset violation
bnx2i: iscsi_error - invalid LUN field
bnx2i: iscsi_error - invalid R2TSN field
bnx2i: iscsi_error - invalid cmd len1
bnx2i: iscsi_error - invalid cmd len2
bnx2i: iscsi_error - pend r2t exceeds MaxOutstandingR2T value
bnx2i: iscsi_error - TTT is rsvd
bnx2i: iscsi_error - MBL violation
bnx2i: iscsi_error - data seg len != 0
bnx2i: iscsi_error - reject pdu len error
bnx2i: iscsi_error - async pdu len error
bnx2i: iscsi_error - nopin pdu len error
bnx2i: iscsi_error - pend r2t in cleanup
bnx2i: iscsi_error - IP fragments rcvd
bnx2i: iscsi_error - IP options error
bnx2i: iscsi_error - urgent flag error

CNIC detects iSCSI protocol violation - non-FATAL, warning

bnx2i: iscsi_warning - invalid TTT
bnx2i: iscsi_warning - invalid DataSN
bnx2i: iscsi_warning - invalid LUN field

NOTE: The driver needs to be configured to consider certain violation to treat as warning and not as a critical error.

Driver puts a session through recovery

conn_err - hostno 3 conn 03fbcd00, iscsi_cid 2 cid a1800

Reject iSCSI PDU received from the target

bnx2i - printing rejected PDU contents
[0]: 1 ffffffa1 0 0 0 0 20 0
[8]: 0 7 0 0 0 0 0 0
[10]: 0 0 40 24 0 0 ffffff80 0
[18]: 0 0 3 ffffff88 0 0 3 4b
[20]: 2a 0 0 2 ffffffc8 14 0 0
[28]: 40 0 0 0 0 0 0 0

Open-iSCSI daemon handing over session to driver

bnx2i: conn update - MBL 0x800 FBL 0x800MRDSL_I 0x800 MRDSL_T 0x2000

Teaming with Channel Bonding

With the Linux drivers, you can team adapters together using the bonding kernel module and a channel bonding interface. For more information, see the Channel Bonding information in your operating system documentation.

Statistics

Detailed statistics and configuration information can be viewed using the ethtool utility. See the ethtool man page for more information.

Linux iSCSI Offload

User Application - brcm_iscsiuio

Run the brcm_iscsiuio daemon before attempting to create iSCSI connections. The driver will not be able to establish connections to the iSCSI target without the daemon's assistance.

brcm_iscsiuio

Open iSCSI User Applications

Install and run the open-iscsi programs, iscsid and iscsiadm, from the Broadcom distributed open-iscsi packages. Refer to Packaging for details. All pre-installed open-iscsi packages need to be removed before the Broadcom iSCSI-supported packages can be installed.

  1. Remove all existing open-iscsi packages.
  2. RHEL5

    rpm -e iscsi-initiator-utils
    
  3. Install the source RPM package.
  4. rpm -ivh <open-iscsi-package-name>.src.rpm
    
  5. CD to the RPM path and build the binary driver for your kernel.
  6. cd /usr/src/{redhat,OpenLinux,turbo,packages,rpm ..}
    
    rpm -bb SPECS/<open-iscsi-package-name>.spec
    

    or

    rpmbuild -bb SPECS/<open-iscsi-package-name>.spec (for RPM version 4.x.x)
    

    Note that the RPM path is different for different Linux distributions.

  7. Install the newly built package.
  8. rpm -ivh RPMS/<arch>/<open-iscsi-package-name>.<arch>.rpm
    

    where <arch> is the machine architecture such as i386.

    rpm -ivh RPMS/i386/<open-iscsi-package-name>.i386.rpm
    
  9. Start the daemon.
  10. iscsid
    

Bind iSCSI Target to Broadcom NX2 iSCSI Transport Name

By default, the open-iscsi daemon connects to discovered targets using software initiator (transport name = 'tcp'). Users who wish to offload iSCSI connection onto CNIC device should explicitly change transport binding of the iSCSI node. This can be done using iscsiadm cli utility as follows,

iscsiadm --mode node --targetname iqn.2004-06.com.broadcom:tg1 \
		--portal 192.168.1.100 --op=update			 \
		--name=iface_file_name

where the iface file includes the following information for RHEL 5.4, RHEL 5.5, and SLES 11 SP1:

iface.net_ifacename = ethX
iface.iscsi_ifacename = <name of the iface file>
iface.hwaddress = XX:XX:XX:XX:XX:XX
iface.ipaddress = XX:XX:XX:XX
iface.transport_name = bnx2i

Ensure that the iface.hwaddress is in lower case format.

If you wish to switch back to use the software initiator, use the following:

iscsiadm --mode node --targetname iqn.2004-06.com.broadcom:tg1 \
		--portal 192.168.1.100 --op=update			 \
		--name=iface_file_name

where the iface file includes the following information:

iface.net_ifacename = ethX
iface.iscsi_ifacename = <name of the iface file>
iface.transport_name = tcp

Making Connections to iSCSI Targets

Refer to open-iscsi documentation for a comprehensive list of iscsiadm commands. This is a sample list of commands to discovery targets and to create iscsi connections to a target.

Add static entry

iscsiadm -m node -p <ipaddr[:port]> -T iqn.2007-05.com.broadcom:target1 -o new 

iSCSI target discovery using 'SendTargets'

iscsiadm -m discovery --type sendtargets -p <ipaddr[:port]> 

Bind iSCSI target to Broadcom NX2 transport (bnx2i)

iscsiadm -m node -p <ipaddr[:port]> -T <iqn.targetname> --op=update --name=iface_file_name

Login to target using 'iscsiadm' command

iscsiadm --mode node --targetname <iqn.targetname> --portal <ipaddr[:port]> --login

List all drives active in the system

fdisk -l

Maximize Offload iSCSI Connections

With default driver parameters set, which includes 128 outstanding commands, bnx2i can offload the following number of connections:

BCM5706/BCM5708: 28

BCM5709: 43

BCM5771x: 128

This is not a hard limit, but just a simple on-chip resource allocation math. bnx2i will be able to offload > 28 connections on 1G devices by reducing the shared queue size, which in turn limits the maximum outstanding tasks on a connection. See Setting Values for Optional Properties for information on sq_size and rq_size. The driver logs the following message to syslog when the maximum allowed connection offload limit is reached - "bnx2i: unable to allocate iSCSI context resources".

Linux iSCSI Offload FAQ


Back to Contents Page