Installing Linux Driver Software
Unloading/Removing the Linux Driver
Setting Values for Optional Properties
This section discusses the Linux drivers for the Broadcom NetXtreme II network adapters.
The current version of the driver has been tested on 2.4.x kernels (starting from 2.4.24) and all 2.6.x kernels. The driver may not compile on kernels older than 2.4.24.
NOTE: Support for the 2.4.21 kernels is provided in Red Hat Enterprise Linux 3.
Testing is concentrated on i386 and x86_64 architectures. Only limited testing has been done on other architectures. Minor changes to some source files and Makefile may be needed on some kernels. Additionally, the Makefile will not compile the cnic driver on kernels older than 2.6.16. iSCSI offload is only supported on 2.6.16 and newer kernels.
NOTE: For Broadcom NetXtreme II BCM5708 devices with a silicon revision prior to B2, the open source bnx2 driver does not support the reporting and configuration of NetXtreme II WOL settings via ethtool. For silicon revisions of B2 or later, the bnx2 driver reports support for Magic Packet WOL via ethtool. Enabling support via ethtool is mandatory to successfully wake the system. To determine the silicon revision of your Broadcom NetXtreme II device, use the
lspci
command, where "10" = revision B0, "11" = revision B1, and "12" = revision B2.
The current version of the driver has been tested on 2.6.x kernels starting from 2.6.9. The driver may not compile on kernels older than 2.6.9. Testing is concentrated on i386 and x86_64 architectures. Only limited testing has been done on some other architectures. Minor changes to some source files and Makefile may be needed on some kernels.
The current version of the driver has been tested on 2.6.x kernels, starting from 2.6.18 kernel. The driver may not compile on older kernels. Testing is concentrated on i386 and x86_64 architectures, Red Hat EL5, and SUSE 11 SP1 distributions.
The Linux driver is released in the packaging formats shown in Table 2. The netxtreme2 package contains the bnx2 (1 Gb network adapter) driver, the bnx2x (10 Gb network adapter) driver, and the bnx2i (iSCSI) driver for source RPM and compressed tar.
Format
|
Drivers
|
---|---|
Source RPM
|
netxtreme2-version.src.rpm
|
Compressed tar
|
netxtreme2-version.tar.gz
|
Supplemental tar
|
netxtreme2_sup-version.tar.gz
|
Identical source files to build the driver are included in both RPM and TAR source packages. The supplemental tar file contains additional utilities such as patches and driver diskette images for network installation.
Updated open-iSCSI components are released in source RPM format.
Installing the Source RPM Package
Building the Driver from the Source TAR File
NOTE: If a bnx2/bnx2x/bnx2i driver is loaded and the Linux kernel is updated, the driver module must be recompiled if the driver module was installed using the source RPM or the TAR package.
NOTE: The examples in this procedure refer to the bnx2 driver, but also apply to the bnx2x and bnx2i drivers.
rpm -ivh netxtreme2-version.src.rpm
cd /usr/src/redhat,OpenLinux,turbo,packages,rpm ...
rpm -bb SPECS/bnx2.spec
or
rpmbuild -bb SPECS/netxtreme2.spec (for RPM version 4.x.x)
NOTE: During your attempt to install a source RPM package, the following message may be displayed:
error: cannot create %sourcedir /usr/src/redhat/SOURCES
The most likely cause of the error is that the rpm-build package has not been installed. Locate the rpm-build package on the Linux installation media and install it using the following command:
rpm -ivh rpm-build-version.arch.rpm
Complete the installation of the source RPM.
rpm -ivh RPMS/i386/netxtreme2-version.arch.rpm
The --force option is needed if installing over an existing distribution that may already contain an older version of the driver.
Depending on the kernel, the driver is installed to one of the following paths:
2.4.x kernels
/lib/modules/kernel_version/kernel/drivers/net/bnx2.o
2.6.x kernels
/lib/modules/kernel_version/kernel/drivers/net/bnx2.ko
2.6.16 kernels and newer (bnx2 driver)
/lib/modules/kernel_version/kernel/drivers/net/bnx2.ko
/lib/modules/kernel_version/kernel/drivers/net/cnic.ko
2.6.16 kernels and newer (bnx2i driver)
/lib/modules/kernel_version/kernel/drivers/scsi/bnx2i/bnx2i.ko
insmod bnx2
or
modprobe bnx2
Verify that your network adapter supports iSCSI by checking the message log. If the message "bnx2i: dev eth0 does not support iSCSI" appears in the message log after loading the bnx2i driver, then iSCSI is not supported. This message may not appear until the interface is opened, as with:
ifconfig eth0 up
insmod cnic.ko
or
modprobe cnic
To configure the network protocol and address, refer to the documentation provided with your operating system.
NOTE: The examples in this procedure refer to the bnx2x driver, but also apply to the bnx2 and bnx2i drivers.
NOTE: The examples used in this procedure refer to the bnx2 driver, but also apply to the bnx2x driver.
tar xvzf netxtreme2-version.tar.gz
cd netxtreme2-version
make
rmmod bnx2
insmod bnx2.o
modprobe crc32 && insmod bnx2.o
or, for Linux 2.6 kernels:
rmmod bnx2
insmod bnx2.ko
Verify that your network adapter supports iSCSI by checking the message log. If the message "bnx2i: dev eth0 does not support iSCSI" appears in the message log after loading the bnx2i driver, then iSCSI is not supported. This message may not appear until the interface is opened, as with:
ifconfig eth0 up
insmod cnic.ko
make install
NOTE: See the RPM instructions above for the location of the installed driver.
Refer to ifconfig eth0 upLoad and Run Necessary iSCSI Software Components for instructions on loading the software components required to use the Broadcom iSCSI offload feature.
To configure the network protocol and address after building the driver, refer to the manuals supplied with your operating system.
Verify that your network adapter supports iSCSI by checking the message log. If the message "bnx2i: dev eth0 does not support iSCSI" appears in the message log after loading the bnx2i driver, then iSCSI is not supported. This message may not appear until the interface is opened, as with:
The Broadcom iSCSI Offload software suite consists of three kernel modules and a user daemon. Required software components can be loaded either manually or through system services.
Unloading/Removing the Driver from an RPM Installation
Removing the Driver from a TAR Installation
NOTES:
To unload the driver, use ifconfig to bring down all eth# interfaces opened by the driver, and then type the following:
rmmod bnx2
NOTE: The above command will also remove bnx2, bnx2x, and cnic modules.
If the driver was installed using RPM, do the following to remove it:
rpm -e netxtreme2
NOTE: The examples used in this procedure refer to the bnx2 driver, but also apply to the bnx2x and bnx2i drivers.
If the driver was installed using make install from the tar file, the bnx2.o or bnx2.ko driver file has to be manually deleted from the operating system. See Installing the Source RPM Package for the location of the installed driver.
NOTE: The examples used in this procedure refer to the bnx2 driver, but also apply to the bnx2x and bnx2i drivers.
For hardware detection utilities such as Red Hat kudzu to properly identify bnx2 supported devices, a number of files containing PCI vendor and device information may need to be updated.
Apply the updates by running the scripts provided in the supplemental tar file. For example, on Red Hat Enterprise Linux, apply the updates by doing the following:
./patch_pcitbl.sh /usr/share/hwdata/pcitable pci.updates
/usr/share/hwdata/pcitable.new bnx2
./patch_pciids.sh /usr/share/hwdata/pci.ids pci.updates
/usr/share/hwdata/pci.ids.new
Next, the old files can be backed up and the new files can be renamed for use.
cp /usr/share/hwdata/pci.ids /usr/share/hwdata/old.pci.ids
cp /usr/share/hwdata/pci.ids.new /usr/share/hwdata/pci.ids
cp /usr/share/hwdata/pcitable /usr/share/hwdata/old.pcitable
cp /usr/share/hwdata/pcitable.new /usr/share/hwdata/pcitable
For network installations through NFS, FTP, or HTTP (using a network boot disk or PXE), a driver disk that contains the bnx2/bnx2x driver may be needed. The driver disk images for the most recent Red Hat and SuSE versions are included. Boot drivers for other Linux versions can be compiled by modifying the Makefile and the make environment. Further information is available from the Red Hat website, http://www.redhat.com.
Optional properties exist for the different drivers:
The disable_msi optional property can be supplied as a command line argument to the insmod or modprobe command. The property can also be set in modprobe.conf. See the man page for more information. All other driver settings can be queried and changed using the ethtool utility. See the ethtool man page for more information. The ethtool settings do not persist across a reboot or module reload. The ethtool commands can be put in a startup script such as /etc/rc.local to preserve the settings across a reboot.
NOTE: Some combinations of property values may conflict and result in failures. The driver cannot detect all such conflicting combinations.
This property is used to disable Message Signal Interrupts (MSI), and the property is valid only on 2.6 kernels that support MSI. On 2.4 kernels, this property cannot be used. By default, the driver enables MSI if it is supported by the kernel. It runs an interrupt test during initialization to determine if MSI is working. If the test passes, the driver enables MSI. Otherwise, it uses legacy INTx mode.
insmod bnx2.ko disable_msi=1
or
modprobe bnx2 disable_msi=1
disable_tpa
The disable_tpa parameter can be supplied as a command line argument to disable the Transparent Packet Aggregation (TPA) feature. By default, the driver will aggregate TCP packets. Use disable_tpa to disable the advanced TPA feature.
Set the disable_tpa parameter to 1 as shown below to disable the TPA feature on all NetXtreme II network adapters in the system. The parameter can also be set in modprobe.conf. See the man page for more information.
insmod bnx2x.ko disable_tpa=1
or
modprobe bnx2x disable_tpa=1
int_mode
The int_mode parameter is used to force using an interrupt mode.
Set the int_mode parameter to 1 to force using the legacy INTx mode on all NetXtreme II adapters in the system.
insmod bnx2x.ko int_mode=1
or
modprobe bnx2x int_mode=1
Set the int_mode parameter to 2 to force using MSI mode on all NetXtreme II adapters in the system.
insmod bnx2x.ko int_mode=2
or
modprobe bnx2x int_mode=2
Set the int_mode parameter to 3 to force using MSI-X mode on all NetXtreme II adapters in the system.
dropless_fc
The dropless_fc parameter can be used to enable a complementary flow control mechanism on BCM57711/BCM57712 adapters. The default flow control mechanism is to send pause frames when the on-chip buffer (BRB) is reaching a certain level of occupancy. This is a performance targeted flow control mechanism. On BCM57711/BCM57712 adapters, one can enable another flow control mechanism to send pause frames, where one of the host buffers (when in RSS mode) are exhausted.
This is a "zero packet drop" targeted flow control mechanism.
Set the dropless_fc parameter to 1 to enable the dropless flow control mechanism feature on all BCM57711/BCM57712 NetXtreme II adapters in the system.
insmod bnx2x.ko dropless_fc=1
or
modprobe bnx2x dropless_fc=1
disable_iscsi_ooo
The disable_iscsi_ooo parameter is to disable the allocation of the iSCSI TCP Out-of-Order (OOO) reception resources, specifically for VMware for low-memory systems.
multi_mode
The optional parameter multi_mode is for use on systems that support multi-queue networking. Multi-queue networking on the receive side depends only on MSI-X cability of the system, multi-queue networking on the transmit side is supported only on kernels starting from 2.6.27. By default, multi_mode parameter is set to 1. Thus, on kernels up to 2.6.26, the driver will allocate on the receive side one queue per-CPU and on the transmit side only one queue. On kernels starting from 2.6.27, the driver will allocate on both receive and transmit sides, one queue per-CPU. In any case, the number of allocated queues will be limited by number of queues supported by hardware.
The multi_mode optional parameter can also be used to enable SAFC (Service Aware Flow Control) by differentiating the traffic to up to 3 CoS (Class of Service) in the hardware according to the VLAN PRI value or according to the IP DSCP value (least 3 bits).
num_queues
The optional parameter num_queues may be used to set the number of queues when multi_mode is set to 1 and interrupt mode is MSI-X. If interrupt mode is different than MSI-X (see int_mode), the number of queues will be set to 1, discarding the value of this parameter.
pri_map
The optional parameter pri_map is used to map the VLAN PRI value or the IP DSCP value to a different or same CoS in the hardware. This 32-bit parameter is evaluated by the driver as an 8 value of 4 bits each. Each nibble sets the desired hardware queue number for that priority. For example, set pri_map to 0x11110000 to map priority 0 to 3 to CoS 0 and map priority 4 to 7 to CoS 1.
qs_per_cos
The optional parameter qs_per_cos is used to specify how many queues will share the same CoS. This parameter is evaluated by the driver up to 3 values of 8 bits each. Each byte sets the desired number of queues for that CoS. The total number of queues is limited by the hardware limit. For example, set qs_per_cos to 0x10101 to create a total of three queues, one per CoS. In another example, set qs_per_cos to 0x404 to create a total of 8 queues, divided into 2 CoS, 4 queues in each CoS.
cos_min_rate
The optional parameter cos_min_rate is used to determine the weight of each CoS for round-robin scheduling in transmission. This parameter is evaluated by the driver as up to 3 values of 8 bits each. Each byte sets the desired weight for that CoS. The weight ranges from 0 to 100. For example, set cos_min_rate to 0x101 for fair transmission rate between 2 CoS. In another example, set the cos_min_rate to 0x30201 to give CoS the higher rate of transmission. To avoid using the fairness algorithm, omit setting cos_min_rate or set it to 0.
Set the multi_mode parameter to 2 as shown below to differentiate the traffic according to the VLAN PRI value.
insmod bnx2x.ko multi_mode=2 pri_map=0x11110000 qs_per_cos=0x404
or
modprobe bnx2x multi_mode=2 pri_map=0x11110000 qs_per_cos=0x404
Set the multi_mode parameter to 4 as shown below to differentiate the traffic according to the IP DSCP value.
insmod bnx2x.ko multi_mode=4 pri_map=0x22221100 qs_per_cos=0x10101 cos_min_rate=0x30201
or
modprobe bnx2x multi_mode=4 pri_map=0x22221100 qs_per_cos=0x10101 cos_min_rate=0x30201
Optional parameters en_tcp_dack, error_mask1, and error_mask2 can be supplied as command line arguments to the insmod or modprobe command for bnx2i.
error_mask1 and error_mask2
"Config FW iSCSI Error Mask #", use to configure certain iSCSI protocol violation to be treated either as a warning or a fatal error. All fatal iSCSI protocol violations will result in session recovery (ERL 0). These are bit masks.
Defaults: All violations will be treated as errors.
CAUTION! Do not use error_mask if you are not sure about the consequences. These values are to be discussed with Broadcom development team on a case-by-case basis. This is just a mechanism to work around iSCSI implementation issues on the target side and without proper knowledge of iSCSI protocol details, users are advised not to experiment with these parameters.
en_tcp_dack
"Enable TCP Delayed ACK", enables/disables TCP delayed ACK feature on offloaded iSCSI connections.
Defaults: TCP delayed ACK is ENABLED. For example:
insmod bnx2i.ko en_tcp_dack=0
or
modprobe bnx2i en_tcp_dack=0
time_stamps
"Enable TCP TimeStamps", enables/disables TCP time stamp feature on offloaded iSCSI connections.
Defaults: TCP time stamp option is DISABLED. For example:
insmod bnx2i.ko time_stamps=1
or
modprobe bnx2i time_stamps=1
sq_size
"Configure SQ size", used to choose send queue size for offloaded connections and SQ size determines the maximum SCSI commands that can be queued. SQ size also has a bearing on the number of connections that can be offloaded; as QP size increases, the number of connections supported will decrease. With the default values, the BCM5706/BCM5708 adapter can offload 28 connections.
Defaults: 128
Range: 32 to 128
Note that Broadcom validation is limited to a power of 2; for example, 32, 64, 128.
rq_size
"Configure RQ size", used to choose the size of asynchronous buffer queue size per offloaded connections. RQ size is not required greater than 16 as it is used to place iSCSI ASYNC/NOP/REJECT messages and SCSI sense data.
Defaults: 16
Range: 16 to 32
Note that Broadcom validation is limited to a power of 2; for example, 16, 32.
event_coal_div
"Event Coalescing Divide Factor", performance tuning parameter used to moderate the rate of interrupt generation by the iscsi firmware.
Defaults: 1
Valid values: 1, 2, 4, 8
last_active_tcp_port
"Last active TCP port used", status parameter used to indicate the last TCP port number used in the iSCSI offload connection.
Defaults: N/A
Valid values: N/A
Note: This is a read-only parameter.
ooo_enable
"Enable TCP out-of-order feature", enables/disables TCP out-of-order rx handling feature on offloaded iSCSI connections.
Defaults: TCP out-of-order feature is ENABLED. For example:
insmod bnx2i.ko ooo_enable=1
or
modprobe bnx2i ooo_enable=1
Speed: Autonegotiation with all speeds advertised
Flow Control: Autonegotiation with RX and TX advertised
MTU: 1500 (range is 469000)
RX Ring Size: 255 (range is 04080)
RX Jumbo Ring Size: 0 (range 016320) adjusted by the driver based on MTU and RX Ring Size
TX Ring Size: 255 (range is (MAX_SKB_FRAGS+1)255). MAX_SKB_FRAGS varies on different kernels and different architectures. On a 2.6 kernel for x86, MAX_SKB_FRAGS is 18.
Coalesce RX Microseconds: 18 (range is 01023)
Coalesce RX Microseconds IRQ: 18 (range is 01023)
Coalesce RX Frames: 6 (range is 0255)
Coalesce RX Frames IRQ: 6 (range is 0255)
Coalesce TX Microseconds: 80 (range is 01023)
Coalesce TX Microseconds IRQ: 80 (range is 01023)
Coalesce TX Frames: 20 (range is 0255)
Coalesce TX Frames IRQ: 20 (range is 0255)
Coalesce Statistics Microseconds: 999936 (approximately 1 second) (range is 016776960 in increments of 256)
MSI: Enabled (if supported by the 2.6 kernel and the interrupt test passes)
TSO: Enabled (on 2.6 kernels)
WoL: Initial setting based on NVRAM's setting
Speed: Autonegotiation with all speeds advertised
Flow control: Autonegotiation with RX and TX advertised
MTU: 1500 (range is 469000)
RX Ring Size: 4078 (range is 04078)
TX Ring Size: 4078 (range is (MAX_SKB_FRAGS+4)4078). MAX_SKB_FRAGS varies on different kernels and different architectures. On a 2.6 kernel for x86, MAX_SKB_FRAGS is 18.
Coalesce RX Microseconds: 25 (range is 03000)
Coalesce TX Microseconds: 50 (range is 012288)
Coalesce Statistics Microseconds: 999936 (approximately 1 second) (range is 016776960 in increments of 256)
MSI-X: Enabled (if supported by the 2.6 kernel and the interrupt test passes)
TSO: Enabled
WoL: Disabled
The following are the most common sample messages that may be logged in the /var/log/messages file. Use dmesg -n <level> to control the level at which messages appear on the console. Most systems are set to level 6 by default. To see all messages, set the level higher.
NOTE: The examples used in this procedure refer to the bnx2 driver, but also apply to the bnx2x driver.
Driver Sign on
Broadcom NetXtreme II Gigabit Ethernet Driver
bnx2 v1.6.3c (July 23, 2007)
CNIC Driver Sign on (bnx2 only)
Broadcom NetXtreme II cnic v1.1.19 (Sep 25, 2007)
NIC Detected
eth#: Broadcom NetXtreme II BCM5708 1000Base-T (B0)
PCI-X 64-bit 133MHz found at mem f6000000, IRQ 16, node addr 0010180476ae
cnic: Added CNIC device: eth0
Link Up and Speed Indication
bnx2: eth# NIC Link is Up, 1000 Mbps full duplex
Link Down Indication
bnx2: eth# NIC Link is Down
MSI enabled successfully (bnx2 only)
bnx2: eth0: using MSI
MSI-X enabled successfully (bnx2x only)
bnx2x: eth0: using MSI-X
BNX2I Driver signon
Broadcom NetXtreme II iSCSI Driver bnx2i v2.1.1D (May 12, 2010)
Network port to iSCSI transport name binding
bnx2i: netif=eth2, iscsi=bcm570x-050000
bnx2i: netif=eth1, iscsi=bcm570x-030c00
Driver completes handshake with iSCSI offload-enabled CNIC device
bnx2i [05:00.00]: ISCSI_INIT passed
NOTE: This message is displayed only when the user attempts to make an iSCSI connection.
Driver detects iSCSI offload is not enabled on the CNIC device
bnx2i: iSCSI not supported, dev=eth3
bnx2i: bnx2i: LOM is not enabled to offload iSCSI connections, dev=eth0
bnx2i: dev eth0 does not support iSCSI
Exceeds maximum allowed iSCSI connection offload limit
bnx2i: alloc_ep: unable to allocate iscsi cid
bnx2i: unable to allocate iSCSI context resources
Network route to target node and transport name binding are two different devices
bnx2i: conn bind, ep=0x... ($ROUTE_HBA) does not belong to hba $USER_CHOSEN_HBA
where ROUTE_HBA --> net device on which connection was offloaded based on route information USER_CHOSEN_HBA --> HBA to which target node is bound (using iscsi transport name)
Target cannot be reached on any of the CNIC devices
bnx2i: check route, cannot connect using cnic
Network route is assigned to network interface, which is down
bnx2i: check route, hba not found
Attempting to offload iSCSI connection onto a Jumbo Frame-enabled device
bnx2i: eth# network i/f mtu is set to #mtu
bnx2i: iSCSI HBA can support mtu of 1500
NOTE: Change mtu to 1500 using ifconfig and restart the interface in order to offload iSCSI connections.
SCSI-ML initiated host reset (session recovery)
bnx2i: attempting to reset host, #3
CNIC detects iSCSI protocol violation - Fatal errors
bnx2i: iscsi_error - wrong StatSN rcvd
bnx2i: iscsi_error - hdr digest err
bnx2i: iscsi_error - data digest err
bnx2i: iscsi_error - wrong opcode rcvd
bnx2i: iscsi_error - AHS len > 0 rcvd
bnx2i: iscsi_error - invalid ITT rcvd
bnx2i: iscsi_error - wrong StatSN rcvd
bnx2i: iscsi_error - wrong DataSN rcvd
bnx2i: iscsi_error - pend R2T violation
bnx2i: iscsi_error - ERL0, UO
bnx2i: iscsi_error - ERL0, U1
bnx2i: iscsi_error - ERL0, U2
bnx2i: iscsi_error - ERL0, U3
bnx2i: iscsi_error - ERL0, U4
bnx2i: iscsi_error - ERL0, U5
bnx2i: iscsi_error - ERL0, U
bnx2i: iscsi_error - invalid resi len
bnx2i: iscsi_error - MRDSL violation
bnx2i: iscsi_error - F-bit not set
bnx2i: iscsi_error - invalid TTT
bnx2i: iscsi_error - invalid DataSN
bnx2i: iscsi_error - burst len violation
bnx2i: iscsi_error - buf offset violation
bnx2i: iscsi_error - invalid LUN field
bnx2i: iscsi_error - invalid R2TSN field
bnx2i: iscsi_error - invalid cmd len1
bnx2i: iscsi_error - invalid cmd len2
bnx2i: iscsi_error - pend r2t exceeds MaxOutstandingR2T value
bnx2i: iscsi_error - TTT is rsvd
bnx2i: iscsi_error - MBL violation
bnx2i: iscsi_error - data seg len != 0
bnx2i: iscsi_error - reject pdu len error
bnx2i: iscsi_error - async pdu len error
bnx2i: iscsi_error - nopin pdu len error
bnx2i: iscsi_error - pend r2t in cleanup
bnx2i: iscsi_error - IP fragments rcvd
bnx2i: iscsi_error - IP options error
bnx2i: iscsi_error - urgent flag error
CNIC detects iSCSI protocol violation - non-FATAL, warning
bnx2i: iscsi_warning - invalid TTT
bnx2i: iscsi_warning - invalid DataSN
bnx2i: iscsi_warning - invalid LUN field
NOTE: The driver needs to be configured to consider certain violation to treat as warning and not as a critical error.
Driver puts a session through recovery
conn_err - hostno 3 conn 03fbcd00, iscsi_cid 2 cid a1800
Reject iSCSI PDU received from the target
bnx2i - printing rejected PDU contents
[0]: 1 ffffffa1 0 0 0 0 20 0
[8]: 0 7 0 0 0 0 0 0
[10]: 0 0 40 24 0 0 ffffff80 0
[18]: 0 0 3 ffffff88 0 0 3 4b
[20]: 2a 0 0 2 ffffffc8 14 0 0
[28]: 40 0 0 0 0 0 0 0
Open-iSCSI daemon handing over session to driver
bnx2i: conn update - MBL 0x800 FBL 0x800MRDSL_I 0x800 MRDSL_T 0x2000
With the Linux drivers, you can team adapters together using the bonding kernel module and a channel bonding interface. For more information, see the Channel Bonding information in your operating system documentation.
Detailed statistics and configuration information can be viewed using the ethtool utility. See the ethtool man page for more information.
Run the brcm_iscsiuio daemon before attempting to create iSCSI connections. The driver will not be able to establish connections to the iSCSI target without the daemon's assistance.
brcm_iscsiuio
Install and run the open-iscsi programs, iscsid and iscsiadm, from the Broadcom distributed open-iscsi packages. Refer to Packaging for details. All pre-installed open-iscsi packages need to be removed before the Broadcom iSCSI-supported packages can be installed.
RHEL5
rpm -e iscsi-initiator-utils
rpm -ivh <open-iscsi-package-name>.src.rpm
cd /usr/src/{redhat,OpenLinux,turbo,packages,rpm ..}
rpm -bb SPECS/<open-iscsi-package-name>.spec
or
rpmbuild -bb SPECS/<open-iscsi-package-name>.spec (for RPM version 4.x.x)
Note that the RPM path is different for different Linux distributions.
rpm -ivh RPMS/<arch>/<open-iscsi-package-name>.<arch>.rpm
where <arch> is the machine architecture such as i386.
rpm -ivh RPMS/i386/<open-iscsi-package-name>.i386.rpm
iscsid
By default, the open-iscsi daemon connects to discovered targets using software initiator (transport name = 'tcp'). Users who wish to offload iSCSI connection onto CNIC device should explicitly change transport binding of the iSCSI node. This can be done using iscsiadm cli utility as follows,
iscsiadm --mode node --targetname iqn.2004-06.com.broadcom:tg1 \
--portal 192.168.1.100 --op=update \
--name=iface_file_name
where the iface file includes the following information for RHEL 5.4, RHEL 5.5, and SLES 11 SP1:
iface.net_ifacename = ethX
iface.iscsi_ifacename = <name of the iface file>
iface.hwaddress = XX:XX:XX:XX:XX:XX
iface.ipaddress = XX:XX:XX:XX
iface.transport_name = bnx2i
Ensure that the iface.hwaddress is in lower case format.
If you wish to switch back to use the software initiator, use the following:
iscsiadm --mode node --targetname iqn.2004-06.com.broadcom:tg1 \
--portal 192.168.1.100 --op=update \
--name=iface_file_name
where the iface file includes the following information:
iface.net_ifacename = ethX
iface.iscsi_ifacename = <name of the iface file>
iface.transport_name = tcp
Refer to open-iscsi documentation for a comprehensive list of iscsiadm commands. This is a sample list of commands to discovery targets and to create iscsi connections to a target.
Add static entry
iscsiadm -m node -p <ipaddr[:port]> -T iqn.2007-05.com.broadcom:target1 -o new
iSCSI target discovery using 'SendTargets'
iscsiadm -m discovery --type sendtargets -p <ipaddr[:port]>
Bind iSCSI target to Broadcom NX2 transport (bnx2i)
iscsiadm -m node -p <ipaddr[:port]> -T <iqn.targetname> --op=update --name=iface_file_name
Login to target using 'iscsiadm' command
iscsiadm --mode node --targetname <iqn.targetname> --portal <ipaddr[:port]> --login
List all drives active in the system
fdisk -l
With default driver parameters set, which includes 128 outstanding commands, bnx2i can offload the following number of connections:
BCM5706/BCM5708: 28
BCM5709: 43
BCM5771x: 128
This is not a hard limit, but just a simple on-chip resource allocation math. bnx2i will be able to offload > 28 connections on 1G devices by reducing the shared queue size, which in turn limits the maximum outstanding tasks on a connection. See Setting Values for Optional Properties for information on sq_size and rq_size. The driver logs the following message to syslog when the maximum allowed connection offload limit is reached - "bnx2i: unable to allocate iSCSI context resources".