Back to Contents Page

Broadcom Teaming Services: Broadcom NetXtreme II™ Network Adapter User Guide

Executive Summary

Teaming Mechanisms

Teaming and Other Advanced Networking Properties

General Network Considerations

Application Considerations

Troubleshooting Teaming Problems

Frequently Asked Questions

Appendix A: Event Log Messages

Executive Summary

Glossary

Teaming Concepts

Software Components

Hardware Requirements

Teaming Support by Processor

Configuring Teaming

Supported Features by Team Type

Selecting a Team Type

This section describes the technology and implementation considerations when working with the network teaming services offered by the Broadcom software shipped with servers and storage products. The goal of Broadcom teaming services is to provide fault tolerance and link aggregation across a team of two or more adapters. The information in this document is provided to assist IT professionals during the deployment and troubleshooting of system applications that require network fault tolerance and load balancing.

Glossary

Table 1: Glossary 
Item
Definition
ARP
Address Resolution Protocol
BACS
Broadcom Advanced Control Suite
BASP
Broadcom Advanced Server Program (intermediate driver)
DNS
domain name service
G-ARP
Gratuitous Address Resolution Protocol
Generic Trunking (FEC/GEC)/802.3ad-Draft Static
Switch-dependent load balancing and failover type of team in which the intermediate driver manages outgoing traffic and the switch manages incoming traffic.
HSRP
Hot Standby Router Protocol
ICMP
Internet Control Message Protocol
IGMP
Internet Group Management Protocol
IP
Internet Protocol
IPv6
Version 6 of the IP Protocol
iSCSI
Internet Small Computer Systems Interface
L2
Layer 2. Used to describe network traffic that is not offloaded, and where hardware only performs Layer 2 operations on the traffic. Layer 3 (IP) and Layer 4 (TCP) protocols are processed in software.
L4
Layer 4. Used to describe network traffic that is heavily offloaded to the hardware, where much of the Layer 3 (IP) and Layer 4 (TCP) processing is done in the hardware to improve performance.
LACP
Link Aggregation Control Protocol
Link Aggregation (802.3ad)
Switch-dependent load balancing and failover type of team with LACP in which the intermediate driver manages outgoing traffic and the switch manages incoming traffic.
LOM
LAN on Motherboard
MAC
media access control
NDIS
Network Driver Interface Specification
NLB
Network Load Balancing (Microsoft)
PXE
Preboot Execution Environment
RAID
redundant array of inexpensive disks
Smart Load Balancing™ and Failover
Switch-independent failover type of team in which the primary team member handles all incoming and outgoing traffic while the standby team member is idle until a failover event (for example, loss of link occurs). The intermediate driver (BASP) manages incoming/outgoing traffic.
Smart Load Balancing (SLB)
Switch-independent load balancing and failover type of team, in which the intermediate driver manages outgoing/incoming traffic.
TCP
Transmission Control Protocol
TOE
TCP Offload Engine. This is the hardware that is capable of handling stateful fastpath offloading of TCP and IP processing.
UDP
User Datagram Protocol
WINS
Windows name service
WLBS
Windows Load Balancing Service

Teaming Concepts

Network Addressing

Teaming and Network Addresses

Description of Teaming Types

TOE Teaming

The concept of grouping multiple physical devices to provide fault tolerance and load balancing is not new. It has been around for years. Storage devices use RAID technology to group individual hard drives. Switch ports can be grouped together using technologies such as Cisco Gigabit EtherChannel, IEEE 802.3ad Link Aggregation, Bay Network Multilink Trunking, and Extreme Network Load Sharing. Network interfaces on servers can be grouped together into a team of physical ports called a virtual adapter.

Network Addressing

To understand how teaming works, it is important to understand how node communications work in an Ethernet network. This document is based on the assumption that the reader is familiar with the basics of IP and Ethernet network communications. The following information provides a high-level overview of the concepts of network addressing used in an Ethernet network. Every Ethernet network interface in a host platform, such as a computer system, requires a globally unique Layer 2 address and at least one globally unique Layer 3 address. Layer 2 is the Data Link Layer, and Layer 3 is the Network layer as defined in the OSI model. The Layer 2 address is assigned to the hardware and is often referred to as the MAC address or physical address. This address is pre-programmed at the factory and stored in NVRAM on a network interface card or on the system motherboard for an embedded LAN interface. The Layer 3 addresses are referred to as the protocol or logical address assigned to the software stack. IP and IPX are examples of Layer 3 protocols. In addition, Layer 4 (Transport Layer) uses port numbers for each network upper level protocol such as Telnet or FTP. These port numbers are used to differentiate traffic flows across applications. Layer 4 protocols such as TCP or UDP are most commonly used in today's networks. The combination of the IP address and the TCP port number is called a socket.

Ethernet devices communicate with other Ethernet devices using the MAC address, not the IP address. However, most applications work with a host name that is translated to an IP address by a Naming Service such as WINS and DNS. Therefore, a method of identifying the MAC address assigned to the IP address is required. The Address Resolution Protocol for an IP network provides this mechanism. For IPX, the MAC address is part of the network address and ARP is not required. ARP is implemented using an ARP Request and ARP Reply frame. ARP Requests are typically sent to a broadcast address while the ARP Reply is typically sent as unicast traffic. A unicast address corresponds to a single MAC address or a single IP address. A broadcast address is sent to all devices on a network.

Teaming and Network Addresses

A team of adapters function as a single virtual network interface and does not appear any different to other network devices than a non-teamed adapter. A virtual network adapter advertises a single Layer 2 and one or more Layer 3 addresses. When the teaming driver initializes, it selects one MAC address from one of the physical adapters that make up the team to be the Team MAC address. This address is typically taken from the first adapter that gets initialized by the driver. When the system hosting the team receives an ARP request, it selects one MAC address from among the physical adapters in the team to use as the source MAC address in the ARP Reply. In Windows operating systems, the IPCONFIG /all command shows the IP and MAC address of the virtual adapter and not the individual physical adapters. The protocol IP address is assigned to the virtual network interface and not to the individual physical adapters.

For switch-independent teaming modes, all physical adapters that make up a virtual adapter must use the unique MAC address assigned to them when transmitting data. That is, the frames that are sent by each of the physical adapters in the team must use a unique MAC address to be IEEE compliant. It is important to note that ARP cache entries are not learned from received frames, but only from ARP requests and ARP replies.

Description of Teaming Types

Smart Load Balancing and Failover

Generic Trunking

Link Aggregation (IEEE 802.3ad LACP)

SLB (Auto-Fallback Disable)

There are three methods for classifying the supported teaming types:

Table 2 shows a summary of the teaming types and their classification.

Table 2: Available Teaming Types
Teaming Type
Switch-Dependent
(Switch must support specific type of team)
Link Aggregation Control Protocol Support Required on the Switch
Load Balancing
Failover
Smart Load Balancing and Failover (with two to eight load balance team members)
 
 

SLB (Auto-Fallback Disable)
 
 

Link Aggregation (802.3ad)

Generic Trunking (FEC/GEC)/802.3ad-Draft Static

 

Smart Load Balancing and Failover

The Smart Load Balancing™ and Failover type of team provides both load balancing and failover when configured for load balancing, and only failover when configured for fault tolerance. This type of team works with any Ethernet switch and requires no trunking configuration on the switch. The team advertises multiple MAC addresses and one or more IP addresses (when using secondary IP addresses). The team MAC address is selected from the list of load balance members. When the system receives an ARP request, the software-networking stack will always send an ARP Reply with the team MAC address. To begin the load balancing process, the teaming driver will modify this ARP Reply by changing the source MAC address to match one of the physical adapters.

Smart Load Balancing enables both transmit and receive load balancing based on the Layer 3/Layer 4 IP address and TCP/UDP port number. In other words, the load balancing is not done at a byte or frame level but on a TCP/UDP session basis. This methodology is required to maintain in-order delivery of frames that belong to the same socket conversation. Load balancing is supported on 2 to 8 ports. These ports can include any combination of add-in adapters and LAN on Motherboard (LOM) devices. Transmit load balancing is achieved by creating a hashing table using the source and destination IP addresses and TCP/UDP port numbers.The same combination of source and destination IP addresses and TCP/UDP port numbers will generally yield the same hash index and therefore point to the same port in the team. When a port is selected to carry all the frames of a given socket, the unique MAC address of the physical adapter is included in the frame, and not the team MAC address. This is required to comply with the IEEE 802.3 standard. If two adapters transmit using the same MAC address, then a duplicate MAC address situation would occur that the switch could not handle.

NOTE: IPv6 addressed traffic will not be load balanced by SLB because ARP is not a feature of IPv6.

Receive load balancing is achieved through an intermediate driver by sending gratuitous ARPs on a client-by-client basis using the unicast address of each client as the destination address of the ARP request (also known as a directed ARP). This is considered client load balancing and not traffic load balancing. When the intermediate driver detects a significant load imbalance between the physical adapters in an SLB team, it will generate G-ARPs in an effort to redistribute incoming frames. The intermediate driver (BASP) does not answer ARP requests; only the software protocol stack provides the required ARP Reply. It is important to understand that receive load balancing is a function of the number of clients that are connecting to the system through the team interface.

SLB receive load balancing attempts to load balance incoming traffic for client machines across physical ports in the team. It uses a modified gratuitous ARP to advertise a different MAC address for the team IP Address in the sender physical and protocol address. This G-ARP is unicast with the MAC and IP Address of a client machine in the target physical and protocol address respectively. This causes the target client to update its ARP cache with a new MAC address map to the team IP address. G-ARPs are not broadcast because this would cause all clients to send their traffic to the same port. As a result, the benefits achieved through client load balancing would be eliminated, and could cause out-of-order frame delivery. This receive load balancing scheme works as long as all clients and the teamed system are on the same subnet or broadcast domain.

When the clients and the system are on different subnets, and incoming traffic has to traverse a router, the received traffic destined for the system is not load balanced. The physical adapter that the intermediate driver has selected to carry the IP flow carries all of the traffic. When the router sends a frame to the team IP address, it broadcasts an ARP request (if not in the ARP cache). The server software stack generates an ARP reply with the team MAC address, but the intermediate driver modifies the ARP reply and sends it over a particular physical adapter, establishing the flow for that session.

The reason is that ARP is not a routable protocol. It does not have an IP header and therefore, is not sent to the router or default gateway. ARP is only a local subnet protocol. In addition, since the G-ARP is not a broadcast packet, the router will not process it and will not update its own ARP cache.

The only way that the router would process an ARP that is intended for another network device is if it has Proxy ARP enabled and the host has no default gateway. This is very rare and not recommended for most applications.

Transmit traffic through a router will be load balanced as transmit load balancing is based on the source and destination IP address and TCP/UDP port number. Since routers do not alter the source and destination IP address, the load balancing algorithm works as intended.

Configuring routers for Hot Standby Routing Protocol (HSRP) does not allow for receive load balancing to occur in the adapter team. In general, HSRP allows for two routers to act as one router, advertising a virtual IP and virtual MAC address. One physical router is the active interface while the other is standby. Although HSRP can also load share nodes (using different default gateways on the host nodes) across multiple routers in HSRP groups, it always points to the primary MAC address of the team.

Generic Trunking

Generic Trunking is a switch-assisted teaming mode and requires configuring ports at both ends of the link: server interfaces and switch ports. This is often referred to as Cisco Fast EtherChannel or Gigabit EtherChannel. In addition, generic trunking supports similar implementations by other switch OEMs such as Extreme Networks Load Sharing and Bay Networks or IEEE 802.3ad Link Aggregation static mode. In this mode, the team advertises one MAC Address and one IP Address when the protocol stack responds to ARP Requests. In addition, each physical adapter in the team uses the same team MAC address when transmitting frames. This is possible since the switch at the other end of the link is aware of the teaming mode and will handle the use of a single MAC address by every port in the team. The forwarding table in the switch will reflect the trunk as a single virtual port.

In this teaming mode, the intermediate driver controls load balancing and failover for outgoing traffic only, while incoming traffic is controlled by the switch firmware and hardware. As is the case for Smart Load Balancing, the BASP intermediate driver uses the IP/TCP/UDP source and destination addresses to load balance the transmit traffic from the server. Most switches implement an XOR hashing of the source and destination MAC address.

NOTE: Generic Trunking is not supported on iSCSI offload adapters.

Link Aggregation (IEEE 802.3ad LACP)

Link Aggregation is similar to Generic Trunking except that it uses the Link Aggregation Control Protocol to negotiate the ports that will make up the team. LACP must be enabled at both ends of the link for the team to be operational. If LACP is not available at both ends of the link, 802.3ad provides a manual aggregation that only requires both ends of the link to be in a link up state. Because manual aggregation provides for the activation of a member link without performing the LACP message exchanges, it should not be considered as reliable and robust as an LACP negotiated link. LACP automatically determines which member links can be aggregated and then aggregates them. It provides for the controlled addition and removal of physical links for the link aggregation so that no frames are lost or duplicated. The removal of aggregate link members is provided by the marker protocol that can be optionally enabled for Link Aggregation Control Protocol (LACP) enabled aggregate links.

The Link Aggregation group advertises a single MAC address for all the ports in the trunk. The MAC address of the Aggregator can be the MAC addresses of one of the MACs that make up the group. LACP and marker protocols use a multicast destination address.

The Link Aggregation control function determines which links may be aggregated and then binds the ports to an Aggregator function in the system and monitors conditions to determine if a change in the aggregation group is required. Link aggregation combines the individual capacity of multiple links to form a high performance virtual link. The failure or replacement of a link in an LACP trunk will not cause loss of connectivity. The traffic will simply be failed over to the remaining links in the trunk.

SLB (Auto-Fallback Disable)

This type of team is identical to the Smart Load Balance and Failover type of team, with the following exception—when the standby member is active, if a primary member comes back on line, the team continues using the standby member rather than switching back to the primary member. This type of team is supported only for situations in which the network cable is disconnected and reconnected to the network adapter. It is not supported for situations in which the adapter is removed/installed through Device Manager or Hot-Plug PCI.

If any primary adapter assigned to a team is disabled, the team functions as a Smart Load Balancing and Failover type of team in which auto-fallback occurs.

TOE Teaming

All four basic teaming modes support failover of traffic from a failed adapter to other working adapters. All four teaming modes also support bidirectional load-balancing of TCP/IP traffic. A primary difference between the modes is that the SLB modes use a Broadcom proprietary algorithm to control how both inbound and outbound traffic is balanced across the network interfaces in the team. This has several advantages. First, with Generic Trunking or Link Aggregation modes, the team of network adapters must be connected to a switch that is specifically configured to support that particular mode of teaming. Since there is a dependency between the switch and the host team configuration when Generic Trunking or Link Aggregation is used, it can often lead to configuration difficulties, because both ends must be configured correctly and be synchronized. Second, with Generic Trunking or Link Aggregation modes, the switch decides how inbound traffic to the team is balanced across the adapters, while BASP only controls the balancing of outbound traffic. This is problematic for TOE environments, because in order for TOE to work, state information about a given TCP connection is stored in the hardware on a given offloaded adapter, but it is not stored in the hardware on every member of the team. So teaming and TOE cannot co-exist if the teaming software cannot steer incoming TCP/IP traffic to the adapter that contains and updates the state information for a given TCP connection.

Because Broadcom's SLB modes can control how both outbound and inbound packets are balanced across the adapters, the SLB modes are capable of ensuring that all offloaded TCP traffic for a given TCP connection goes in and out of a particular adapter. This architectural feature allows the SLB modes to also support load-balancing on adapters that have TOE enabled, since BASP is able to steer traffic on a particular TCP connection to the adapter hardware that contains offloaded state information for that TCP connection. BASP can simultaneously use TCP offload in conjunction with the SLB modes of teaming. Other teaming modes (Generic Trunking or Link Aggregation) can still be used on TOE capable devices, but if those other modes are enabled the TOE feature is disabled.

Since the TOE offloaded state is stored in only one member of a team, it might not be intuitive as to how BASP can support failover on TOE teams. When a TOE connection has been offloaded to a given adapter, and if that network interface fails in some way (that is, it loses its network link due to a cable disconnection), then BASP will detect the error and force an upload of the offloaded TCP state for each previously offloaded TCP connection on that adapter to the host. Once all of the previously offloaded state has been uploaded, BASP will rebalance the recently uploaded TCP connections and offload those connections evenly to the remaining members of the team. Basically, if there is a failure on a TOE-enabled adapter, any TCP connections that had been offloaded to that adapter are migrated to the remaining nonfailed members in the team.

For Broadcom NetXtreme II adapters, there are no specific setup requirements in order for TCP Offload Engine (TOE) to work with BASP. Once the individual adapters are configured to enable TOE, they can be added to a team and the offload is transparent to BASP. For information on configuring TOE, see Viewing Resource Reservations.

Limitations of Teaming with Offloading

Software Components

Teaming is implemented via an NDIS intermediate driver in the Windows Operating System environment. This software component works with the miniport driver, the NDIS layer, and the protocol stack to enable the teaming architecture (see Figure 2). The miniport driver controls the host LAN controller directly to enable functions such as sends, receives, and interrupt processing. The intermediate driver fits between the miniport driver and the protocol layer multiplexing several miniport driver instances, and creating a virtual adapter that looks like a single adapter to the NDIS layer. NDIS provides a set of library functions to enable the communications between either miniport drivers or intermediate drivers and the protocol stack. The protocol stack implements IP, IPX and ARP. A protocol address such as an IP address is assigned to each miniport device instance, but when an Intermediate driver is installed, the protocol address is assigned to the virtual team adapter and not to the individual miniport devices that make up the team.

The Broadcom supplied teaming support is provided by three individual software components that work together and are supported as a package. When one component is upgraded, all the other components must be upgraded to the supported versions. Table 3 describes the four software components and their associated files for supported operating systems.

Table 3: Broadcom Teaming Software Component
Software Component
Broadcom Name
Network Adapter/Operating System
System Architecture
Windows File Name
 
Virtual Bus Driver (VBD)
BCM5706, BCM5708, BCM5709
32-bit
bxvbdx.sys
BCM5706, BCM5708, BCM5709
64-bit
bxvbda.sys
BCM57710, BCM57711, BCM57712
32-bit
evbdx.sys
BCM57710, BCM57711, BCM57712
64-bit
evbda.sys
Miniport Driver
Broadcom Base Driver
Windows Server 2008 (NDIS 6.0)
32-bit
bxnd60x.sys
Windows Server 2008 (NDIS 6.0)
64-bit
bxnd60a.sys
Windows Server 2008 R2 (NDIS 6.0)
64-bit
bxnd60a.sys
Intermediate Driver
Broadcom Advanced Server Program (BASP)
Windows Server 2008
32-bit, 64-bit
basp.sys
Windows Server 2008 R2
64-bit
basp.sys
Configuration User Interface
Broadcom Advanced Control Suite (BACS)
bacs.exe

Hardware Requirements

Repeater Hub

Switching Hub

Router

The various teaming modes described in this document place certain restrictions on the networking equipment used to connect clients to teamed systems. Each type of network interconnect technology has an effect on teaming as described in the following sections.

Repeater Hub

A Repeater Hub allows a network administrator to extend an Ethernet network beyond the limits of an individual segment. The repeater regenerates the input signal received on one port onto all other connected ports, forming a single collision domain. This means that when a station attached to a repeater sends an Ethernet frame to another station, every station within the same collision domain will also receive that message. If two stations begin transmitting at the same time, a collision occurs, and each transmitting station must retransmit its data after waiting a random amount of time.

The use of a repeater requires that each station participating within the collision domain operate in half-duplex mode. Although half-duplex mode is supported for Gigabit Ethernet adapters in the IEEE 802.3 specification, half-duplex mode is not supported by the majority of Gigabit Ethernet adapter manufacturers. Therefore, half-duplex mode is not considered here.

Teaming across hubs is supported for troubleshooting purposes (such as connecting a network analyzer) for SLB teams only.

Switching Hub

Unlike a repeater hub, a switching hub (or more simply a switch) allows an Ethernet network to be broken into multiple collision domains. The switch is responsible for forwarding Ethernet packets between hosts based solely on Ethernet MAC addresses. A physical network adapter that is attached to a switch may operate in half-duplex or full-duplex mode.

To support Generic Trunking and 802.3ad Link Aggregation, a switch must specifically support such functionality. If the switch does not support these protocols, it may still be used for Smart Load Balancing.

NOTE: All modes of network teaming are supported across switches when operating as a stackable switch.

Router

A router is designed to route network traffic based on Layer 3 or higher protocols, although it often also works as a Layer 2 device with switching capabilities. The teaming of ports connected directly to a router is not supported.

Teaming Support by Processor

All team types are supported by the IA-32, AMD-64, and EM64T processors.

Configuring Teaming

The Broadcom Advanced Control Suite utility is used to configure teaming in the supported operating system environments.

The Broadcom Advanced Control Suite (BACS) utility is designed to run on 32-bit and 64-bit Windows family of operating systems. BACS is used to configure load balancing and fault tolerance teaming, and VLANs. In addition, it displays the MAC address, driver version, and status information about each network adapter. BACS also includes a number of diagnostics tools such as hardware diagnostics, cable testing, and a network topology test.

Supported Features by Team Type

Table 4 provides a feature comparison across the team types. Use this table to determine the best type of team for your application. The teaming software supports up to eight ports in a single team and up to four teams in a single system. The four teams can be any combination of the supported teaming types, but each team must be on a separate network or subnet.

Table 4: Comparison of Team Types 
Type of Team
Fault Tolerance
Load Balancing
Switch-Dependent Static Trunking
Switch-Independent
Dynamic Link Aggregation
(IEEE 802.3ad)
Function
SLB with Standbya
SLB
Generic Trunking
Link Aggregation
Number of ports per team (same broadcast domain)
2–8
2–8
2–8
2–8
Number of teams
8
8
8
8
Adapter fault tolerance
Yes
Yes
Yes
Yes
Switch link fault tolerance (same broadcast domain)
Yes
Yes
Switch-dependent
Switch-dependent
TX load balancing
No
Yes
Yes
Yes
RX load balancing
No
Yes
Yes (performed by the switch)
Yes (performed by the switch)
Requires compatible switch
No
No
Yes
Yes
Heartbeats to check connectivity
No
No
No
No
Mixed media (adapters with different media)
Yes
Yes
Yes (switch-dependent)
Yes
Mixed speeds (adapters that do not support a common speed(s), but can operate at different speeds)
Yes
Yes
No
No
Mixed speeds (adapters that support a common speed(s), but can operate at different speeds)
Yes
Yes
No (must be the same speed)
Yes
Load balances TCP/IP
No
Yes
Yes
Yes
Mixed vendor teaming
Yesb
Yesb
Yesb
Yesb
Load balances non-IP
No
Yes (IPX outbound traffic only)
Yes
Yes
Same MAC address for all team members
No
No
Yes
Yes
Same IP address for all team members
Yes
Yes
Yes
Yes
Load balancing by IP address
No
Yes
Yes
Yes
Load balancing by MAC address
No
Yes (used for no-IP/IPX)
Yes
Yes
Allows TOE functionality to co-exist when all team members support TOEc
Yes
Yes
No
No

a SLB with one primary and one standby member.
b Requires at least one Broadcom adapter in the team.
c TOE functionality can only be achieved with SLB teams that consist of all Broadcom TOE-enabled adapters.

Selecting a Team Type

The following flow chart provides the decision flow when planning for Layer 2 teaming. For TOE teaming, only Smart Load Balancing™ and Failover type team is supported. The primary rationale for teaming is the need for additional network bandwidth and fault tolerance. Teaming offers link aggregation and fault tolerance to meet both of these requirements. Preference teaming should be selected in the following order: Link Aggregation as the first choice, Generic Trunking as the second choice, and SLB teaming as the third choice when using unmanaged switches or switches that do not support the first two options. if switch fault tolerance is a requirement, then SLB is the only choice (see Figure 1).

Figure 1: Process for Selecting a Team Type

Teaming Mechanisms

Architecture

Types of Teams

Attributes of the Features Associated with Each Type of Team

Speeds Supported for Each Type of Team

Architecture

The Broadcom Advanced Server Program is implemented as an NDIS intermediate driver (see Figure 2). It operates below protocol stacks such as TCP/IP and IPX and appears as a virtual adapter. This virtual adapter inherits the MAC Address of the first port initialized in the team. A Layer 3 address must also be configured for the virtual adapter. The primary function of BASP is to balance inbound (for SLB) and outbound traffic (for all teaming modes) among the physical adapters installed on the system selected for teaming. The inbound and outbound algorithms are independent and orthogonal to each other. The outbound traffic for a particular session can be assigned to a given port while its corresponding inbound traffic can be assigned to a different port.

Figure 2: Intermediate Driver

Outbound Traffic Flow

The Broadcom Intermediate Driver manages the outbound traffic flow for all teaming modes. For outbound traffic, every packet is first classified into a flow, and then distributed to the selected physical adapter for transmission. The flow classification involves an efficient hash computation over known protocol fields. The resulting hash value is used to index into an Outbound Flow Hash Table.The selected Outbound Flow Hash Entry contains the index of the selected physical adapter responsible for transmitting this flow. The source MAC address of the packets will then be modified to the MAC address of the selected physical adapter. The modified packet is then passed to the selected physical adapter for transmission.

The outbound TCP and UDP packets are classified using Layer 3 and Layer 4 header information. This scheme improves the load distributions for popular Internet protocol services using well-known ports such as HTTP and FTP. Therefore, BASP performs load balancing on a TCP session basis and not on a packet-by-packet basis.

In the Outbound Flow Hash Entries, statistics counters are also updated after classification. The load-balancing engine uses these counters to periodically distribute the flows across teamed ports. The outbound code path has been designed to achieve best possible concurrency where multiple concurrent accesses to the Outbound Flow Hash Table are allowed.

For protocols other than TCP/IP, the first physical adapter will always be selected for outbound packets. The exception is Address Resolution Protocol (ARP), which is handled differently to achieve inbound load balancing.

Inbound Traffic Flow (SLB Only)

The Broadcom intermediate driver manages the inbound traffic flow for the SLB teaming mode. Unlike outbound load balancing, inbound load balancing can only be applied to IP addresses that are located in the same subnet as the load-balancing server. Inbound load balancing exploits a unique characteristic of Address Resolution Protocol (RFC0826), in which each IP host uses its own ARP cache to encapsulate the IP Datagram into an Ethernet frame. BASP carefully manipulates the ARP response to direct each IP host to send the inbound IP packet to the desired physical adapter. Therefore, inbound load balancing is a plan-ahead scheme based on statistical history of the inbound flows. New connections from a client to the server will always occur over the primary physical adapter (because the ARP Reply generated by the operating system protocol stack will always associate the logical IP address with the MAC address of the primary physical adapter).

Like the outbound case, there is an Inbound Flow Head Hash Table. Each entry inside this table has a singly linked list and each link (Inbound Flow Entries) represents an IP host located in the same subnet.

When an inbound IP Datagram arrives, the appropriate Inbound Flow Head Entry is located by hashing the source IP address of the IP Datagram. Two statistics counters stored in the selected entry are also updated. These counters are used in the same fashion as the outbound counters by the load-balancing engine periodically to reassign the flows to the physical adapter.

On the inbound code path, the Inbound Flow Head Hash Table is also designed to allow concurrent access. The link lists of Inbound Flow Entries are only referenced in the event of processing ARP packets and the periodic load balancing. There is no per packet reference to the Inbound Flow Entries. Even though the link lists are not bounded; the overhead in processing each non-ARP packet is always a constant. The processing of ARP packets, both inbound and outbound, however, depends on the number of links inside the corresponding link list.

On the inbound processing path, filtering is also employed to prevent broadcast packets from looping back through the system from other physical adapters.

Protocol Support

ARP and IP/TCP/UDP flows are load balanced. If the packet is an IP protocol only, such as ICMP or IGMP, then all data flowing to a particular IP address will go out through the same physical adapter. If the packet uses TCP or UDP for the L4 protocol, then the port number is added to the hashing algorithm, so two separate L4 flows can go out through two separate physical adapters to the same IP address.

For example, assume the client has an IP address of 10.0.0.1. All IGMP and ICMP traffic will go out the same physical adapter because only the IP address is used for the hash. The flow would look something like this:

IGMP ------> PhysAdapter1 ------> 10.0.0.1

ICMP ------> PhysAdapter1 ------> 10.0.0.1

If the server also sends an TCP and UDP flow to the same 10.0.0.1 address, they can be on the same physical adapter as IGMP and ICMP, or on completely different physical adapters from ICMP and IGMP. The stream may look like this:

IGMP ------> PhysAdapter1 ------> 10.0.0.1

ICMP ------> PhysAdapter1 ------> 10.0.0.1

TCP------> PhysAdapter1 ------> 10.0.0.1

UDP------> PhysAdatper1 ------> 10.0.0.1

Or the streams may look like this:

IGMP ------> PhysAdapter1 ------> 10.0.0.1

ICMP ------> PhysAdapter1 ------> 10.0.0.1

TCP------> PhysAdapter2 ------> 10.0.0.1

UDP------> PhysAdatper3 ------> 10.0.0.1

The actual assignment between adapters may change over time, but any protocol that is not TCP/UDP based goes over the same physical adapter because only the IP address is used in the hash.

Performance

Modern network interface cards provide many hardware features that reduce CPU utilization by offloading certain CPU intensive operations (see Teaming and Other Advanced Networking Properties). In contrast, the BASP intermediate driver is a purely software function that must examine every packet received from the protocol stacks and react to its contents before sending it out through a particular physical interface. Though the BASP driver can process each outgoing packet in near constant time, some applications that may already be CPU bound may suffer if operated over a teamed interface. Such an application may be better suited to take advantage of the failover capabilities of the intermediate driver rather than the load balancing features, or it may operate more efficiently over a single physical adapter that provides a particular hardware feature such as Large Send Offload.

Types of Teams

Switch-Independent

The Broadcom Smart Load Balancing type of team allows two to eight physical adapters to operate as a single virtual adapter. The greatest benefit of the SLB type of team is that it operates on any IEEE compliant switch and requires no special configuration.

Smart Load Balancing and Failover

SLB provides for switch-independent, bidirectional, fault-tolerant teaming and load balancing. Switch independence implies that there is no specific support for this function required in the switch, allowing SLB to be compatible with all switches. Under SLB, all adapters in the team have separate MAC addresses. The load-balancing algorithm operates on Layer 3 addresses of the source and destination nodes, which enables SLB to load balance both incoming and outgoing traffic.

The BASP intermediate driver continually monitors the physical ports in a team for link loss. In the event of link loss on any port, traffic is automatically diverted to other ports in the team. The SLB teaming mode supports switch fault tolerance by allowing teaming across different switches- provided the switches are on the same physical network or broadcast domain.

Network Communications

The following are the key attributes of SLB:

Applications

The SLB algorithm is most appropriate in home and small business environments where cost is a concern or with commodity switching equipment. SLB teaming works with unmanaged Layer 2 switches and is a cost-effective way of getting redundancy and link aggregation at the server. Smart Load Balancing also supports teaming physical adapters with differing link capabilities. In addition, SLB is recommended when switch fault tolerance with teaming is required.

Configuration Recommendations

SLB supports connecting the teamed ports to hubs and switches if they are on the same broadcast domain. It does not support connecting to a router or Layer 3 switches because the ports must be on the same subnet.

Switch-Dependent
Generic Static Trunking

This mode supports a variety of environments where the adapter link partners are statically configured to support a proprietary trunking mechanism. This mode could be used to support Lucent's Open Trunk, Cisco's Fast EtherChannel (FEC), and Cisco's Gigabit EtherChannel (GEC). In the static mode, as in generic link aggregation, the switch administrator needs to assign the ports to the team, and this assignment cannot be altered by the BASP, as there is no exchange of the Link Aggregation Control Protocol (LACP) frame.

With this mode, all adapters in the team are configured to receive packets for the same MAC address. Trunking operates on Layer 2 addresses and supports load balancing and failover for both inbound and outbound traffic. The BASP driver determines the load-balancing scheme for outbound packets, using Layer 4 protocols previously discussed, whereas the team link partner determines the load-balancing scheme for inbound packets.

The attached switch must support the appropriate trunking scheme for this mode of operation. Both the BASP and the switch continually monitor their ports for link loss. In the event of link loss on any port, traffic is automatically diverted to other ports in the team.

Network Communications

The following are the key attributes of Generic Static Trunking:

Applications

Generic trunking works with switches that support Cisco Fast EtherChannel, Cisco Gigabit EtherChannel, Extreme Networks Load Sharing and Bay Networks or IEEE 802.3ad Link Aggregation static mode. Since load balancing is implemented on Layer 2 addresses, all higher protocols such as IP, IPX, and NetBEUI are supported. Therefore, this is the recommended teaming mode when the switch supports generic trunking modes over SLB.

Configuration Recommendations

Static trunking supports connecting the teamed ports to switches if they are on the same broadcast domain and support generic trunking. It does not support connecting to a router or Layer 3 switches since the ports must be on the same subnet.

Dynamic Trunking (IEEE 802.3ad Link Aggregation)

This mode supports link aggregation through static and dynamic configuration via the Link Aggregation Control Protocol (LACP). With this mode, all adapters in the team are configured to receive packets for the same MAC address. The MAC address of the first adapter in the team is used and cannot be substituted for a different MAC address. The BASP driver determines the load-balancing scheme for outbound packets, using Layer 4 protocols previously discussed, whereas the team's link partner determines the load-balancing scheme for inbound packets. Because the load balancing is implemented on Layer 2, all higher protocols such as IP, IPX, and NetBEUI are supported. The attached switch must support the 802.3ad Link Aggregation standard for this mode of operation. The switch manages the inbound traffic to the adapter while the BASP manages the outbound traffic. Both the BASP and the switch continually monitor their ports for link loss. In the event of link loss on any port, traffic is automatically diverted to other ports in the team.

Network Communications

The following are the key attributes of Dynamic Trunking:

Applications

Dynamic trunking works with switches that support IEEE 802.3ad Link Aggregation dynamic mode using LACP. Inbound load balancing is switch dependent. In general, the switch traffic is load balanced based on L2 addresses. In this case, all network protocols such as IP, IPX, and NetBEUI are load balanced. Therefore, this is the recommended teaming mode when the switch supports LACP, except when switch fault tolerance is required. SLB is the only teaming mode that supports switch fault tolerance.

Configuration Recommendations

Dynamic trunking supports connecting the teamed ports to switches as long as they are on the same broadcast domain and supports IEEE 802.3ad LACP trunking. It does not support connecting to a router or Layer 3 switches since the ports must be on the same subnet.

LiveLink™

LiveLink™ is a feature of BASP that is available for the Smart Load Balancing (SLB) and SLB (Auto-Fallback Disable) types of teaming. The purpose of LiveLink is to detect link loss beyond the switch and to route traffic only through team members that have a live link. This function is accomplished though the teaming software. The teaming software periodically probes (issues a link packet from each team member) one or more specified target network device(s). The probe target(s) responds when it receives the link packet. If a team member does not detect the response within a specified amount of time, this indicates that the link has been lost, and the teaming software discontinues passing traffic through that team member. Later, if that team member begins to detect a response from a probe target, this indicates that the link has been restored, and the teaming software automatically resumes passing traffic through that team member. LiveLink works only with TCP/IP.

LiveLink™ functionality is supported in both 32-bit and 64-bit Windows operating systems. For similar functionality in Linux operating systems, see the Channel Bonding information in your Red Hat documentation.

Attributes of the Features Associated with Each Type of Team

The attributes of the features associated with each type of team are summarized in Table 5.

Table 5: Attributes 
Feature
Attribute
Smart Load Balancing™
User interface
Broadcom Advanced Control Suite (BACS)
Number of teams
Maximum 8
Number of adapters per team
Maximum 8
Hot replace
Yes
Hot add
Yes
Hot remove
Yes
Link speed support
Different speeds
Frame protocol
IP
Incoming packet management
BASP
Outgoing packet management
BASP
LiveLink support
Yes
Failover event
Loss of link
Failover time
<500 ms
Fallback time
1.5 sb (approximate)
MAC address
Different
Multivendor teaming
Yes
Generic Trunking
User interface
Broadcom Advanced Control Suite (BACS)
Number of teams
Maximum 8
Number of adapters per team
Maximum 8
Hot replace
Yes
Hot add
Yes
Hot remove
Yes
Link speed support
Different speedsa
Frame protocol
All
Incoming packet management
Switch
Outgoing packet management
BASP
Failover event
Loss of link only
Failover time
<500 ms
Fallback time
1.5 sb (approximate)
MAC address
Same for all adapters
Multivendor teaming
Yes
Dynamic Trunking
User interface
Broadcom Advanced Control Suite (BACS)
Number of teams
Maximum 8
Number of adapters per team
Maximum 8
Hot replace
Yes
Hot add
Yes
Hot remove
Yes
Link speed support
Different speeds
Frame protocol
All
Incoming packet management
Switch
Outgoing packet management
BASP
Failover event
Loss of link only
Failover time
<500 ms
Fallback time
1.5 sb (approximate)
MAC address
Same for all adapters
Multivendor teaming
Yes

a Some switches require matching link speeds to correctly negotiate between trunk connections.
b Make sure that Port Fast or Edge Port is enabled.

Speeds Supported for Each Type of Team

The various link speeds that are supported for each type of team are listed in Table 6. Mixed speed refers to the capability of teaming adapters that are running at different link speeds.

Table 6: Link Speeds in Teaming
Type of Team
Link Speed
Traffic Direction
Speed Support
SLB
10/100/1000/10000
Incoming/outgoing
Mixed speed
FEC
100
Incoming/outgoing
Same speed
GEC
1000
Incoming/outgoing
Same speed
IEEE 802.3ad
10/100/1000/10000
Incoming/outgoing
Mixed speed

Teaming and Other Advanced Networking Properties

Checksum Offload

IEEE 802.1p QoS Tagging

Large Send Offload

TCP Offload Engine (TOE)

Jumbo Frames

IEEE 802.1Q VLANs

Wake On LAN

Preboot Execution Environment

Before creating a team, adding or removing team members, or changing advanced settings of a team member, make sure each team member has been configured similarly. Settings to check include VLANs and QoS Packet Tagging, Jumbo Frames, and the various offloads. Advanced adapter properties and teaming support are listed in Table 7.

Table 7: Advanced Adapter Properties and Teaming Support
Adapter Properties
Supported by Teaming Virtual Adapter
Checksum Offload
Yes
IEEE 802.1p QoS Tagging
No
Large Send Offload
Yesa
TCP Offload Engine (TOE)
Yesb, c
Jumbo Frames
Yesb
IEEE 802.1Q VLANs
Yesc
Wake on LAN
Nod
Preboot Execution environment (PXE)
Yese

a All adapters on the team must support this feature. Some adapters may not support this feature if ASF/IPMI is also enabled.
b Must be supported by all adapters in the team.
c Only for Broadcom adapters.
d See Wake On LAN.
e As a PXE sever only, not as a client.

A team does not necessarily inherit adapter properties; rather various properties depend on the specific capability. For instance, an example would be flow control, which is a physical adapter property and has nothing to do with BASP, and will be enabled on a particular adapter if the miniport driver for that adapter has flow control enabled.

NOTE: All adapters on the team must support the property listed in Table 7 in order for the team to support the property.

Checksum Offload

Checksum Offload is a property of the Broadcom network adapters that allows the TCP/IP/UDP checksums for send and receive traffic to be calculated by the adapter hardware rather than by the host CPU. In high-traffic situations, this can allow a system to handle more connections more efficiently than if the host CPU were forced to calculate the checksums. This property is inherently a hardware property and would not benefit from a software-only implementation. An adapter that supports Checksum Offload advertises this capability to the operating system so that the checksum does not need to be calculated in the protocol stack. Checksum Offload is only supported for IPv4 at this time.

IEEE 802.1p QoS Tagging

The IEEE 802.1p standard includes a 3-bit field (supporting a maximum of 8 priority levels), which allows for traffic prioritization. The BASP intermediate driver does not support IEEE 802.1p QoS tagging.

Large Send Offload

Large Send Offload (LSO) is a feature provided by Broadcom network adapters that prevents an upper level protocol such as TCP from breaking a large data packet into a series of smaller packets with headers appended to them. The protocol stack need only generate a single header for a data packet as large as 64 KB, and the adapter hardware breaks the data buffer into appropriately-sized Ethernet frames with the correctly sequenced header (based on the single header originally provided).

TCP Offload Engine (TOE)

The TCP/IP protocol suite is used to provide transport services for a wide range of applications for the Internet, LAN, and for file transfer. Without the TCP Offload Engine, the TCP/IP protocol suite runs on the host CPU, consuming a very high percentage of its resources and leaving little resources for the applications. With the use of the Broadcom NetXtreme II adapter, the TCP/IP processing can be moved to hardware, freeing the CPU for more important tasks such as application processing.

The Broadcom NetXtreme II adapter's TOE functionality allows simultaneous operation of up to 1024 fully offloaded TCP connections for 1-Gbps network adapters and 1880 fully offloaded TCP connections for 10-Gbps network adapters. The TOE support on the adapter significantly reduces the host CPU utilization while preserving the implementation of the operating system stack.

Jumbo Frames

The use of jumbo frames was originally proposed by Alteon Networks, Inc. in 1998 and increased the maximum size of an Ethernet frame to a maximum size of 9000 bytes. Though never formally adopted by the IEEE 802.3 Working Group, support for jumbo frames has been implemented in Broadcom NetXtreme II adapters. The BASP intermediate driver supports jumbo frames, provided that all of the physical adapters in the team also support jumbo frames and the same size is set on all adapters in the team.

IEEE 802.1Q VLANs

In 1998, the IEEE approved the 802.3ac standard, which defines frame format extensions to support Virtual Bridged Local Area Network tagging on Ethernet networks as specified in the IEEE 802.1Q specification. The VLAN protocol permits insertion of a tag into an Ethernet frame to identify the VLAN to which a frame belongs. If present, the 4-byte VLAN tag is inserted into the Ethernet frame between the source MAC address and the length/type field. The first 2-bytes of the VLAN tag consist of the IEEE 802.1Q tag type, whereas the second 2 bytes include a user priority field and the VLAN identifier (VID). Virtual LANs (VLANs) allow the user to split the physical LAN into logical subparts. Each defined VLAN behaves as its own separate network, with its traffic and broadcasts isolated from the others, thus increasing bandwidth efficiency within each logical group. VLANs also enable the administrator to enforce appropriate security and quality of service (QoS) policies. The BASP supports the creation of 64 VLANs per team or adapter: 63 tagged and 1 untagged. The operating system and system resources, however, limit the actual number of VLANs. VLAN support is provided according to IEEE 802.1Q and is supported in a teaming environment as well as on a single adapter. Note that VLANs are supported only with homogeneous teaming and not in a multivendor teaming environment. The BASP intermediate driver supports VLAN tagging. One or more VLANs may be bound to a single instance of the intermediate driver.

Wake On LAN

Wake on LAN (WOL) is a feature that allows a system to be awakened from a sleep state by the arrival of a specific packet over the Ethernet interface. Because a Virtual Adapter is implemented as a software only device, it lacks the hardware features to implement Wake on LAN and cannot be enabled to wake the system from a sleeping state via the Virtual Adapter. The physical adapters, however, support this property, even when the adapter is part of a team.

Preboot Execution Environment

The Preboot Execution Environment (PXE) allows a system to boot from an operating system image over the network. By definition, PXE is invoked before an operating system is loaded, so there is no opportunity for the BASP intermediate driver to load and enable a team. As a result, teaming is not supported as a PXE client, though a physical adapter that participates in a team when the operating system is loaded may be used as a PXE client. Whereas a teamed adapter cannot be used as a PXE client, it can be used for a PXE server, which provides operating system images to PXE clients using a combination of Dynamic Host Control Protocol (DHCP) and the Trivial File Transfer Protocol (TFTP). Both of these protocols operate over IP and are supported by all teaming modes.

General Network Considerations

Teaming with Microsoft Virtual Server 2005

Teaming Across Switches

Spanning Tree Algorithm

Layer 3 Routing/Switching

Teaming with Hubs (for troubleshooting purposes only)

Teaming with Microsoft NLB

Teaming with Microsoft Virtual Server 2005

The only supported BASP team configuration when using Microsoft Virtual Server 2005 is with a Smart Load Balancing (TM) team-type consisting of a single primary Broadcom adapter and a standby Broadcom adapter. Make sure to unbind or deselect "Virtual Machine Network Services" from each team member prior to creating a team and prior to creating Virtual networks with Microsoft Virtual Server. Additionally, a virtual network should be created in this software and subsequently bound to the virtual adapter created by a team. Directly binding a Guest operating system to a team virtual adapter may not render the desired results.

NOTE: As of this writing, Windows Server 2008 is not a supported operating system for Microsoft Virtual Server 2005; thus, teaming may not function as expected with this combination.

Teaming Across Switches

SLB teaming can be configured across switches. The switches, however, must be connected together. Generic Trunking and Link Aggregation do not work across switches because each of these implementations requires that all physical adapters in a team share the same Ethernet MAC address. It is important to note that SLB can only detect the loss of link between the ports in the team and their immediate link partner. SLB has no way of reacting to other hardware failures in the switches and cannot detect loss of link on other ports.

Switch-Link Fault Tolerance

The diagrams below describe the operation of an SLB team in a switch fault tolerant configuration. We show the mapping of the ping request and ping replies in an SLB team with two active members. All servers (Blue, Gray and Red) have a continuous ping to each other. Figure 3 is a setup without the interconnect cable in place between the two switches. Figure 4 has the interconnect cable in place, and Figure 5 is an example of a failover event with the Interconnect cable in place. These scenarios describe the behavior of teaming across the two switches and the importance of the interconnect link.

The diagrams show the secondary team member sending the ICMP echo requests (yellow arrows) while the primary team member receives the respective ICMP echo replies (blue arrows). This illustrates a key characteristic of the teaming software. The load balancing algorithms do not synchronize how frames are load balanced when sent or received. In other words, frames for a given conversation can go out and be received on different interfaces in the team. This is true for all types of teaming supported by Broadcom. Therefore, an interconnect link must be provided between the switches that connect to ports in the same team.

In the configuration without the interconnect, an ICMP Request from Blue to Gray goes out port 82:83 destined for Gray port 5E:CA, but the Top Switch has no way to send it there because it cannot go along the 5E:C9 port on Gray. A similar scenario occurs when Gray attempts to ping Blue. An ICMP Request goes out on 5E:C9 destined for Blue 82:82, but cannot get there. Top Switch does not have an entry for 82:82 in its CAM table because there is no interconnect between the two switches. Pings, however, flow between Red and Blue and between Red and Gray.

Furthermore, a failover event would cause additional loss of connectivity. Consider a cable disconnect on the Top Switch port 4. In this case, Gray would send the ICMP Request to Red 49:C9, but because the Bottom switch has no entry for 49:C9 in its CAM Table, the frame is flooded to all its ports but cannot find a way to get to 49:C9.

Figure 3: Teaming Across Switches Without an Interswitch Link

The addition of a link between the switches allows traffic from/to Blue and Gray to reach each other without any problems. Note the additional entries in the CAM table for both switches. The link interconnect is critical for the proper operation of the team. As a result, it is highly advisable to have a link aggregation trunk to interconnect the two switches to ensure high availability for the connection.

Figure 4: Teaming Across Switches With Interconnect

Figure 5 represents a failover event in which the cable is unplugged on the Top Switch port 4. This is a successful failover with all stations pinging each other without loss of connectivity.

Figure 5: Failover Event

Spanning Tree Algorithm

Topology Change Notice (TCN)

Port Fast/Edge Port

In Ethernet networks, only one active path may exist between any two bridges or switches. Multiple active paths between switches can cause loops in the network. When loops occur, some switches recognize stations on both sides of the switch. This situation causes the forwarding algorithm to malfunction allowing duplicate frames to be forwarded. Spanning tree algorithms provide path redundancy by defining a tree that spans all of the switches in an extended network and then forces certain redundant data paths into a standby (blocked) state. At regular intervals, the switches in the network send and receive spanning tree packets that they use to identify the path. If one network segment becomes unreachable, or if spanning tree costs change, the spanning tree algorithm reconfigures the spanning tree topology and re-establishes the link by activating the standby path. Spanning tree operation is transparent to end stations, which do not detect whether they are connected to a single LAN segment or a switched LAN of multiple segments.

Spanning Tree Protocol (STP) is a Layer 2 protocol designed to run on bridges and switches. The specification for STP is defined in IEEE 802.1d. The main purpose of STP is to ensure that you do not run into a loop situation when you have redundant paths in your network. STP detects/disables network loops and provides backup links between switches or bridges. It allows the device to interact with other STP compliant devices in your network to ensure that only one path exists between any two stations on the network.

After a stable network topology has been established, all bridges listen for hello BPDUs (Bridge Protocol Data Units) transmitted from the root bridge. If a bridge does not get a hello BPDU after a predefined interval (Max Age), the bridge assumes that the link to the root bridge is down. This bridge then initiates negotiations with other bridges to reconfigure the network to re-establish a valid network topology. The process to create a new topology can take up to 50 seconds. During this time, end-to-end communications are interrupted.

The use of Spanning Tree is not recommended for ports that are connected to end stations, because by definition, an end station does not create a loop within an Ethernet segment. Additionally, when a teamed adapter is connected to a port with Spanning Tree enabled, users may experience unexpected connectivity problems. For example, consider a teamed adapter that has a lost link on one of its physical adapters. If the physical adapter were to be reconnected (also known as fallback), the intermediate driver would detect that the link has been reestablished and would begin to pass traffic through the port. Traffic would be lost if the port was temporarily blocked by the Spanning Tree Protocol.

Topology Change Notice (TCN)

A bridge/switch creates a forwarding table of MAC addresses and port numbers by learning the source MAC address that received on a particular port. The table is used to forward frames to a specific port rather than flooding the frame to all ports. The typical maximum aging time of entries in the table is 5 minutes. Only when a host has been silent for 5 minutes would its entry be removed from the table. It is sometimes beneficial to reduce the aging time. One example is when a forwarding link goes to blocking and a different link goes from blocking to forwarding. This change could take up to 50 seconds. At the end of the STP re-calculation a new path would be available for communications between end stations. However, because the forwarding table would still have entries based on the old topology, communications may not be reestablished until after 5 minutes when the affected ports entries are removed from the table. Traffic would then be flooded to all ports and re-learned. In this case it is beneficial to reduce the aging time. This is the purpose of a topology change notice (TCN) BPDU. The TCN is sent from the affected bridge/switch to the root bridge/switch. As soon as a bridge/switch detects a topology change (a link going down or a port going to forwarding) it sends a TCN to the root bridge via its root port. The root bridge then advertises a BPDU with a Topology Change to the entire network.This causes every bridge to reduce the MAC table aging time to 15 seconds for a specified amount of time. This allows the switch to re-learn the MAC addresses as soon as STP re-converges.

Topology Change Notice BPDUs are sent when a port that was forwarding changes to blocking or transitions to forwarding. A TCN BPDU does not initiate an STP recalculation. It only affects the aging time of the forwarding table entries in the switch.It will not change the topology of the network or create loops. End nodes such as servers or clients trigger a topology change when they power off and then power back on.

Port Fast/Edge Port

To reduce the effect of TCNs on the network (for example, increasing flooding on switch ports), end nodes that are powered on/off often should use the Port Fast or Edge Port setting on the switch port they are attached to. Port Fast or Edge Port is a command that is applied to specific ports and has the following effects:

Layer 3 Routing/Switching

The switch that the teamed ports are connected to must not be a Layer 3 switch or router. The ports in the team must be in the same network.

Teaming with Hubs (for troubleshooting purposes only)

Hub Usage in Teaming Network Configurations

SLB Teams

SLB Team Connected to a Single Hub

Generic and Dynamic Trunking (FEC/GEC/IEEE 802.3ad)

SLB teaming can be used with 10/100 hubs, but it is only recommended for troubleshooting purposes, such as connecting a network analyzer in the event that switch port mirroring is not an option.

Hub Usage in Teaming Network Configurations

Although the use of hubs in network topologies is functional in some situations, it is important to consider the throughput ramifications when doing so. Network hubs have a maximum of 100 Mbps half-duplex link speed, which severely degrades performance in either a Gigabit or 100 Mbps switched-network configuration. Hub bandwidth is shared among all connected devices; as a result, when more devices are connected to the hub, the bandwidth available to any single device connected to the hub is reduced in direct proportion to the number of devices connected to the hub.

It is not recommended to connect team members to hubs; only switches should be used to connect to teamed ports. An SLB team, however, can be connected directly to a hub for troubleshooting purposes. Other team types can result in a loss of connectivity if specific failures occur and should not be used with hubs.

SLB Teams

SLB teams are the only teaming type not dependant on switch configuration. The server intermediate driver handles the load balancing and fault tolerance mechanisms with no assistance from the switch. These elements of SLB make it the only team type that maintains failover and fallback characteristics when team ports are connected directly to a hub.

SLB Team Connected to a Single Hub

SLB teams configured as shown in Figure 6 maintain their fault tolerance properties. Either server connection could potentially fail, and network functionality is maintained. Clients could be connected directly to the hub, and fault tolerance would still be maintained; server performance, however, would be degraded.

Figure 6: Team Connected to a Single Hub

Generic and Dynamic Trunking (FEC/GEC/IEEE 802.3ad)

FEC/GEC and IEEE 802.3ad teams cannot be connected to any hub configuration. These team types must be connected to a switch that has also been configured for this team type.

Teaming with Microsoft NLB

Teaming does not work in Microsoft's Network Load Balancing (NLB) unicast mode, only in multicast mode. Due to the mechanism used by the NLB service, the recommended teaming configuration in this environment is Failover (SLB with a standby NIC) as load balancing is managed by NLB. The TOE functionality in teaming will not operate in NLB.

Application Considerations

Teaming and Clustering

Teaming and Network Backup

Teaming and Clustering

Microsoft Cluster Software

High-Performance Computing Cluster

Oracle

Microsoft Cluster Software

In each cluster node, it is strongly recommended that customers install at least two network adapters (on-board adapters are acceptable). These interfaces serve two purposes. One adapter is used exclusively for intra-cluster heartbeat communications. This is referred to as the private adapter and usually resides on a separate private subnetwork. The other adapter is used for client communications and is referred to as the public adapter.

Multiple adapters may be used for each of these purposes: private, intracluster communications and public, external client communications. All Broadcom teaming modes are supported with Microsoft Cluster Software for the public adapter only. Private network adapter teaming is not supported. Microsoft indicates that the use of teaming on the private interconnect of a server cluster is not supported because of delays that could possibly occur in the transmission and receipt of heartbeat packets between the nodes. For best results, when you want redundancy for the private interconnect, disable teaming and use the available ports to form a second private interconnect. This achieves the same end result and provides dual, robust communication paths for the nodes to communicate over.

For teaming in a clustered environment, customers are recommended to use the same brand of adapters.

Figure 7 shows a 2-node Fibre-Channel cluster with three network interfaces per cluster node: one private and two public. On each node, the two public adapters are teamed, and the private adapter is not. Teaming is supported across the same switch or across two switches. Figure 8 shows the same 2-node Fibre-Channel cluster in this configuration.

Figure 7: Clustering With Teaming Across One Switch

NOTE: Microsoft Network Load Balancing is not supported with Microsoft Cluster Software.

High-Performance Computing Cluster

Gigabit Ethernet is typically used for the following three purposes in high-performance computing cluster (HPCC) applications:

In our current HPCC offerings, only one of the on-board adapters is used. If Myrinet or IB is present, this adapter serves I/O and administration purposes; otherwise, it is also responsible for IPC. In case of an adapter failure, the administrator can use the Felix package to easily configure adapter 2. Adapter teaming on the host side is neither tested nor supported in HPCC.

Advanced Features

PXE is used extensively for the deployment of the cluster (installation and recovery of compute nodes). Teaming is typically not used on the host side and it is not a part of our standard offering. Link aggregation is commonly used between switches, especially for large configurations. Jumbo frames, although not a part of our standard offering, may provide performance improvement for some applications due to reduced CPU overhead.

Oracle

In our Oracle Solution Stacks, we support adapter teaming in both the private network (interconnect between RAC nodes) and public network with clients or the application layer above the database layer.

Figure 8: Clustering With Teaming Across Two Switches

Teaming and Network Backup

Load Balancing and Failover

Fault Tolerance

When you perform network backups in a nonteamed environment, overall throughput on a backup server adapter can be easily impacted due to excessive traffic and adapter overloading. Depending on the number of backup servers, data streams, and tape drive speed, backup traffic can easily consume a high percentage of the network link bandwidth, thus impacting production data and tape backup performance. Network backups usually consist of a dedicated backup server running with tape backup software such as NetBackup, Galaxy or Backup Exec. Attached to the backup server is either a direct SCSI tape backup unit or a tape library connected through a fiber channel storage area network (SAN). Systems that are backed up over the network are typically called clients or remote servers and usually have a tape backup software agent installed. Figure 9 shows a typical 1 Gbps nonteamed network environment with tape backup implementation.

Figure 9: Network Backup without Teaming

Because there are four client servers, the backup server can simultaneously stream four backup jobs (one per client) to a multidrive autoloader. Because of the single link between the switch and the backup server; however, a 4-stream backup can easily saturate the adapter and link. If the adapter on the backup server operates at 1 Gbps (125 MB/s), and each client is able to stream data at 20 MB/s during tape backup, the throughput between the backup server and switch will be at 80 MB/s (20 MB/s x 4), which is equivalent to 64% of the network bandwidth. Although this is well within the network bandwidth range, the 64% constitutes a high percentage, especially if other applications share the same link.

Load Balancing and Failover

As the number of backup streams increases, the overall throughput increases. Each data stream, however, may not be able to maintain the same performance as a single backup stream of 25 MB/s. In other words, even though a backup server can stream data from a single client at 25 MB/s, it is not expected that four simultaneously-running backup jobs will stream at 100 MB/s (25 MB/s x 4 streams). Although overall throughput increases as the number of backup streams increases, each backup stream can be impacted by tape software or network stack limitations.

For a tape backup server to reliably use adapter performance and network bandwidth when backing up clients, a network infrastructure must implement teaming such as load balancing and fault tolerance. Data centers will incorporate redundant switches, link aggregation, and trunking as part of their fault tolerant solution. Although teaming device drivers will manipulate the way data flows through teamed interfaces and failover paths, this is transparent to tape backup applications and does not interrupt any tape backup process when backing up remote systems over the network. Figure 10 shows a network topology that demonstrates tape backup in a Broadcom teamed environment and how smart load balancing can load balance tape backup data across teamed adapters.

There are four paths that the client-server can use to send data to the backup server, but only one of these paths will be designated during data transfer. One possible path that Client-Server Red can use to send data to the backup server is:

Example Path: Client-Server Red sends data through Adapter A, Switch 1, Backup Server Adapter A.

The designated path is determined by two factors:

The teamed interface on the backup server transmits a gratuitous address resolution protocol (G-ARP) to Client-Server Red, which in turn, causes the client server ARP cache to get updated with the Backup Server MAC address. The load balancing mechanism within the teamed interface determines the MAC address embedded in the G-ARP. The selected MAC address is essentially the destination for data transfer from the client server.On Client-Server Red, the SLB teaming algorithm will determine which of the two adapter interfaces will be used to transmit data. In this example, data from Client Server Red is received on the backup server Adapter A interface. To demonstrate the SLB mechanisms when additional load is placed on the teamed interface, consider the scenario when the backup server initiates a second backup operation: one to Client-Server Red, and one to Client-Server Blue. The route that Client-Server Blue uses to send data to the backup server is dependant on its ARP cache, which points to the backup server MAC address. Because Adapter A of the backup server is already under load from its backup operation with Client-Sever Red, the Backup Server invokes its SLB algorithm to inform Client-Server Blue (through an G-ARP) to update its ARP cache to reflect the backup server Adapter B MAC address. When Client-Server Blue needs to transmit data, it uses either one of its adapter interfaces, which is determined by its own SLB algorithm. What is important is that data from Client-Server Blue is received by the Backup Server Adapter B interface, and not by its Adapter A interface. This is important because with both backup streams running simultaneously, the backup server must load balance data streams from different clients. With both backup streams running, each adapter interface on the backup server is processing an equal load, thus load-balancing data across both adapter interfaces.

The same algorithm applies if a third and fourth backup operation is initiated from the backup server. The teamed interface on the backup server transmits a unicast G-ARP to backup clients to inform them to update their ARP cache. Each client then transmits backup data along a route to the target MAC address on the backup server.

Fault Tolerance

If a network link fails during tape backup operations, all traffic between the backup server and client stops and backup jobs fail. If, however, the network topology was configured for both Broadcom SLB and switch fault tolerance, then this would allow tape backup operations to continue without interruption during the link failure. All failover processes within the network are transparent to tape backup software applications. To understand how backup data streams are directed during network failover process, consider the topology in Figure 10. Client-Server Red is transmitting data to the backup server through Path 1, but a link failure occurs between the backup server and the switch. Because the data can no longer be sent from Switch #1 to the Adapter A interface on the backup server, the data is redirected from Switch #1 through Switch #2, to the Adapter B interface on the backup server. This occurs without the knowledge of the backup application because all fault tolerant operations are handled by the adapter team interface and trunk settings on the switches. From the client server perspective, it still operates as if it is transmitting data through the original path.

Figure 10: Network Backup With SLB Teaming Across Two Switches

Troubleshooting Teaming Problems

Teaming Configuration Tips

Troubleshooting Guidelines

When running a protocol analyzer over a virtual adapter teamed interface, the MAC address shown in the transmitted frames may not be correct. The analyzer does not show the frames as constructed by BASP and shows the MAC address of the team and not the MAC address of the interface transmitting the frame. It is suggested to use the following process to monitor a team:

Teaming Configuration Tips

When troubleshooting network connectivity or teaming functionality issues, ensure that the following information is true for your configuration.

  1. Although mixed-speed SLB teaming is supported, it is recommended that all adapters in a team be the same speed (either all Gigabit Ethernet or all Fast Ethernet). For speeds of 10 Gbps, it is highly recommended that all adapters in a team be the same speed.
  2. If LiveLink is not enabled, disable Spanning Tree Protocol or enable an STP mode that bypasses the initial phases (for example, Port Fast, Edge Port) for the switch ports connected to a team.
  3. All switches that the team is directly connected to must have the same hardware revision, firmware revision, and software revision to be supported.
  4. To be teamed, adapters should be members of the same VLAN. In the event that multiple teams are configured, each team should be on a separate network.
  5. Do not assign a Locally Administered Address on any physical adapter that is a member of a team.
  6. Verify that power management is disabled on all physical members of any team.
  7. Remove any static IP address from the individual physical team members before the team is built.
  8. A team that requires maximum throughput should use LACP or GEC\FEC. In these cases, the intermediate driver is only responsible for the outbound load balancing while the switch performs the inbound load balancing.
  9. Aggregated teams (802.3ad \ LACP and GEC\FEC) must be connected to only a single switch that supports IEEE 802.3a, LACP or GEC/FEC.
  10. It is not recommended to connect any team to a hub, as hubs only support half duplex. Hubs should be connected to a team for troubleshooting purposes only. Disabling the device driver of a network adapter participating in an LACP or GEC/FEC team may have adverse affects with network connectivity. Broadcom recommends that the adapter first be physically disconnected from the switch before disabling the device driver in order to avoid a network connection loss.
  11. Verify the base (Miniport) and team (intermediate) drivers are from the same release package. The mixing of base and teaming drivers from different releases is not supported.
  12. Test the connectivity to each physical adapter prior to teaming.
  13. Test the failover and fallback behavior of the team before placing into a production environment.
  14. When moving from a nonproduction network to a production network, it is strongly recommended to test again for failover and fallback.
  15. Test the performance behavior of the team before placing into a production environment.
  16. Network teaming is not supported when running iSCSI traffic via Microsoft iSCSI initiator or iSCSI offload. MPIO should be used instead of Broadcom network teaming for these ports.
  17. For information on iSCSI boot and iSCSI offload restrictions, see iSCSI Protocol.

Troubleshooting Guidelines

Before you call for support, make sure you have completed the following steps for troubleshooting network connectivity problems when the server is using adapter teaming.

  1. Make sure the link light is ON for every adapter and all the cables are attached.
  2. Check that the matching base and intermediate drivers belong to the same release and are loaded correctly.
  3. Check for a valid IP Address using the Windows ipconfig command.
  4. Check that STP is disabled or Edge Port/Port Fast is enabled on the switch ports connected to the team or that LiveLink is being used.
  5. Check that the adapters and the switch are configured identically for link speed and duplex.
  6. If possible, break the team and check for connectivity to each adapter independently to confirm that the problem is directly associated with teaming.
  7. Check that all switch ports connected to the team are on the same VLAN.
  8. Check that the switch ports are configured properly for Generic Trunking (FEC/GEC)/802.3ad-Draft Static type of teaming and that it matches the adapter teaming type. If the system is configured for an SLB type of team, make sure the corresponding switch ports are not configured for Generic Trunking (FEC/GEC)/802.3ad-Draft Static types of teams.

Frequently Asked Questions

Question: Under what circumstances is traffic not load balanced? Why is all traffic not load balanced evenly across the team members?
Answer: The bulk of traffic does not use IP/TCP/UDP or the bulk of the clients are in a different network. The receive load balancing is not a function of traffic load, but a function of the number of clients that are connected to the server.

Question: What network protocols are load balanced when in a team?
Answer: Broadcom's teaming software only supports IP/TCP/UDP traffic. All other traffic is forwarded to the primary adapter.

Question: Which protocols are load balanced with SLB and which ones are not?
Answer: Only IP/TCP/UDP protocols are load balanced in both directions: send and receive. IPX is load balanced on the transmit traffic only.

Question: Can I team a port running at 100 Mbps with a port running at 1000 Mbps?
Answer: Mixing link speeds within a team is only supported for Smart Load Balancing™ teams and 802.3ad teams.

Question: Can I team a fiber adapter with a copper Gigabit Ethernet adapter?
Answer: Yes with SLB, and yes if the switch allows for it in FEC/GEC and 802.3ad.

Question: What is the difference between adapter load balancing and Microsoft's Network Load Balancing (NLB)?
Answer: Adapter load balancing is done at a network session level, whereas NLB is done at the server application level.

Question: Can I connect the teamed adapters to a hub?
Answer: Teamed ports can be connected to a hub for troubleshooting purposes only. However, this practice is not recommended for normal operation because the performance would be degraded due to hub limitations. Connect the teamed ports to a switch instead.

Question: Can I connect the teamed adapters to ports in a router?
Answer: No. All ports in a team must be on the same network; in a router, however, each port is a separate network by definition. All teaming modes require that the link partner be a Layer 2 switch.

Question: Can I use teaming with Microsoft Cluster Services?
Answer: Yes. Teaming is supported on the public network only, not on the private network used for the heartbeat link.

Question: Can PXE work over a virtual adapter (team)?
Answer: A PXE client operates in an environment before the operating system is loaded; as a result, virtual adapters have not been enabled yet. If the physical adapter supports PXE, then it can be used as a PXE client, whether or not it is part of a virtual adapter when the operating system loads. PXE servers may operate over a virtual adapter.

Question: Can WOL work over a virtual adapter (team)?
Answer: Wake-on-LAN functionality operates in an environment before the operating system is loaded. WOL occurs when the system is off or in standby, so no team is configured.

Question: What is the maximum number of ports that can be teamed together?
Answer: Up to eight ports can be assigned to a team.

Question: What is the maximum number of teams that can be configured on the same server?
Answer: Up to eight teams can be configured on the same server.

Question: Why does my team loose connectivity for the first 30 to 50 seconds after the Primary adapter is restored (fallback)?
Answer: Because Spanning Tree Protocol is bringing the port from blocking to forwarding. You must enable Port Fast or Edge Port on the switch ports connected to the team or use LiveLink to account for the STP delay.

Question: Can I connect a team across multiple switches?
Answer: Smart Load Balancing can be used with multiple switches because each physical adapter in the system uses a unique Ethernet MAC address. Link Aggregation and Generic Trunking cannot operate across switches because they require all physical adapters to share the same Ethernet MAC address.

Question: How do I upgrade the intermediate driver (BASP)?
Answer: The intermediate driver cannot be upgraded through the Local Area Connection Properties. It must be upgraded using the Setup installer.

Question: How can I determine the performance statistics on a virtual adapter (team)?
Answer: In Broadcom Advanced Control Suite, click the Statistics tab for the virtual adapter.

Question: Can I configure NLB and teaming concurrently?
Answer: Yes, but only when running NLB in a multicast mode (NLB is not supported with MS Cluster Services).

Question: Should both the backup server and client servers that are backed up be teamed?
Answer: Because the backup server is under the most data load, it should always be teamed for link aggregation and failover. A fully redundant network, however, requires that both the switches and the backup clients be teamed for fault tolerance and link aggregation.

Question: During backup operations, does the adapter teaming algorithm load balance data at a byte-level or a session-level?
Answer: When using adapter teaming, data is only load balanced at a session level and not a byte level to prevent out-of-order frames. Adapter teaming load balancing does not work the same way as other storage load balancing mechanisms such as EMC PowerPath.

Question: Is there any special configuration required in the tape backup software or hardware to work with adapter teaming?
Answer: No special configuration is required in the tape software to work with teaming. Teaming is transparent to tape backup applications.

Question: How do I know what driver I am currently using?
Answer: In all operating systems, the most accurate method for checking the driver revision is to physically locate the driver file and check the properties.

Question: Can SLB detect a switch failure in a Switch Fault Tolerance configuration?
Answer: No. SLB can only detect the loss of link between the teamed port and its immediate link partner. SLB cannot detect link failures on other ports.

Question: Why does my team lose connectivity for the first 30 to 50 seconds after the primary adapter is restored (fall-back after a failover)?
Answer: During a fall-back event, link is restored causing Spanning Tree Protocol to configure the port for blocking until it determines that it can move to the forwarding state. You must enable Port Fast or Edge Port on the switch ports connected to the team to prevent the loss of communications caused by STP.

Question: Where do I monitor real time statistics for an adapter team in a Windows server?
Answer: Use the Broadcom Advanced Control Suite (BACS) to monitor general, IEEE 802.3 and custom counters.

Question: What features are not supported on a multivendor team?
Answer: TOE, VLAN tagging, and RSS are not supported on a multivendor team.

Appendix A: Event Log Messages

Windows System Event Log Messages

Base Driver (Physical Adapter/Miniport)

Intermediate Driver (Virtual Adapter/Team)

Virtual Bus Driver (VBD)

Windows System Event Log Messages

The known base and intermediate Windows System Event Log status messages for the Broadcom NetXtreme II adapters are listed in Table 8 and Table 9. As a Broadcom adapter driver loads, Windows places a status code in the system event viewer. There may be up to two classes of entries for these event codes depending on whether both drivers are loaded (one set for the base or miniport driver and one set for the intermediate or teaming driver).

Base Driver (Physical Adapter/Miniport)

The base driver is identified by source L2ND. Table 8 lists the event log messages supported by the base driver, explains the cause for the message, and provides the recommended action.

 

Note: In Table 8, message numbers 1 through 17 apply to both NDIS 5.x and NDIS 6.x drivers, message numbers 18 through 23 apply only to the NDIS 6.x driver.

Table 8: Base Driver Event Log Messages 
Message Number
Severity
Message
Cause
Corrective Action
1
Error
Failed to allocate memory for the device block. Check system memory resource usage.
The driver cannot allocate memory from the operating system.
Close running applications to free memory.
2
Error
Failed to allocate map registers.
The driver cannot allocate map registers from the operating system.
Unload other drivers that may allocate map registers.
3
Error
Failed to access configuration information. Reinstall the network driver.
The driver cannot access PCI configuration space registers on the adapter.
For add-in adapters: reseat the adapter in the slot, move the adapter to another PCI slot, or replace the adapter.
4
Warning
The network link is down. Check to make sure the network cable is properly connected.
The adapter has lost its connection with its link partner.
Check that the network cable is connected, verify that the network cable is the right type, and verify that the link partner (for example, switch or hub) is working correctly.
5
Informational
The network link is up.
The adapter has established a link.
No action is required.
6
Informational
Network controller configured for 10Mb half-duplex link.
The adapter has been manually configured for the selected line speed and duplex settings.
No action is required.
7
Informational
Network controller configured for 10Mb full-duplex link.
The adapter has been manually configured for the selected line speed and duplex settings.
No action is required.
8
Informational
Network controller configured for 100Mb half-duplex link.
The adapter has been manually configured for the selected line speed and duplex settings.
No action is required.
9
Informational
Network controller configured for 100Mb full-duplex link.
The adapter has been manually configured for the selected line speed and duplex settings.
No action is required.
10
Informational
Network controller configured for 1Gb half-duplex link.
The adapter has been manually configured for the selected line speed and duplex settings.
No action is required.
11
Informational
Network controller configured for 1Gb full-duplex link.
The adapter has been manually configured for the selected line speed and duplex settings.
No action is required.
12
Informational
Network controller configured for 2.5Gb full-duplex link.
The adapter has been manually configured for the selected line speed and duplex settings.
No action is required.
13
Error
Medium not supported.
The operating system does not support the IEEE 802.3 medium.
Reboot the operating system, run a virus check, run a disk check (chkdsk), and reinstall the operating system.
14
Error
Unable to register the interrupt service routine.
The device driver cannot install the interrupt handler.
Reboot the operating system; remove other device drivers that may be sharing the same IRQ.
15
Error
Unable to map IO space.
The device driver cannot allocate memory-mapped I/O to access driver registers.
Remove other adapters from the system, reduce the amount of physical memory installed, and replace the adapter.
16
Informational
Driver initialized successfully.
The driver has successfully loaded.
No action is required.
17
Informational
NDIS is resetting the miniport driver.
The NDIS layer has detected a problem sending/receiving packets and is resetting the driver to resolve the problem.
Run Broadcom Advanced Control Suite diagnostics; check that the network cable is good.
18
Error
Unknown PHY detected. Using a default PHY initialization routine.
The driver could not read the PHY ID.
Replace the adapter.
19
Error
This driver does not support this device. Upgrade to the latest driver.
The driver does not recognize the installed adapter.
Upgrade to a driver version that supports this adapter.
20
Error
Driver initialization failed.
Unspecified failure during driver initialization.
Reinstall the driver, update to a newer driver, run Broadcom Advanced Control Suite diagnostics, or replace the adapter.
21
Informational
Network controller configured for 10Gb full-duplex link.
The adapter has been manually configured for the selected line speed and duplex settings.
No action is required.
22
Error
Network controller failed initialization because it cannot allocate system memory.
Insufficient system memory prevented the initialization of the driver.
Increase system memory.
23
Error
Network controller failed to exchange the interface with the bus driver.
The driver and the bus driver are not compatible.
Update to the latest driver set, ensuring the major and minor versions for both NDIS and the bus driver are the same.

Intermediate Driver (Virtual Adapter/Team)

The intermediate driver is identified by source BLFM, regardless of the base driver revision. Table 9 lists the event log messages supported by the intermediate driver, explains the cause for the message, and provides the recommended action.

Table 9: Intermediate Driver Event Log Messages 
System Event
Message Number
Severity
Message
Cause
Corrective Action
1
Informational
Event logging enabled for Broadcom Advanced Server Program driver.
No action is required.
2
Error
Unable to register with NDIS.
The driver cannot register with the NDIS interface.
Unload other NDIS drivers.
3
Error
Unable to instantiate the management interface.
The driver cannot create a device instance.
Reboot the operating system.
4
Error
Unable to create symbolic link for the management interface.
Another driver has created a conflicting device name.
Unload the conflicting device driver that uses the name Blf.
5
Informational
Broadcom Advanced Server Program Driver has started.
The driver has started.
No action is required.
6
Informational
Broadcom Advanced Server Program Driver has stopped.
The driver has stopped.
No action is required.
7
Error
Could not allocate memory for internal data structures.
The driver cannot allocate memory from the operating system.
Close running applications to free memory.
8
Warning
Could not bind to adapter.
The driver could not open one of the team physical adapters.
Unload and reload the physical adapter driver, install an updated physical adapter driver, or replace the physical adapter.
9
Informational
Successfully bind to adapter.
The driver successfully opened the physical adapter.
No action is required.
10
Warning
Network adapter is disconnected.
The physical adapter is not connected to the network (it has not established link).
Check that the network cable is connected, verify that the network cable is the right type, and verify that the link partner (switch or hub) is working correctly.
11
Informational
Network adapter is connected.
The physical adapter is connected to the network (it has established link).
No action is required.
12
Error
Broadcom Advanced Program Features Driver is not designed to run on this version of Operating System.
The driver does not support the operating system on which it is installed.
Consult the driver release notes and install the driver on a supported operating system or update the driver.
13
Informational
Hot-standby adapter is selected as the primary adapter for a team without a load balancing adapter.
A standby adapter has been activated.
Replace the failed physical adapter.
14
Informational
Network adapter does not support Advanced Failover.
The physical adapter does not support the Broadcom NIC Extension (NICE).
Replace the adapter with one that does support NICE.
15
Informational
Network adapter is enabled via management interface.
The driver has successfully enabled a physical adapter through the management interface.
No action is required.
16
Warning
Network adapter is disabled via management interface.
The driver has successfully disabled a physical adapter through the management interface.
No action is required.
17
Informational
Network adapter is activated and is participating in network traffic.
A physical adapter has been added to or activated in a team.
No action is required.
18
Informational
Network adapter is de-activated and is no longer participating in network traffic.
The driver does not recognize the installed adapter.
No action is required.
19
Informational
The LiveLink feature in BASP connected the link for the network adapter.
The connection with the remote target(s) for the LiveLink-enabled team member has been established or restored
No action is required.
20
Informational
The LiveLink feature in BASP disconnected the link for the network adapter.
The LiveLink-enabled team member is unable to connect with the remote target(s).
No action is required.

Virtual Bus Driver (VBD)

Table 10: Virtual Bus Driver (VBD) Event Log Messages
Message Number
Severity
Message
Cause
Corrective Action
1
Error
Failed to allocate memory for the device block. Check system memory resource usage.
The driver cannot allocate memory from the operating system.
Close running applications to free memory.
2
Informational
The network link is down. Check to make sure the network cable is properly connected.
The adapter has lost its connection with its link partner.
Check that the network cable is connected, verify that the network cable is the right type, and verify that the link partner (for example, switch or hub) is working correctly.
3
Informational
The network link is up.
The adapter has established a link.
No action is required.
4
Informational
Network controller configured for 10Mb half-duplex link.
The adapter has been manually configured for the selected line speed and duplex settings.
No action is required.
5
Informational
Network controller configured for 10Mb full-duplex link.
The adapter has been manually configured for the selected line speed and duplex settings.
No action is required.
6
Informational
Network controller configured for 100Mb half-duplex link.
The adapter has been manually configured for the selected line speed and duplex settings.
No action is required.
7
Informational
Network controller configured for 100Mb full-duplex link.
The adapter has been manually configured for the selected line speed and duplex settings.
No action is required.
8
Informational
Network controller configured for 1Gb half-duplex link.
The adapter has been manually configured for the selected line speed and duplex settings.
No action is required.
9
Informational
Network controller configured for 1Gb full-duplex link.
The adapter has been manually configured for the selected line speed and duplex settings.
No action is required.
10
Error
Unable to register the interrupt service routine.
The device driver cannot install the interrupt handler.
Reboot the operating system; remove other device drivers that may be sharing the same IRQ.
11
Error
Unable to map IO space.
The device driver cannot allocate memory-mapped
I/O to access driver registers.
Remove other adapters from the system, reduce the amount of physical memory installed, and replace the adapter.
12
Informational
Driver initialized successfully.
The driver has successfully loaded.
No action is required.
13
Error
Driver initialization failed.
Unspecified failure during driver initialization.
Reinstall the driver, update to a newer driver, run Broadcom Advanced Control Suite diagnostics, or replace the adapter.
14
Error
This driver does not support this device. Upgrade to the latest driver.
The driver does not recognize the installed adapter.
Upgrade to a driver version that supports this adapter.
15
Error
This driver fails initialization because the system is running out of memory.
Insufficient system memory prevented the initialization of the driver.
Increase system memory.


Back to Contents Page