The system supports 25 Gbps Ethernet adapters that provide both iSCSI and iSCSI
Extensions over RDMA (iSER) connections. Review prerequisite system requirements and installation
considerations before deployment.
It is helpful to be familiar with the following terms when you are considering the
features of the 25 Gbps Ethernet adapter for these connections.
- Remote Direct Memory Access (RDMA)
- RDMA is an networking standard that allows the adapters to transfer data directly to or from the
endpoints in a connection without using CPU resources on either of the endpoints. These transfers
occur in simultaneously with other system operations and do not impact overall system performance.
When system performs an I/O operation over an RDMA-based connection, data is sent directly to the
network, which reduces latency and increases the speed of data transfers.
- iSCSI Extensions for RDMA (iSER)
- iSER is a network protocol that extends the iSCSI protocol to use RDMA. You can implement
RDMA-based connections that use existing Ethernet networking structures and connections without
upgrading current hardware. Currently the system supports RDMA-based connections with RDMA over
Converged Ethernet (RoCE) or Internet Wide Area RDMA Protocol (iWARP). If you are undecided whether
to use RoCE or iWARP in your data center, search the internet for iSCSI Extensions for RDMA
to obtain more information.
Review the following elements when you are deciding to install the optional 25 Gbps adapter:
- The minimum software level that is needed to
support the 25 Gbps Ethernet adapter is 8.1.1.1 for host attachment and external storage system
connections. For node-to-node connections, the minimum software level of 8.2.1 or later is
required.
- Two types of 25 Gbps Ethernet adapters are available:
- 2-port 25 Gbps Ethernet (RoCE) adapter
- 2-port 25 Gbps Ethernet (iWARP) adapter
These types
of adapters can be used for connections to hosts, external storage systems, or between nodes to
create a system. For host and external storage system connections, both these 25 Gbps adapters
support iSCSI and RDMA-based connections. When the 25 Gbps adapter is installed on nodes in the
system, RDMA technology can be used for node-to-node communications.
- Depending on your system model, the number of Ethernet ports that are supported per
node or node canister varies. Verify your system model and the support before installing or using 25
Gbps Ethernet adapters. To find the maximum number of Ethernet ports that can be used by the
software on this node, search for Configuration Limits and Restrictions and
your product name at the following
website:
http://www.ibm.com/support/
Three or four onboard ports are available depending on the model, plus up to eight Ethernet ports
on PCIe adapters.
- If you are using
RDMA-technology for node-to-node communications, ensure that port identifiers on the nodes and the
25 Gbps adapters are the same. To ensure these port identifiers are the same, install the 25 Gbps
adapters into the correct slot in the node so ports for node-to-node communications are identified
on the same slot of the node.
- 10 Gbps and 25 Gbps adapters are supported on the same platform.
- Most host operating systems that support 10 Gbps Ethernet also support 25 Gbps Ethernet. To
verify host operating system support, search for Configuration Limits and
Restrictions and your product name at the following website:
http://www.ibm.com/support/
- The 25 Gbps adapter has better latency, port consolidation, and approximately 25-40% better
bandwidth usage when compared to the 10 Gbps adapter.
- Using 25 Gbps adapters for IP replication is not currently supported.
- The correct 25 Gbps Ethernet adapter feature code must be used, depending on whether the SAN
implements iWARP or RoCE, as the protocols do not inter-operate.
- iWARP adapters can communicate RDMA only to iWARP adapters.
- RoCE adapters can communicate RDMA only to RoCE adapters.
- An iSER link from a RoCE host or storage system
Ethernet port works only to a RoCE Ethernet port on a control enclosure or node.
- RoCE is supported by the InfiniBand® Trade Association (IBTA) and the RoCE Alliance.
- iWARP is supported by Chelsio and Intel.