iSCSI performance analysis and tuning

This procedure provides a solution for Internet Small Computer Systems Interface (iSCSI) host performance problems while connected to a Lenovo Storage V7000 system and its connectivity to the network switch.

1. Disable the TCP delayed acknowledgment feature.
   To disable this feature, refer to OS/platform documentation.

   • VMWare: http://kb.vmware.com/selfservice/ microsites/microsite.do
   • Windows: http://support.microsoft.com/kb/823764

   The primary signature of this issue: read performance is significantly lower than write performance. Transmission Control Protocol (TCP) delayed acknowledgment is a technique that is used by some implementations of the TCP in an effort to improve network performance. However, in this scenario where the number of outstanding I/O is 1, the technique can significantly reduce I/O performance.

   In essence, several ACK responses can be combined together into a single response, reducing protocol overhead. As described in RFC 1122, a host can delay sending an ACK response by up to 500 ms. Additionally, with a stream of full-sized incoming segments, ACK responses must be sent for every second segment.

   Important: The host must be rebooted for these settings to take effect. A few platforms (for example, standard Linux distributions) do not provide a way to disable this feature. However, the issue was resolved with the version 7.1 release, and no host configuration changes are required to manage TcpDelayedAck behavior.
2. Enable jumbo frame for iSCSI.

   Jumbo frames are Ethernet frames with a size in excess of 1500 bytes. The maximum transmission unit (MTU) parameter is used to measure the size of jumbo frames.

   The Lenovo Storage V7000 supports 9000-bytes MTU. Refer to the CLI command cfgportip to enable jumbo frame. This command is disruptive as the link flips and the I/O operation through that port pauses.

   The network must support jumbo frames end-to-end for this to be effective; verify this by sending a ping packet to be delivered without fragmentation. For example:

   • Windows:

     ping -t <iscsi target ip> -S <iscsi initiator ip> -f -l <new mtu size - packet overhead (usually 36, might differ)>

     The following command is an example of a command that is used to check whether a 9000-bytes MTU is set correctly on a Windows 7 system:

     ping -t -S 192.168.1.117 192.168.1.217 -f -l 8964

     The following output is an example of a successful reply:

     192.168.1.217: bytes=8964 time=1ms TTL=52

   • Linux:

     ping -l <source iscsi initatior ip> -s <new mtu size> -M do <iscsi target ip> 

   • ESXi:

     ping <iscsi target ip> -I <source iscsi initiator ip> -s <new mtu size - 28> -d

3. Verify the switch's port statistic where initiator/target ports are connected to make sure that packet drops are not high.

   Review network architecture to avoid any bottlenecks and oversubscription. The network needs to be balanced to avoid any packet drop; packet drop significantly reduces storage performance. Involve networking support to fix any such issues.

4. Optimize and utilize all iSCSI ports.
   To optimize Lenovo Storage V7000 resource utilization, all iSCSI ports must be used.

   • Each port is assigned to one CPU, and by balancing the login, one can maximize CPU utilization and achieve better performance. Ideally, configure subnets equal to the number of iSCSI ports on the Lenovo Storage V7000 node. Configure each port of a node with an IP on a different subnet and keep it the same for other nodes.The following example displays an ideal configuration:

     
     Node 1
     Port 1: 192.168.1.11
     Port 2: 192.168.2.21
     Port 3: 192.168.3.31
     
     Node 2:
     Port 1: 192.168.1.12
     Port 2: 192.168.2.22
     Port 3: 192.168.3.33
     

   • Avoid situations where 50 hosts are logged in to port 1 and only five hosts are logged in to port 2.
   • Use proper subnetting to achieve a balance between the number of sessions and redundancy.
5. Troubleshoot problems with PFC settings.

   You do not need to enable PFC on the Lenovo Storage V7000 system. Lenovo Storage V7000 reads the data center bridging exchange (DCBx) packet and enables PFC for iSCSI automatically if it is enabled on the switch. In the lsportip command output, the fields lossless_iscsi and lossless_iscsi6 show [on/off] depending on whether PFC is enabled or not for iSCSI on the system.

   If the fields lossless_iscsi and lossless_iscsi6 are showing off, it might be due to one of the following reasons:

   1. VLAN is not set for that IP. Verify the following checks:
      • For IP address type IPv4, check the vlan field in the lsportip output. It should not be blank.
      • For IP address type IPv6, check the vlan_6 field in the lsportip output. It should not be blank.
      • If the vlan and vlan_6 fields are blank, set the VLAN for the IP type using Configuring VLAN for iSCSI.
   2. Host flag is not set for that IP. Verify the following checks:
      • For IP address type IPv4, check the host field in the lsportip output. It should be yes.
      • For IP address type IPv6, check the host_6 field in the lsportip output. It should be yes.
      • If the host and host_6 fields are not yes, set the host flag for the IP type using the cfgportip CLI command.
   3. PFC is not properly set on the switch.

      If the VLAN is properly set, and the host flag is also set, but the lossless_iscsi or lossless_iscsi6 field is still showing off, some switch settings might be missing or incorrect.

      Verify the following settings in the switch:

      • Priority tag is set for iSCSI traffic.
      • PFC is enabled for priority tag that is assigned to iSCSI CoS.
      • DCBx is enabled on the switch.
      Also check the appropriate documentation:

      • Consult the documentation for enabling PFC on your specific switch.
      • Consult the documentation for enabling PFC on Red Hat Enterprise Linux (RHEL) and Windows hosts specific to your configuration.
6. Ensure that proper bandwidth is given to iSCSI on the network.

   You can divide the bandwidth among the various types of traffic. It is important to assign proper bandwidth for good performance. To assign bandwidth for iSCSI traffic, you need to first enable the priority flow control for iSCSI.