Originated: November 4, 2015
********************************************************************************

Patch-163030: Patch for OneFS 7.1.1.0 - 7.1.1.7, 7.2.0.0 - 7.2.0.4, 7.2.1.0 and 
7.2.1.1.
This patch installs a new kernel driver for the QLogic (formerly Broadcom)
NetXtreme Ethernet (BXE) 10 GigE NIC.

--------------------------------------------------------------------------------

This patch can be installed on clusters running the following OneFS versions: 
7.1.1.0 - 7.1.1.7
7.2.0.0 - 7.2.0.4
7.2.1.0 - 7.2.1.1

This patch deprecates the following patches: 
patch-148695
patch-140498
patch-150883
patch-155678
patch-159992
patch-157789
patch-162062

********************************************************************************

RESOLVED ISSUES

NOTE:

* This patch contains several changes provided by QLogic to address a variety
  of driver issues. Some of these fixes may already be resolved in certain
  versions of OneFS. If your cluster is running any version of OneFS listed
  above, it is strongly recommended that you install this patch to avoid a
  serious data integrity issue. 

* In addition to the issues listed below, QLogic has also provided a set of
  stability fixes to enhance overall driver functionality. 

*  Bug ID 162042, 163324
   If all of the following conditions were met, it was possible for data to
   become inconsistent while it was being transferred to an Isilon node:

   * The node contained a 10GigE network interface card (NIC) that was using
   the QLogic NetXtreme Ethernet (BXE) driver.
  
   * VLAN tagging was enabled for the subnet assigned to the NIC.
  
   * Large Receive Offload (LRO) was enabled on the NIC. (LRO enabled is the
     default setting on BXE NICs.) 

   This issue occurred because, under these conditions, the version of the
   BXE driver installed on the node incorrectly interpreted the size of a
   TCP frame.

*  Bug ID 139997, 138521
   HD400, NL410, S210, X210 and X410 nodes that were configured to communicate 
   through a 10 GigE network interface card that was using the QLogic NetXtreme 
   Ethernet(BXE) driver that was introduced in OneFS 7.1.1 might have restarted
   unexpectedly. If this occurred, a message similar to the following appeared
   in the var/log/messages file:

   Node panicked with Panic Msg: sleeping thread
   0xffffff04692a0000 owns a nonsleepable lock
   
*  Bug ID 154455, 154456
   HD400, NL410, S210, X210 and X410 nodes that were configured to communicate
   through a 10 GigE network interface card that was using the QLogic
   NetXtreme Ethernet (BXE) driver which was introduced in OneFS 7.1.1 could
   have encountered an issue where the link reported "no carrier" in the output
   of the ifconfig command. Toggling the interface did not resolve the issue
   and the node had to be rebooted to reestablish the link.    

*  Bug ID 152083, 152089
   HD400, NL410, S210, X210 and X410 nodes that were configured to communicate
   through a 10 GigE network interface card that was using the QLogic NetXtreme 
   Ethernet (BXE) and were also configured to use the aggregate interface with
   LACP experienced connectivity issues with those links after rebooting the
   node or re-configuring the MTU on those interfaces.

*  Bug ID 152089, 152083
   If the cluster contained HD400, NL410, S210, X210 or X410  nodes that had
   QLogic NetExtreme 10 GigE NIC cards and any external network subnets
   connected to the cluster were set to 9000 MTU, an error similar to the
   following appeared in the /var/log/messages file and the affected nodes
   rebooted:
   
   ERROR: mbuf alloc fail for fp[01] rx chain (55)

********************************************************************************

CHANGED FILES

* /boot/kernel.amd64/if_bxe.ko.gz
  (MD5 = 31d8b624a23bbb2c12878dfb5b5d66cd)

* /boot/loader.conf
  (MD5 = 61dba0d98d73cb02727b314b528eba89)

********************************************************************************

INSTALLATION/REMOVAL IMPACTS

During installation or removal of this patch clients and applications that
are connected to the cluster through BXE 10 GigE network interfaces will be 
disconnected. These connections must be reestablished after the patch
installation is complete. 

To ensure the changes made by this patch take effect, you must reboot all
nodes in the cluster at the end of the patch installation process. You must
also reboot all nodes in the cluster if you remove this patch. You can either
reboot the nodes in succession (also known as a rolling reboot) or perform a 
simultaneous reboot of all nodes in the cluster.  
 
A simultaneous reboot can be performed from the OneFS web administration
interface or from the command-line interface. A simultaneous reboot of the 
cluster is faster than a rolling reboot; however, because all nodes are 
rebooted simultaneously, this type of reboot causes a temporary 
interruption of service while the nodes reboot.
 
A rolling reboot requires that you reboot each node in succession manually
from the command-line interface. When a node is rebooted, client connections
to that node will be disconnected and might need to be reestablished. 
 
Whether you perform a simultaneous reboot or rolling reboot of the nodes, we 
recommend that you install the patch during an off-hours maintenance window
to minimize the disruption of service to clients.
 
If you cannot install the patch during a scheduled maintenance window, inform
users when the reboot will take place and how they will be affected. 
 
Instructions for how to perform a simultaneous reboot or rolling reboot from
the command-line interface appear later in this document.

This patch updates the /boot/loader.conf file. This file contains
descriptive information about bootstrapping the system. If you want to
customize this information, the changes should be made to the
/boot/loader.conf.local file. If you inadvertently customized the
/boot/loader.conf file, those changes should be added to the
/boot/loader.conf.local file prior to installing this patch. Changes made to
the /boot/loader.conf file will be overwritten when this patch is installed.

********************************************************************************

IMPORTANT!  
VERIFY CLUSTER-FREE SPACE
If there is insufficient free space and you attempt to install or remove a
patch, the process might not complete successfully. An incomplete patch
installation or patch removal can leave the cluster in an inaccessible state.

Before installing or removing a patch, verify that your cluster has enough
free space to perform the procedure. For more information about maintaining
sufficient free space on an Isilon cluster, see the Best Practices Guide for
Maintaining Enough Free Space on Isilon Clusters and Pools:
https://support.emc.com/docu48119.
   
********************************************************************************

REMOVE DEPRECATED PATCHES

If any deprecated patches are installed on the cluster, perform the
following procedure to remove them before installing this patch.

IMPORTANT!

Read the INSTALLATION/REMOVAL IMPACTS in the README for the deprecated patch
before performing this procedure.

1. Verify that your cluster has enough free space to remove this patch.

2. Open an SSH connection on any node in the cluster and log in using the root 
   account.

   NOTE: If the procedure is being done on a compliance mode cluster, log in
   using the compadmin account.

3. To remove this patch, run the following command, where <ID> is the patch ID
   of the patch to be removed:
   
   isi pkg delete patch-<ID>

4. To verify that this patch was removed, run the following command:
     
   isi pkg info

5. Confirm that patch-<ID> does not appear in the list of installed packages.

********************************************************************************

INSTALLING THIS PATCH

IMPORTANT!
Read INSTALLATION/REMOVAL IMPACTS before performing this procedure.
   
1. Verify that your cluster has enough free space to install this patch.

2. Open an SSH connection on any node in the cluster and log in by using the
   root account.

   NOTE: If the patch is being installed on a compliance mode cluster, log in
   using the compadmin account.

3. Copy the patch-163030.tgz file to the /ifs/data directory on the cluster.

4. Run the following command to change to the /ifs/data directory:
   
   cd /ifs/data

5. To extract the patch file, run the following command:

   tar -zxvf patch-163030.tgz

6. To install this patch, run the following command:

   isi pkg install patch-163030.tar

7. To verify that this patch is installed, run the following command:
  
   isi pkg info

8. Confirm that patch-163030 appears in the list of installed packages.

********************************************************************************

PERFORMING A SIMULTANEOUS REBOOT

After the patch is installed, manually reboot each node in succession. 

1. Open an SSH connection on any node in the cluster and log on using the 
   root account.

   NOTE: If the procedure is being done on a compliance mode cluster, log in
   using the compadmin account.
 
2. Run the following cluster administration command to enter the isi config 
   subsystem:
  
   isi config
 
3. To reboot all the nodes in the cluster, run the following command:
 
   reboot all
 
4. At the prompt, enter yes.

********************************************************************************

PERFORMING A ROLLING REBOOT

After the patch is installed, manually reboot each node in succession. 

1. Open an SSH connection on any node in the cluster and log on using the 
   root account.

   NOTE: If the procedure is being done on a compliance mode cluster, log in 
   using the compadmin account.
   
2. Run the following command to start the Isilon IQ configuration console: 
   
   isi config
   
3. Within the configuration console, reboot any node in the cluster, except
   the node that you are connected to, by running the following command,
   where <node> is the node that you want to reboot:

   reboot <node>

4. To exit the configuration console, run the following command:

   exit

5. To monitor the progress of the reboot, run the following command at any
   time from the node that you are connected to: 
   
   sysctl efs.gmp.group
   
   Output is displayed in the following format:   
   
   efs.gmp.group: <Node,Sequential number>: { <Array IDs>:<Drive IDs> }
      
   For example, 
      
   if node 6 is marked for group removal:
   efs.gmp.group: <2,15>: { 1-5:0-3, shutdown_read_only: 6 }
      
   if node 6 is rebooting:
   efs.gmp.group: <2,15>: { 1-5:0-3, down: 6 }
   
   if node 6 is back online:
   efs.gmp.group: <2,15>: { 1-6:0-3 }
   
   NOTE: This example assumes that only one node at a time is being rebooted
   and no other devices are down.  The array ID might not match the node
   number. 
   
   For more information about the efs.gmp.group sysctl, see article 89502 on
   the EMC Online Support site.
   
6. Wait for the node to successfully reboot.
   
7. Repeat step 3 for each remaining node, except for the node that you are 
   connected to.

8. To exit the configuration console, run the following command:

   exit

9. To reboot the node that you are connected to, run the following command:
   
    shutdown -r now   

    NOTE: Running the command in step 5 will close your SSH connection to the 
    cluster.  This is true only for this step; no other step will disconnect
    the SSH session. 

********************************************************************************

REMOVING THIS PATCH

If you need to remove this patch, complete the steps below.

IMPORTANT!
Read INSTALLATION/REMOVAL IMPACTS before performing this procedure.

1. Verify that your cluster has enough free space to remove this patch.

2. Open an SSH connection on any node in the cluster and log in using the
   root account.

   NOTE: If the patch is being removed from a compliance mode cluster, log in
   using the compadmin account.

3. To remove this patch, run the following command:

   isi pkg delete patch-163030

4. To verify that this patch was removed, run the following command:

   isi pkg info

5. Confirm that patch-163030 does not appear in the list of installed packages.

6. Reboot the cluster after removing the patch.

********************************************************************************
