Originated: March 24, 2016
********************************************************************************

Patch-168778: Patch for OneFS 7.1.1.0 - 7.1.1.9, 7.2.0.0 - 7.2.0.5 and 7.2.1.0
- 7.2.1.2. This patch installs a new kernel driver for the QLogic (formerly
Broadcom) NetXtreme Ethernet (BXE) 10 GigE NIC.

--------------------------------------------------------------------------------

This patch can be installed on clusters running the following OneFS versions: 
7.2.1.0 - 7.2.1.2
7.2.0.0 - 7.2.0.5
7.1.1.0 - 7.1.1.9

This patch conflicts with the following patches: 
patch-166632
patch-165448
patch-164644
patch-162062
patch-159992

This patch deprecates the following patches: 
patch-163030
patch-157789
patch-155678
patch-150883
patch-148695
patch-140498

********************************************************************************

RESOLVED ISSUES

NOTE:
This patch contains several changes provided by QLogic to address a variety of
driver issues. Some of these fixes may already be present in certain versions
of OneFS. If your cluster is running a version of OneFS that is listed in the
previous section, you should install this patch to avoid serious functional
issues.  

*  Bug ID 168265, 168792
   HD400, NL410, S210, X210 and X410 nodes that were configured to communicate
   through a 10 GigE network interface card that was using the QLogic
   NetXtreme Ethernet (BXE) driver which was introduced in OneFS 7.1.1 could
   have encountered an issue where connectivity over the BXE interface failed.
   In addition, a warning similar to the following appeared in the
   /var/log/messages file:

   WARNING: Interface TX queue is full, ignoring transmit request

*  Bug ID 164251, 152089, 152083
   If the cluster contained HD400, NL410, S210, X210 or X410  nodes that had
   QLogic NetExtreme 10 GigE NIC cards and any external network subnets
   connected to the cluster were set to 9000 MTU, an error similar to the
   following appeared in the /var/log/messages file and the affected nodes
   rebooted:
   
   ERROR: mbuf alloc fail for fp[01] rx chain (55)

*  Bug ID 164251, 152089, 152083 
   HD400, NL410, S210, X210 and X410 nodes that were configured to communicate
   through a 10 GigE network interface card that was using the QLogic NetXtreme 
   Ethernet (BXE) and were also configured to use the aggregate interface with
   LACP experienced connectivity issues with those links after rebooting the
   node or re-configuring the MTU on those interfaces.

*  Bug ID 162042, 163447, 163324
   HD400, NL410, S210, X210 and X410 nodes that were configured to communicate 
   through a 10 GigE network interface card that was using the Broadcom 
   NetXtreme Ethernet (BXE) and were also configured with LRO enable are 
   susceptibale a silent data corruption could occur when transferring data to 
   an Isilon cluster via any network protocol. Corruption manifest itself by 
   applications complaining about invalid data accessed from the cluster. This 
   issue is not readily identifiable, and it does not generate an alert.

*  Bug ID 154456, 154455, 154454 
   HD400, NL410, S210, X210 and X410 nodes that were configured to communicate
   through a 10 GigE network interface card that was using the QLogic NetXtreme 
   Ethernet (BXE) driver which was introduced in OneFS 7.1.1 could have
   encountered an issue where the link reported "no carrier" in the output of
   the ifconfig command. Toggling the interface did not resolve the issue and
   the node had to be rebooted to reestablish the link. 

*  Bug ID 146448, 139997, 138521 
   HD400, NL410, S210, X210 and X410 nodes that were configured to communicate 
   through a 10GigE network interface card that was using the Broadcom
   NetXtreme Ethernet (BXE) driver that was introduced in OneFS 7.1.1 might
   have restarted unexpectedly. If this occurred, a message similar to the
   following appeared in the var/log/messages file:

   Node panicked with Panic Msg: sleeping thread
   0xffffff04692a0000 owns a nonsleepable lock

********************************************************************************

CHANGED FILES

* /boot/kernel.amd64/if_bxe.ko.gz
  (MD5 = 683f9010a02d270e0bded4e0cb2cc8b4)

* /boot/loader.conf
  (MD5 = 61dba0d98d73cb02727b314b528eba89)

********************************************************************************

INSTALLATION/REMOVAL IMPACTS

During installation or removal of this patch, clients and applications that are
connected to the cluster through BXE 10 GigE network interfaces will be 
disconnected. These connections must be reestablished after the patch
installation is complete. 

To ensure the changes made by this patch take effect, you must reboot all nodes
in the cluster at the end of the patch installation process. You must also 
reboot all nodes in the cluster if you remove this patch. You can either reboot 
the nodes in succession (also known as a rolling reboot) or perform a 
simultaneous reboot of all nodes in the cluster.  
 
A simultaneous reboot can be performed from the OneFS web administration
interface or from the command-line interface. A simultaneous reboot of the 
cluster is faster than a rolling reboot; however, because all nodes are 
rebooted simultaneously, this type of reboot causes a temporary 
interruption of service while the nodes reboot.
 
A rolling reboot requires that you reboot each node in succession manually
from the command-line interface. When a node is rebooted, client connections
to that node will be disconnected and might need to be reestablished. 
 
Whether you perform a simultaneous reboot or rolling reboot of the nodes, we 
recommend that you install the patch during an off-hours maintenance window
to minimize the disruption of service to clients.
 
If you cannot install the patch during a scheduled maintenance window, inform
users when the reboot will take place and how they will be affected. 
 
Instructions for how to perform a simultaneous reboot or rolling reboot from
the command-line interface appear later in this document.

********************************************************************************

IMPORTANT!  
VERIFY CLUSTER-FREE SPACE
If there is insufficient free space and you attempt to install or remove a
patch, the process might not complete successfully. An incomplete patch
installation or patch removal can leave the cluster in an inaccessible state.

Before installing or removing a patch, verify that your cluster has enough
free space to perform the procedure. For more information about maintaining
sufficient free space on an Isilon cluster, see the Best Practices Guide for
Maintaining Enough Free Space on Isilon Clusters and Pools:
https://support.emc.com/docu48119.
   
********************************************************************************

REMOVE DEPRECATED PATCHES

If any deprecated patches are installed on the cluster, perform the following
procedure to remove them before installing this patch.

IMPORTANT!

Read the INSTALLATION/REMOVAL IMPACTS in the README for the deprecated patch
before performing this procedure.

1. Verify that your cluster has enough free space to remove this patch.

2. Open an SSH connection on any node in the cluster and log in using the
   root account.

   NOTE: If the procedure is being done on a compliance mode cluster, log in
   using the compadmin account.

3. To remove this patch, run the following command, where <ID> is the patch
   ID of the patch to be removed:
   
   isi pkg delete patch-<ID>

4. To verify that this patch was removed, run the following command:
     
   isi pkg info

5. Confirm that patch-<ID> does not appear in the list of installed packages.

********************************************************************************

INSTALLING THIS PATCH

IMPORTANT!
Read INSTALLATION/REMOVAL IMPACTS before performing this procedure.
   
1. Verify that your cluster has enough free space to install this patch.

2. Open an SSH connection on any node in the cluster and log in by using the
   root account.

   NOTE: If the patch is being installed on a compliance mode cluster, log in
   using the compadmin account.

3. Copy the patch-168778.tgz file to the /ifs/data directory on the cluster.

4. Run the following command to change to the /ifs/data directory:
   
   cd /ifs/data

5. To extract the patch file, run the following command:

   tar -zxvf patch-168778.tgz

6. To install this patch, run the following command:

   isi pkg install patch-168778.tar

7. To verify that this patch is installed, run the following command:
  
   isi pkg info

8. Confirm that patch-168778 appears in the list of installed packages.

9. Reboot the cluster by performing one of the procedures that follow.

********************************************************************************

PERFORMING A SIMULTANEOUS REBOOT

After the patch is installed, manually reboot all nodes. 

1. Open an SSH connection on any node in the cluster and log on using the 
   root account.

   NOTE: If the procedure is being done on a compliance mode cluster, log in
   using the compadmin account.
 
2. Run the following cluster administration command to enter the isi config
   subsystem:
  
   isi config
 
3. To reboot all the nodes in the cluster, run the following command:
 
   reboot all
 
4. At the prompt, enter yes.


********************************************************************************

PERFORMING A ROLLING REBOOT

After the patch is installed, manually reboot each node in succession. 

1. Open an SSH connection on any node in the cluster and log on using the 
   root account.

   NOTE: If the procedure is being done on a compliance mode cluster, log in 
   using the compadmin account.
   
2. Run the following command to start the Isilon IQ configuration console: 
   
   isi config

3. Within the configuration console, reboot any node in the cluster, except the 
   node that you are connected to, by running the following command, where 
   <node> is the node that you want to reboot:

   reboot <node>

4. To exit the configuration console, run the following command:

   exit

5. To monitor the progress of the reboot, run the following command at any time 
   from the node that you are connected to: 
   
   sysctl efs.gmp.group
   
   Output is displayed in the following format:   
   
   efs.gmp.group: <Node,Sequential number>: { <Array IDs>:<Drive IDs> }
      
   For example, 
      
   if node 6 is marked for group removal:
   efs.gmp.group: <2,15>: { 1-5:0-3, shutdown_read_only: 6 }
      
   if node 6 is rebooting:
   efs.gmp.group: <2,15>: { 1-5:0-3, down: 6 }
   
   if node 6 is back online:
   efs.gmp.group: <2,15>: { 1-6:0-3 }
   
   NOTE: This example assumes that only one node at a time is being rebooted and 
   no other devices are down.  The array ID might not match the node number. 
   
   For more information about the efs.gmp.group sysctl, see article 89502 on the 
   EMC Online Support site.
   
6. Wait for the node to successfully reboot.
   
7. Repeat step 3 for each remaining node, except for the node that you are 
   connected to.

8. To exit the configuration console, run the following command:

   exit

9. To reboot the node that you are connected to, run the following command:
   
   shutdown -r now   

   NOTE: Running the command in step 6 will close your SSH connection to the 
   cluster.  This is true only for this step; no other step will disconnect the 
   SSH session. 

********************************************************************************

REMOVING THIS PATCH

If you need to remove this patch, complete the steps below.

IMPORTANT!
Read INSTALLATION/REMOVAL IMPACTS before performing this procedure.

1. Verify that your cluster has enough free space to remove this patch.

2. Open an SSH connection on any node in the cluster and log in using the
   root account.

   NOTE: If the patch is being removed from a compliance mode cluster, log in
   using the compadmin account.

3. To remove this patch, run the following command:

   isi pkg delete patch-168778

4. To verify that this patch was removed, run the following command:

   isi pkg info

5. Confirm that patch-168778 does not appear in the list of installed packages.

6. Reboot the cluster after removing the patch.

********************************************************************************
