Originated: July 8, 2016
********************************************************************************

Patch-175451: Patch for OneFS 7.1.1.0 - 7.1.1.9, 7.2.0.0 - 7.2.0.5 and 7.2.1.0 
- 7.2.1.3.  This patch installs a new kernel driver for the QLogic NetXtreme 
Ethernet (BXE) 10 GigE network interface card.  In addition, this patch installs 
the necessary utilities to perform a firmware upgrade on the BXE network 
interface card.

--------------------------------------------------------------------------------

This patch can be installed on clusters running the following OneFS versions: 
7.2.1.0 - 7.2.1.3
7.2.0.0 - 7.2.0.5
7.1.1.0 - 7.1.1.9

This patch conflicts with the following patches: 
patch-166632
patch-165448
patch-164644
patch-162062
patch-159992

This patch deprecates the following patches: 
patch-168778 
patch-163030
patch-157789
patch-155678
patch-150883
patch-148695
patch-140498

If any conflicting or deprecated patches are installed on the cluster, you must 
remove them before installing this patch.

********************************************************************************

ENHANCEMENTS

This patch installs a new kernel driver for the QLogic NetXtreme Ethernet (BXE) 
10 GigE network interface card.  

This patch installs the necessary utilities to perform a firmware upgrade on 
the BXE network interface card.

In order to complete the desired changes to the 10GigE network interface 
card, this patch must be installed on the cluster, and then you must upgrade the 
BXE network interface card firmware.  After you install this patch, we recommend 
that you install Node Firmware Package version 9.3.4.  For more information 
about the firmware upgrade, including installation instructions, see the Isilon 
Node Firmware Package Release Notes: https://support.emc.com/docu69984

In addition, this patch may alleviate or reduce the occurrence of the following 
issues: 

Bug ID 146502, 153356
   HD400, NL410, S210, X210 and X410 nodes configured to communicate through a 
   10 GigE network interface card using the QLogic NetXtreme Ethernet (BXE) 
   driver encountered an issue where connectivity over the BXE interface failed 
   which resulted in users being unable to connect.  In addition, warnings 
   similar to the following appeared in the /var/log/messages file on the 
   cluster:
   
   ERROR: DMAE timeout!
   
   or 
   
   ELINK EVENT LOG (3)
      
   or
   
   ERROR: ECORE: timeout waiting for state 0
   ERROR: Failed to delete MACs (-4)
   ERROR: ECORE: timeout waiting for state 7
   ERROR: ECORE: timeout waiting for state 2
   ERROR: FUNC_STOP ramrod failed. Running a dry transaction 
   
*  Bug ID 168265, 168792
   HD400, NL410, S210, X210 and X410 nodes that were configured to communicate
   through a 10 GigE network interface card that was using the QLogic
   NetXtreme Ethernet (BXE) driver which was introduced in OneFS 7.1.1 could
   have encountered an issue where connectivity over the BXE interface failed.
   In addition, a warning similar to the following appeared in the
   /var/log/messages file:

   WARNING: Interface TX queue is full, ignoring transmit request

*  Bug ID 164251, 152089, 152083
   If the cluster contained HD400, NL410, S210, X210 or X410  nodes that had
   QLogic NetExtreme 10 GigE NIC cards and any external network subnets
   connected to the cluster were set to 9000 MTU, an error similar to the
   following appeared in the /var/log/messages file and the affected nodes
   rebooted:
   
   ERROR: mbuf alloc fail for fp[01] rx chain (55)

*  Bug ID 164251, 152089, 152083 
   HD400, NL410, S210, X210 and X410 nodes that were configured to communicate
   through a 10 GigE network interface card that was using the QLogic NetXtreme 
   Ethernet (BXE) and were also configured to use the aggregate interface with
   LACP experienced connectivity issues with those links after rebooting the
   node or re-configuring the MTU on those interfaces.

*  Bug ID 162042, 163447, 163324
   HD400, NL410, S210, X210 and X410 nodes that were configured to communicate 
   through a 10 GigE network interface card that was using the Broadcom 
   NetXtreme Ethernet (BXE) and were also configured with LRO enable are 
   susceptible.  A silent data corruption could occur when transferring data to 
   an Isilon cluster via any network protocol. Corruption manifest itself by 
   applications complaining about invalid data accessed from the cluster. This 
   issue is not readily identifiable, and it does not generate an alert.

*  Bug ID 154456, 154455, 154454 
   HD400, NL410, S210, X210 and X410 nodes that were configured to communicate
   through a 10 GigE network interface card that was using the QLogic NetXtreme 
   Ethernet (BXE) driver which was introduced in OneFS 7.1.1 could have
   encountered an issue where the link reported "no carrier" in the output of
   the ifconfig command. Toggling the interface did not resolve the issue and
   the node had to be rebooted to reestablish the link. 

*  Bug ID 146448, 139997, 138521 
   HD400, NL410, S210, X210 and X410 nodes that were configured to communicate 
   through a 10GigE network interface card that was using the Broadcom
   NetXtreme Ethernet (BXE) driver that was introduced in OneFS 7.1.1 might
   have restarted unexpectedly. If this occurred, a message similar to the
   following appeared in the var/log/messages file:

   Node panicked with Panic Msg: sleeping thread
   0xffffff04692a0000 owns a nonsleepable lock
 
********************************************************************************

CHANGED FILES

* /boot/kernel.amd64/if_bxe.ko.gz
  (MD5 = 17fa5988c7710f0147e76302d423ceb8)

* /boot/loader.conf
  (MD5 = 61dba0d98d73cb02727b314b528eba89)

* /usr/bin/isi_hwtools/bsdfwnx2
  (MD5 = 7a5947b424251a1169c06e99a343ce3b

* /usr/lib/compat/libtcl8.4.so
  (MD5 = 6f0a93aabf2a34bdb7462f1a765c1553)

* /usr/local/lib/python2.6/site-packages/isi/hw/fw/modules/bxe.pyc
  (MD5 = 0a68d4368a1ce4d91c3fd0083ce25fdb)

* /etc/ifs/hw/firmware.xml
  (MD5 = b96e28089ebfa5fffbca92475eaafd42)

* /etc/ifs/hw/assembly.xml
  (MD5 = 3218c4d2702cd960abcf97b1914fbf57)

* /etc/ifs/hw/devices.xml
  (MD5 = a85b1dd322e26756ca554c587b550d20)

********************************************************************************

INSTALLATION/REMOVAL IMPACTS

During installation or removal of this patch, clients and applications that are
connected to the cluster through BXE 10 GigE network interfaces may be 
disconnected. These connections must be reestablished after the patch
installation is complete. 

Because installation or removal of this patch requires that the cluster be 
rebooted, the patch should be installed or removed when cluster use is at a 
minimum, such as during an off-hours maintenance window. 

If you cannot install or remove the patch during a scheduled maintenance window, 
inform users when the reboot will take place and how they will be affected.

********************************************************************************

IMPORTANT!  
VERIFY CLUSTER-FREE SPACE
If there is insufficient free space and you attempt to install or remove a
patch, the process might not complete successfully. An incomplete patch
installation or patch removal can leave the cluster in an inaccessible state.

Before installing or removing a patch, verify that your cluster has enough
free space to perform the procedure. For more information about maintaining
sufficient free space on an Isilon cluster, see the Best Practices Guide for
Maintaining Enough Free Space on Isilon Clusters and Pools:
https://support.emc.com/docu48119.
   
********************************************************************************

REMOVE CONFLICTING AND DEPRECATED PATCHES

If any conflicting or deprecated patches are installed on the cluster, perform
the following procedure to remove them before installing this patch.

IMPORTANT!

Read the INSTALLATION/REMOVAL IMPACTS in the README for the conflicting and 
deprecated patches before performing this procedure.

1. Verify that your cluster has enough free space to remove this patch.

2. Open an SSH connection on any node in the cluster and log in using the root
   account.

   NOTE: If the procedure is being done on a compliance mode cluster, log in
   using the compadmin account.

3. To remove this patch, run the following command, where <ID> is the patch ID 
   of the patch to be removed:
   
   isi pkg delete patch-<ID>

4. To verify that this patch was removed, run the following command:
     
   isi pkg info

5. Confirm that patch-<ID> does not appear in the list of installed packages.

********************************************************************************

INSTALLING THIS PATCH

IMPORTANT!
Read INSTALLATION/REMOVAL IMPACTS before performing this procedure.
   
1. Verify that your cluster has enough free space to install this patch.

2. Open an SSH connection on any node in the cluster and log in by using the
   root account.

   NOTE: If the patch is being installed on a compliance mode cluster, log in
   using the compadmin account.

3. Copy the patch-175451.tgz file to the /ifs/data directory on the cluster.

4. Run the following command to change to the /ifs/data directory:
   
   cd /ifs/data

5. To extract the patch file, run the following command:

   tar -zxvf patch-175451.tgz

6. To install this patch, run the following command:

   isi pkg install patch-175451.tar

7. To verify that this patch is installed, run the following command:

   isi pkg info

8. Confirm that patch-175451 appears in the list of installed packages and the 
   status is set to Installed.

9. If the cluster contains S210, X210, X410, NL410 or HD400 nodes, perform 
   a rolling reboot for those nodes or a simultaneous reboot for all nodes.
   Instructions for how to perform a simultaneous reboot or a rolling reboot
   from the command-line interface appear later in this document.

10. To verify that that the BXE driver was updated, run the following command:

    sysctl dev.bxe.0.version
	
    Information similar to the following appears:
	
    1.78.89.ISILON.01

********************************************************************************

REBOOTING IN SUCCESSION

After the patch is installed, manually reboot each node in succession. 

1. Open an SSH connection on any node in the cluster and log on using the 
   root account.

   NOTE: If the procedure is being done on a compliance mode cluster, log in 
   using the compadmin account.
   
2. Run the following command to start the Isilon IQ configuration console: 
   
   isi config

3. Within the configuration console, reboot any node in the cluster, except the 
   node that you are connected to, by running the following command, where 
   <node> is the node that you want to reboot:

   reboot <node>

4. To exit the configuration console, run the following command:

   exit

5. To monitor the progress of the reboot, run the following command at any time 
   from the node that you are connected to: 
   
   sysctl efs.gmp.group
   
   Output is displayed in the following format:   
   
   efs.gmp.group: <Node,Sequential number>: { <Array IDs>:<Drive IDs> }
      
   For example, 
      
   if node 6 is marked for group removal:
   efs.gmp.group: <2,15>: { 1-5:0-3, shutdown_read_only: 6 }
      
   if node 6 is rebooting:
   efs.gmp.group: <2,15>: { 1-5:0-3, down: 6 }
   
   if node 6 is back online:
   efs.gmp.group: <2,15>: { 1-6:0-3 }
   
   NOTE: This example assumes that only one node at a time is being rebooted and 
   no other devices are down.  The array ID might not match the node number. 
   
   For more information about the efs.gmp.group sysctl, see article 89502 on the 
   EMC Online Support site.
   
6. Wait for the node to successfully reboot.
   
7. Repeat step 3 for each remaining node, except for the node that you are 
   connected to.

8. To exit the configuration console, run the following command:

   exit

9. To reboot the node that you are connected to, run the following command:
   
   shutdown -r now   

   NOTE: Running the command in step 9 will close your SSH connection to the 
   cluster.  This is true only for this step; no other step will disconnect the 
   SSH session. 

********************************************************************************

REBOOTING SIMULTANEOUSLY

After the patch is installed, manually reboot all nodes. 

1. Open an SSH connection on any node in the cluster and log on using the 
   root account.

   NOTE: If the procedure is being done on a compliance mode cluster, log in 
   using the compadmin account.
 
2. Run the following cluster administration command to enter the isi config 
   subsystem:
  
    isi config
 
3. To reboot all the nodes in the cluster, run the following command:
 
    reboot all
 
4. At the prompt, enter yes.

********************************************************************************

REMOVING THIS PATCH

If you need to remove this patch, complete the steps below.

IMPORTANT!
Read INSTALLATION/REMOVAL IMPACTS before performing this procedure.

1. Verify that your cluster has enough free space to remove this patch.

2. Open an SSH connection on any node in the cluster and log in using the
   root account.

   NOTE: If the patch is being removed from a compliance mode cluster, log in
   using the compadmin account.

3. To remove this patch, run the following command:

   isi pkg delete patch-175451

4. To verify that this patch was removed, run the following command:

   isi pkg info

5. Confirm that patch-175451 does not appear in the list of installed packages.

6. If the cluster contains S210, X210, X410, NL410 or HD400 nodes, perform 
   a rolling reboot for those nodes or a simultaneous reboot for all nodes.
   Instructions for how to perform a simultaneous reboot or a rolling reboot
   from the command-line interface appear earlier in this document.

7. To verify that the BXE driver has been downgraded, run the following
   command:
  
   sysctl dev.bxe.0.version
 
8. Confirm that 1.78.89.ISILON.01 does not appear.     

********************************************************************************
