Originated: January 27, 2017
********************************************************************************

Patch-188239: Patch for OneFS 8.0.0.0 - 8.0.0.4. This patch installs a new 
kernel driver for the QLogic NetXtreme Ethernet (BXE) 10 GigE network interface 
card. In addition, this patch installs the necessary utilities to perform a 
firmware upgrade on the BXE network interface card as well as to debug certain
connectivity issues.

--------------------------------------------------------------------------------

This patch can be installed on clusters running the following OneFS versions: 
8.0.0.0 - 8.0.0.4

This patch deprecates the following patches:
patch-182721
patch-175450
patch-171054

If any deprecated patches are installed on the cluster, you must remove them 
before installing this patch.
 
********************************************************************************

ENHANCEMENTS

This patch installs a new kernel driver for the QLogic NetXtreme Ethernet (BXE) 
10 GigE network interface card.  

This patch installs the necessary utilities to perform a firmware upgrade on 
the BXE network interface card.

In order to complete the desired changes to the 10GigE network interface 
card, this patch must be installed on the cluster, and then you must upgrade the 
BXE network interface card firmware.  After you install this patch, we recommend 
that you install Node Firmware Package version 9.3.4.  For more information 
about the firmware upgrade, including installation instructions, see the Isilon 
Node Firmware Package Release Notes: https://support.emc.com/docu69984

In addition, this patch might alleviate or reduce the occurrence of the 
following issues: 

*  Bug ID 185812
   HD400, NL410, S210, X210 and X410 nodes configured to communicate through a 
   10 GigE network interface card using the QLogic NetXtreme Ethernet (BXE) 
   driver might encounter a panic when rebooting while the NIC is actively 
   processing network traffic. In addition, messages similar to the following 
   will appear in the /var/log/messages file: 

   panic @ time 1479296879.182, thread 0xfffff805c4c3e000: Fatal trap 12: page 
   fault while in kernel mode

   Stack: --------------------------------------------------
   kernel:trap_fatal+0xa2
   kernel:trap_pfault+0x25e
   kernel:trap+0x618
   kernel:taskqueue_enqueue_timeout+0x17
   if_bxe.ko:bxe_tx_mq_start_locked+0x371
   if_bxe.ko:bxe_tx_mq_start+0x75
   if_lagg.ko:lagg_transmit+0xbe
   kernel:vlan_transmit+0x121
   kernel:ether_output+0x5a2
   kernel:ip_output+0xe9a
   kernel:udp_send+0x7f0
   kernel:sosend_dgram+0x394
   kernel:kern_sendit+0x1d2
   kernel:sendit+0x115
   kernel:sys_sendto+0x4c
   kernel:amd64_syscall+0x366
   --------------------------------------------------    

*  Bug ID 180883
   HD400, NL410, S210, X210 and X410 nodes configured to communicate through a 
   10 GigE network interface card using the QLogic NetXtreme Ethernet (BXE) 
   driver might encounter an issue where connectivity over the BXE interface 
   fails preventing users from establishing a connection to the cluster. In 
   addition, warnings similar to the following appear in the /var/log/messages 
   file:

   ERROR: DMAE timeout!
   or
   ELINK EVENT LOG (3)
   or
   ERROR: ECORE: timeout waiting for state 0
   ERROR: Failed to delete MACs (-4)
   ERROR: ECORE: timeout waiting for state 7
   ERROR: ECORE: timeout waiting for state 2
   ERROR: FUNC_STOP ramrod failed. Running a dry transaction

*  Bug ID 178930 
   HD400, NL410, S210, X210 and X410 nodes configured to communicate through a 
   10 GigE network interface card using the QLogic NetXtreme Ethernet (BXE) 
   driver encountered an issue where connectivity over the BXE interface failed 
   which resulted in users being unable to connect.  This connection failure was
   encoutered when transferring large files over SMB while TSO was enabled.

*  Bug ID 168265, 168792
   HD400, NL410, S210, X210 and X410 nodes that were configured to communicate
   through a 10 GigE network interface card that was using the QLogic
   NetXtreme Ethernet (BXE) driver which was introduced in OneFS 7.1.1 could
   have encountered an issue where connectivity over the BXE interface failed.
   In addition, a warning similar to the following appeared in the
   /var/log/messages file:

   WARNING: Interface TX queue is full, ignoring transmit request

*  Bug ID 164251, 152089, 152083
   If the cluster contained HD400, NL410, S210, X210 or X410  nodes that had
   QLogic NetExtreme 10 GigE NIC cards and any external network subnets
   connected to the cluster were set to 9000 MTU, an error similar to the
   following appeared in the /var/log/messages file and the affected nodes
   rebooted:
   
   ERROR: mbuf alloc fail for fp[01] rx chain (55)

*  Bug ID 164251, 152089, 152083 
   HD400, NL410, S210, X210 and X410 nodes that were configured to communicate
   through a 10 GigE network interface card that was using the QLogic NetXtreme 
   Ethernet (BXE) and were also configured to use the aggregate interface with
   LACP experienced connectivity issues with those links after rebooting the
   node or re-configuring the MTU on those interfaces.

*  Bug ID 162042, 163447, 163324
   HD400, NL410, S210, X210 and X410 nodes that were configured to communicate 
   through a 10 GigE network interface card that was using the Broadcom 
   NetXtreme Ethernet (BXE) and were also configured with LRO enable are 
   susceptibale a silent data corruption could occur when transferring data to 
   an Isilon cluster via any network protocol. Corruption manifest itself by 
   applications complaining about invalid data accessed from the cluster. This 
   issue is not readily identifiable, and it does not generate an alert.

*  Bug ID 154456, 154455, 154454 
   HD400, NL410, S210, X210 and X410 nodes that were configured to communicate
   through a 10 GigE network interface card that was using the QLogic NetXtreme 
   Ethernet (BXE) driver which was introduced in OneFS 7.1.1 could have
   encountered an issue where the link reported "no carrier" in the output of
   the ifconfig command. Toggling the interface did not resolve the issue and
   the node had to be rebooted to reestablish the link. 
  
********************************************************************************

CHANGED FILES

* /boot/kernel.amd64/if_bxe.ko.gz
  (MD5 = 104a71ee621c8501f0dd60633f76337f)

* /usr/bin/isi_hwtools/bsdfwnx2
  (MD5 = 5084fac217b1374b18484eb3c6d2b1e6)

* /usr/lib/compat/libtcl8.4.so
  (MD5 = 4ee2747a00e5138bf5461e919f528228)

* /usr/local/lib/python2.6/site-packages/isi/hw/fw/modules/bxe.pyc
  (MD5 = 9a923aa07a0123d639a58903ce43c7ad)

* /etc/ifs/hw/firmware.xml
  (MD5 = 6668c45489bdc6c2915f3ccaaba8e2a8)

* /etc/ifs/hw/assembly.xml
  (MD5 = 49b9360abb63772018c646eff91537cb)

* /etc/ifs/hw/devices.xml
  (MD5 = 0e25b9bac0282798eaf148171c138906)

* /usr/bin/isi_hwtools/bxe_grc_monitor.sh
  (MD5 = a85ee50a5e5f75e952d5a737e34c3afb)

* /usr/bin/isi_hwtools/bxedump
  (MD5 = ac5130ed5fa9edc3f1b311029c7160e6)
  
********************************************************************************

INSTALLATION/REMOVAL IMPACTS

During installation or removal of this patch, clients and applications that are
connected to the cluster through BXE 10 GigE network interfaces might be 
disconnected. These connections must be reestablished after the patch
installation is complete. 

Because installation or removal of this patch requires that the cluster be 
rebooted, the patch should be installed or removed when cluster use is at a 
minimum, such as during an off-hours maintenance window. 

If you cannot install or remove the patch during a scheduled maintenance window, 
inform users when the reboot will take place and how they will be affected.

To ensure the changes made by this patch take effect, you must manually 
perform a rolling reboot for all S210, X210, X410, NL410 and HD400 nodes in the 
cluster, or a simultaneous reboot for all nodes at the end of the patch 
installation or removal process.
 
If you perform a rolling reboot, client connections to the cluster over 
protocols that are configured or designed to fail over to another node will not 
be disrupted. However, client connections to the cluster over other 
protocols might be disconnected when a node reboots. Clients that are 
disconnected must manually reestablish a connection to the cluster.  If you 
perform a simultaneous reboot, all of the nodes in the cluster are rebooted 
simultaneously, all client connections to the cluster are disconnected, and 
client connections must be manually reestablished after the patch is installed.

This patch addresses the issue where a node might restart unexpectedly during a 
reboot with active network traffic through the BXE interface. For the patch to 
take effect, a reboot is required after patch installation. During this reboot 
the original issue may be experienced. In order to avoid the unexpected restart 
during reboot, Isilon Engineering recommends that the lagg interface be brought 
down prior to the reboot.  NOTE: You cannot perform a simultaneous reboot
if you are connected to the cluster through the lagg interface.

********************************************************************************

IMPORTANT!  
VERIFY CLUSTER-FREE SPACE
If there is insufficient free space and you attempt to install or remove a
patch, the process might not complete successfully. An incomplete patch
installation or patch removal can leave the cluster in an inaccessible state.

Before installing or removing a patch, verify that your cluster has enough
free space to perform the procedure. For more information about maintaining
sufficient free space on an Isilon cluster, see the Best Practices Guide for
Maintaining Enough Free Space on Isilon Clusters and Pools:
https://support.emc.com/docu48119.

********************************************************************************

REMOVE DEPRECATED PATCHES

If any deprecated patches are installed on the cluster, perform the following 
procedure to remove them before installing this patch.

IMPORTANT!

Read the INSTALLATION/REMOVAL IMPACTS in the README for the deprecated patch 
before performing this procedure.

1. Verify that your cluster has enough free space to remove this patch.

2. Open an SSH connection on any node in the cluster and log in using the root 
   account.

   NOTE: If the procedure is being done on a compliance mode cluster, log in 
   using the compadmin account.

3. To remove this patch, run the following command, where <ID> is the patch ID 
   of the patch to be removed:
   
   isi pkg uninstall patch-<ID> --rolling=false 

   NOTE:  If you run the isi upgrade patches uninstall command without the 
   --rolling=false parameter, the patch will be removed from each node in the 
   cluster in succession and each node will be rebooted when the patch removal 
   is complete.

   Confirm that you want to continue with this operation:
   
   Are you sure?  (yes/[no]): yes

4. To verify that this patch was removed, run the following command:
     
   isi upgrade patches list

5. Confirm that patch-<ID> does not appear in the list of installed packages.

********************************************************************************


INSTALLING THIS PATCH

IMPORTANT!
Read INSTALLATION/REMOVAL IMPACTS before performing this procedure.
   
1. Verify that your cluster has enough free space to install this patch.

2. Open an SSH connection on any node in the cluster and log in by using the
   root account.

   NOTE: If the patch is being installed on a compliance mode cluster, log in
   using the compadmin account.

3. Copy the patch-188239.tgz file to the /ifs/data directory on the cluster.

4. Run the following command to change to the /ifs/data directory:
   
   cd /ifs/data

5. To extract the patch file, run the following command:

   tar -zxvf patch-188239.tgz

6. To install this patch, run the following command:

   isi upgrade patches install patch-188239.pkg

7. To verify that this patch is installed, run the following command:

   isi upgrade patches list

8. Confirm that patch-188239 appears in the list of installed packages and the 
   status is set to Installed.

9. If the cluster contains S210, X210, X410, NL410 or HD400 nodes, manually 
   perform a rolling reboot for those nodes or a simultaneous reboot for all 
   nodes.  Instructions for how to perform a simultaneous reboot or a rolling 
   reboot from the command-line interface appear later in this document.

10. To verify that that the BXE driver was updated, run the following command:

    sysctl dev.bxe.0.version
	
    Information similar to the following appears:
	
    1.78.89.ISILON.05

********************************************************************************

REBOOTING IN SUCCESSION

After the patch is installed, manually reboot each node in succession. 

1. Open an SSH connection on any node in the cluster and log on using the 
   root account.
   
2. Run the following command to confirm the network is using link aggregation:

   isi network interfaces list |grep 10gige-agg

   Output similar to the following is displayed, where "Up" confirms the cable 
   is connected to the switch and both ends are active:   

   1    10gige-agg-1 Up            groupnet0.10gige.10Gige 172.16.205.3

3. Run the following command to confirm the link aggregation is using BXE:

   isi_for_array -s "ifconfig |grep lagg"

   Output similar to the following is displayed, where <clustername> is the name 
   of the cluster and where the link aggregation is using BXE:  

   <clustername>: lagg1: flags=808802<BROADCAST,SIMPLEX,MULTICAST,ISIUSERDOWN> 
   metric 0 mtu 1500
   <clustername>:      laggport: bxe1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>


4. Run the following command to bring the BXE LAGG interfaces down:

   isi_for_array -n <node> "ifconfig lagg1 down"

5. Run the following command to start the Isilon IQ configuration console: 
   
   isi config

6. Within the configuration console, reboot any node in the cluster, except the 
   node that you are connected to, by running the following command, where 
   <node> is the node that you want to reboot:

   reboot <node>

7. To exit the configuration console, run the following command:

   exit

8. To monitor the progress of the reboot, run the following command at any time 
   from the node that you are connected to: 
   
   sysctl efs.gmp.group
   
   Output is displayed in the following format:   
   
   efs.gmp.group: <Node,Sequential number>: { <Array IDs>:<Drive IDs> }
      
   For example, 
      
   if node 6 is marked for group removal:
   efs.gmp.group: <2,15>: { 1-5:0-3, shutdown_read_only: 6 }
      
   if node 6 is rebooting:
   efs.gmp.group: <2,15>: { 1-5:0-3, down: 6 }
   
   if node 6 is back online:
   efs.gmp.group: <2,15>: { 1-6:0-3 }
   
   NOTE: This example assumes that only one node at a time is being rebooted and 
   no other devices are down.  The array ID might not match the node number. 
   
   For more information about the efs.gmp.group sysctl, see article 89502 on the 
   EMC Online Support site.
   
9. Wait for the node to successfully reboot.
   
10. Repeat step 4 to step 9 for each remaining node, except for the node that 
    you are connected to.

11. To exit the configuration console, run the following command:

   exit

12. If you are currently connected through a lagg interface, you will need to 
    switch to a node that has already been updated and repeat steps 4 through 
	9 to complete the installation. Otherwise, continue with the next step.

13. Run the following command if you need to bring down the BXE LAGG interface 
    on the node that you are connected to:

   ifconfig lagg1 down 
   
14. To reboot the node that you are connected to, run the following command:
   
   shutdown -r now   

   NOTE: Running the command in step 14 will close your SSH connection to the 
   cluster.  

********************************************************************************

REBOOTING SIMULTANEOUSLY

After the patch is installed, manually reboot all nodes simultaneously. 

NOTE: You cannot perform a simultaneous reboot if you are connected to the 
cluster through the lagg interface.

1. Open an SSH connection on any node in the cluster and log on using the 
   root account.
 
2. Run the following command to confirm the network is using link aggregation:

   isi network interfaces list |grep 10gige-agg

   Output similar to the following is displayed, where "Up" confirms the cable 
   is connected to the switch and both ends are active:   

   1    10gige-agg-1 Up            groupnet0.10gige.10Gige 172.16.205.3

3. Run the following command to confirm the link aggregation is using BXE:

   isi_for_array -s "ifconfig |grep lagg"

   Output similar to the following is displayed, where <clustername> is the name 
   of the cluster and where the link aggregation is using BXE:  

   <clustername>: lagg1: flags=808802<BROADCAST,SIMPLEX,MULTICAST,ISIUSERDOWN> 
   metric 0 mtu 1500
   <clustername>:      laggport: bxe1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>


4. Run the following command to bring the BXE LAGG interfaces down:

   isi_for_array "ifconfig lagg1 down"
   
5. Run the following cluster administration command to enter the isi config 
   subsystem:
  
    isi config
 
6. To reboot all the nodes in the cluster, run the following command:
 
     reboot all
 
7. At the prompt, enter yes.

********************************************************************************

REMOVING THIS PATCH

If you need to remove this patch, complete the steps below.

IMPORTANT!
Read INSTALLATION/REMOVAL IMPACTS before performing this procedure.

1. Verify that your cluster has enough free space to remove this patch.

2. Open an SSH connection on any node in the cluster and log in using the
   root account.

   NOTE: If the patch is being removed from a compliance mode cluster, log in
   using the compadmin account.

3. To remove this patch, run the following command:

   isi upgrade patches uninstall patch-188239
   
   Confirm that you want to continue with this operation:
   
   Are you sure?  (yes/[no]): yes

4. To verify that this patch was removed, run the following command:

   isi upgrade patches list

5. Confirm that patch-188239 does not appear in the list of installed packages.

6. If the cluster contains S210, X210, X410, NL410 or HD400 nodes, perform 
   a rolling reboot for those nodes or a simultaneous reboot for all nodes.
   Instructions for how to perform a simultaneous reboot or a rolling reboot
   from the command-line interface appear earlier in this document.

7. To verify that the BXE driver has been downgraded, run the following
   command:
  
   sysctl dev.bxe.0.version
 
8. Confirm that 1.78.89.ISILON.05 does not appear.   

********************************************************************************
