Oracle Corp. and/or its affiliates.

Diagnostic Assistant: Exadata Main Menu - Documentation

main - Diagnostic Assistant: Exadata Main Menu
   
  The Oracle Diagnostic Assistant Exadata Main Menu enables you to collect
  diagnostics based on a specific issue. Exadata system components have been
  grouped by functional categories below and includes access to Exachk.
    
  Select one of the following options:
    (1) Database
    (2) Exadata - Cell
    (3) Exadata - Hardware
    (4) Network
    (5) Oracle Errors
    (6) Processes
    (7) RAC
    
    ----------Generic Diagnostic Tools ----------
    (11) Exachk

    (H)elp    (E)xit  
   
   

[+] Toggle Help

Showing Menu Help:
  Oracle Diagnostic Assistant Exadata Main Menu Help 
     
  The Exadata Main Menu enables you to diagnose your Exadata environment.
   
  Explanation of available options:
     
    (1) Database
        o Select this option if you suspect a database issue
 
    (2) Exadata - Cell
        o Select this option if you suspect a cell failure, performance
          issue, or patching problem

    (3) Exadata - Hardware
        o Select this option if you suspect underperforming flash drives
          or failed drives on Exadata Cell or Compute Node

    (4) Network
        o Select this option if you suspect issues with the listener, Network
          cabling, IB Switch or any other Network related problem 

    (5) Oracle Errors
        o Select this option to identify and package incidents
 
    (6) Processes
        o Currently the only option available in this section is to analyze
          a process you suspect is spinning/hanging using procwatcher
        o It is important to note that ProcWatcher is NOT delivered with 
          the Diagnostic Assistant

    (7) RAC
        o Select this option if you suspect issues with the RAC instance or
          node evictions or you encounter clusterware startup issues
          
    Generic Diagnostic Tools 
    
    This section gives direct access to diagnostic tools that are NOT
    delivered with the Diagnsotic Assistant, but are commonly used to
    troubleshoot issues related to the menu target.
    
    (11) Exachk Collection
        o Access to the Exachk diagnostic tool
  
   

Menu Option: 1  -  [+] Show Menu Details
Menu Option: 2  -  [+] Show Menu Details
Menu Option: 3  -  [+] Show Menu Details
Menu Option: 4  -  [+] Show Menu Details
Menu Option: 5  -  [+] Show Menu Details
Menu Option: 6  -  [+] Show Menu Details
Menu Option: 7  -  [+] Show Menu Details
Menu Option: 11  -  [+] Show Menu Details
Top | Back

DaExaDB - Database Issues
  
  Use this menu if you suspect you are having database problems.
  
  Select one of the following options:
    (1) Oracle Errors 
    (2) Database HANG
    (3) Database / Instance Crash 
    (4) Lost Write
    (5) ASM Issues and Errors
          
    (M)ain    (B)ack    (H)elp    (E)xit
   

[+] Toggle Help

Showing Menu Help:
  Exadata: Database Issues Help

  Explanation of available options:
 
  Diagnosis is currently available for the following database issues:
      
    (1) Oracle Errors 
        o Selecting this option will allow you to identify the following
          Oracle Error incidents and package the incidents
            o ORA-600 Internal Errors
            o ORA-1578 ORACLE data block corrupted
            o ORA-7445 exception encountered: core dump 
            o ORA-3113 end-of-file on communication channel
            o ORA-4031 unable to allocate string bytes of shared memory
  
    (2) Database HANG
        o Selecting this option will execute the Exadata_DatabaseHang RDA 
          profile which includes the following modules:
            o RACD (performs a database hang analysis)
            o PERF (collects performance information)
            o LOG (collects database trace and log information)
 
    (3) Database / Instance Crash
        o Selecting this option will execute the following RDA module:
            o LOG (collects database trace and log information)  
 
    (4) Lost Write
        o Select this option to collect the controlfile and redohdr dump data
 
    (5) ASM Issues and Errors 
        o Selecting this option will execute the following RDA module:
            o ASM (collects Automatic Storage Management Information)
   
   

Menu Option: 1  -  [+] Show Menu Details
Menu Option: 2  -  [+] Show Command Details
Menu Option: 3  -  [+] Show Menu Details
Menu Option: 4  -  [+] Show Command Details
Menu Option: 5  -  [+] Show Command Details
Menu Option: M  -  [+] Show Menu Details
Menu Option: B  -  [+] Show Menu Details
Top | Back

DaExaDBOraErr - Database: Oracle Errors
  

  For Oracle Errors, a 2 step process is needed to gather the incident
  details.  Option 1 will identify the incidents and option 2 will create
  the incident package.
   
  Select one of the following options:
    (1) View incidents, limited by number
    (2) Create incident package
          
    (M)ain    (B)ack    (H)elp    (E)xit
   

[+] Toggle Help

Showing Menu Help:
  Database Issues: Oracle Errors Help

  Select option 1 to find the incident number needed for option 2.

  Explanation of available options:
    
    (1) Show incident, limited by number
        o You will be prompted for MODE
          --> The 'MODE' of output includes BASIC, BRIEF or DETAIL where
              each option provides a different level of detail
        o you will be prompted for the number of incidents you wish to view
          --> default is 50

    (2) Create package
        o incident id is required
        o this option will create a logical incident package

   

Menu Option: 1  -  [+] Show Command Details
Menu Option: 2  -  [+] Show Command Details
Menu Option: M  -  [+] Show Menu Details
Menu Option: B  -  [+] Show Menu Details
Top | Back

DaExaDBCrash - Database / Instance Crash
  

  RDA will execute the following module=LOG to help diagnose your 
  database orinstance crash.
   
  Select one of the following options:
    (1) Run A New Collection
    (2) View Existing Collection
          
    (M)ain    (B)ack    (H)elp    (E)xit
   

[+] Toggle Help

Showing Menu Help:
  Database Issues: Database or Instance Crash Help

  Explanation of available options:

    (1) Run A New Collection
        o RDA will execute the following module:
            - LOG (collects database trace and log information)
  
    (2) View Existing Collection
        o Selecting this option will allow for the previous like  
          collection to be displayed
        o If no previous collection exists, it will be executed and 
          results displayed and next time, this file will be available
          for viewing

   

Menu Option: 1  -  [+] Show Command Details
Menu Option: 2  -  [+] Show Command Details
Menu Option: M  -  [+] Show Menu Details
Menu Option: B  -  [+] Show Menu Details
Top | Back

DaExaCell - exadata - Cell Issues
  Select one of the following options:
    (1) Failed Drives on Exadata Cell or Compute Node 
    (2) Exadata Cell Failure                              >15 mins
    (3) Cell Patching
    (4) Sick or underperforming Cell / Performance Issue  >15 mins
    (5) Long Brownout Due to Cell Failure                 >15 mins       
    (6) ILOM issues 

    (M)ain    (B)ack    (H)elp    (E)xit  
   

[+] Toggle Help

Showing Menu Help:
  Exadata - Cell Issues Help

  Explanation of available options:

    This option enables the following Exadata modules to be selected 
    for execution:
      
    (1) Failed Drives on Exadata Cell or Compute Node 
        o The 'Exadata_FailedDrives' RDA profile will execute
  
    (2) Exadata Cell Failure
        o The 'Exadata_SickCell' RDA profile will execute
        o This collection may take over 15 minutes to complete
 
    (3) Cell Patching
        o Collects important patching and validataion logs used for 
          identifying potential patching issues
 
    (4) Sick or underperforming Cell / Performance Issue
        o The 'Exadata_SickCell' RDA profile will execute
        o You will be prompted to enter a specific cell name/IP or 
          hit enter to take the default of all cells
        o This collection may take over 15 minutes to complete
          
    (5) Long Brownout Due to Cell Failure
        o The 'Exadata_Assessment' RDA profile will execute
        o This collection may take over 15 minutes to complete
          
    (6) ILOM issues 
        o Generates ILOM Snapshot
        o Collects ILOM Data
 
   

Menu Option: 1  -  [+] Show Command Details
Menu Option: 2  -  [+] Show Command Details
Menu Option: 3  -  [+] Show Menu Details
Menu Option: 4  -  [+] Show Command Details
Menu Option: 5  -  [+] Show Command Details
Menu Option: 6  -  [+] Show Menu Details
Menu Option: M  -  [+] Show Menu Details
Menu Option: B  -  [+] Show Menu Details
Top | Back

DaExaCellPatch - Cell Patching Issues
  Required Cell Patching Validation logs.  
     
  Select one of the following options:
    (1) Remote Cell/Compute Node Patching Logs
    (2) Database Node Patching Logs
    (3) Patching Logs using ILOM

    (M)ain    (B)ack    (H)elp    (E)xit  
   

[+] Toggle Help

Showing Menu Help:
  Exadata - Cell Issues: Cell Patching Issues Help
 
  This menu includes options to collect important patching and validation 
  logs. Options 1 and 2 will collect cell-side and database-side data and
  and generally run as a 2 step process. Option 3 - Collect Patching Logs 
  using ILOM, can be used when the cell is not accessible but the ILOM can 
  be accessed.
      
  Explanation of available options:

    (1) Collect Remote Cell/Compute Node Patching Logs
        o This option requires "root" user credentials and captures 
          these files from the cell side:
            /var/log/cellos/CheckHWnFWProfile.log
            /var/log/cellos/validations.log
            /var/log/cellos/checkconfigs_configuration_summaries.txt
            /var/log/cellos/validations/upgradecbusb.log
            /var/log/cellos/patch.out.place.log
            /root/_patch_hctap_/_p_/wait_out 
            /root/_patch_hctap_/ all .log and .txt files
            /boot/ all .log and .txt files
            imageinfo
            ipmitool event logs 
            /opt/oracle.SupportTools/CheckHWnFWProfile hardware and 
                firmware checks
            /var/log/messages
            /SP/console/history
            /root/_patch_hctap_/ all .log and .txt files
            /boot/ all .log and .txt files
            imageinfo
            ipmitool event logs 
            /opt/oracle.SupportTools/CheckHWnFWProfile hardware and 
                firmware checks
            /var/log/messages
            /SP/console/history
      
    (2) Database Node Patching Logs
        o This option captures patching logs from the node location 
          of where the patchmgr was last run
        o Root user credentials are required
        o Files collected: 
            {PATCHMGR_DIR}/patchmgr.stderr
            {PATCHMGR_DIR}/patchmgr.stderr
            {PATCHMGR_DIR}/patchmgr_output
            {PATCHMGR_DIR}/.../...logs
            
    (3) Collect Patching Logs using ILOM
        o Select this option when the cell is not accessible but 
          the ILOM can be accessed
        o Root user credentials are required
         
   

Menu Option: 1  -  [+] Show Command Details
Menu Option: 2  -  [+] Show Command Details
Menu Option: 3  -  [+] Show Command Details
Menu Option: M  -  [+] Show Menu Details
Menu Option: B  -  [+] Show Menu Details
Top | Back

DaExaCellILOM - ILOM Issues
  This menu will help diagnose your ILOM issues.  Select option 1
  to generate the snapshot followed by option 2 to collect option
  1 data.

  Select one of the following options:
    (1) Generate ILOM Snapshot
    (2) Collect ILOM Data

    (M)ain    (B)ack    (H)elp    (E)xit  
   

[+] Toggle Help

Showing Menu Help:
  Exadata - Cell Issues: ILOM Issues Help

  Explanation of available options:

    (1) Generate ILOM Snapshot
        o Select this option to view instructions to manually generate
          the ILOM Snapshot for a cell
        o This process requires root access to the cell

    (2) Collect ILOM Data
        o Select this option to collect the ILOM Snapshot data generated 
          by step 1
        o User will be prompted for the directory path and the node name
        o The snapshot data will be included with all the other data
          captured in the session zip file

   

Menu Option: 1  -  [+] Show Menu Details
Menu Option: 2  -  [+] Show Command Details
Menu Option: M  -  [+] Show Menu Details
Menu Option: B  -  [+] Show Menu Details
Top | Back

DAexaCellILOMGen - Generate ILOM Snapshot
  You must run the ILOM Snapshot outside the Diagnostic Assistant
  at this time. 
  Please perform the following steps in order given.
  Detailed instructions for each step are available in (H)elp.

    1. ssh to [node name]-ilom and enter the password
    2. Confirm header details
    3. Run command: set /SP/diag/snapshot dataset=normal
    4. Confirm response
    5. Run set command to set output directory
    6. Run command: cd /SP/diag/snapshot
    7. Type command: show
    8. Verify the status under properties is 'Running'
    9. When Snapshot Complete, end ILOM session

    (M)ain    (B)ack    (H)elp    (E)xit
   

[+] Toggle Help

Showing Menu Help:
  ILOM Issues: Generate ILOM Snapshot Help

  Explanation of available options:
    1 ssh to the [node name]-ilom and enter the password at the prompt
       example: ssh cell01-ilom
                password
    2 You should see a header similar to the following output:
        Sun(TM) Integrated Lights Out Manager
        Version 3.0.9.19.a r55943
        Copyright 2009 Sun Microsystems, Inc. All rights reserved.
        Use is subject to license terms.

    3 After the prompt,type this command: 
        set /SP/diag/snapshot dataset=normal

    4 You will get a response of: 
        Set 'dataset' to 'normal'

    5 After the prompt, type this command:
        set /SP/diag/snapshot dump_uri=sftp://root:[password]@
          [IP address of host]/[directory to upload snapshot to]
        ex: set /SP/diag/snapshot dump_uri=sftp://root:
            passwd@192.168.1.7/tmp
 
    6 After the prompt, type this command: 
        cd /SP/diag/snapshot

    7 Type command: show

    8 Verify that the status shown under properties is 'Running'
        example: show
                 /SP/diag/snapshot
                 Targets:
                 Properties:
                    dataset = normal
                    dump_uri = (Cannot show property)
                    encrypt_output = false
                    result = Running
 
    9 Continue to enter the "show" command until these lines 
      appear under Properties, indicating the process is complete:
        result = Collecting data into [sftp location from step 5]
        Snapshot Complete.
        Done.

   10 To end the ILOM session, type command: exit

    See My Oracle Support Knowledge Article 1062544.1: Diagnostic 
    information for ILOM, ILO , LO100 issues (https://support.oracle.com/
    CSP/main/article?cmd=show&type=NOT&id=1062544.1) for additional 
    information.

   

Menu Option: M  -  [+] Show Menu Details
Menu Option: B  -  [+] Show Menu Details
Top | Back

DaExaHW - exadata - Hardware Issues
          
  Select one of the following options:
    (1) Underperforming flash drive
    (2) Failed Drives on Exadata Cell or Compute Node
    (3) LSI MegaRAID Diagnostics
      
    (M)ain    (B)ack    (H)elp    (E)xit  
   

[+] Toggle Help

Showing Menu Help:
  Exadata: Exadata - Hardware Issues Help

  Explanation of available options:
      
   (1) Underperforming flash drive
       o The 'Exadata_FlashDrives' RDA profile will execute
   
   (2) Failed Drives on Exadata Cell or Compute Node
       o The 'Exadata_FailedDrives' RDA profile will execute

   (3) LSI MegaRAID Diagnostics
       o The following diagnostics run various MegaCli64 Commands:
            /opt/MegaRAID/MegaCli/MegaCli64 -h

   

Menu Option: 1  -  [+] Show Command Details
Menu Option: 2  -  [+] Show Command Details
Menu Option: 3  -  [+] Show Menu Details
Menu Option: M  -  [+] Show Menu Details
Menu Option: B  -  [+] Show Menu Details
Top | Back

DaExaHWMegaRAID - LSI MegaRAID
          
  Your LSI MegaRAID is about to be diagnosed.

  Select one of the following options:
    (1) Status of the battery(s)
    (2) Status of the Virtual Drive(s)
    (3) Status of the cache in the dbnodes

    (M)ain    (B)ack    (H)elp    (E)xit  
   

[+] Toggle Help

Showing Menu Help:
  Exadata - Hardware Issues LSI MegaRAID Help
  
  The following diagnostics run various MegaCli64 Commands:
        /opt/MegaRAID/MegaCli/MegaCli64 -h
          
  Explanation of available options:

   (1) Provides information about LSI battery state
        MegaCli64 -AdpBbuCmd ?aALL

   (2) Provides information about Virtual Drive(s)
        MegaCli64 -LDInfo -Lall -aAll 

   (3) Verify the state of cache in the dbnodes 
        MegaCli64 -CfgDsply -a0 |grep -i writethrough
          
   

Menu Option: 1  -  [+] Show Command Details
Menu Option: 2  -  [+] Show Command Details
Menu Option: 3  -  [+] Show Command Details
Menu Option: M  -  [+] Show Menu Details
Menu Option: B  -  [+] Show Menu Details
Top | Back

DaExaNetwork - Network Issues
  Select one of the following options:
    (1) Listener Hang/Performance
    (2) Network cabling issue
    (3) IB Switch Issues
    (4) General Network Issues
      
    (M)ain    (B)ack    (H)elp    (E)xit  
   

[+] Toggle Help

Showing Menu Help:
  Exadata: Network Issues Help
    
  Explanation of available options:

    (1) Listener Hang/Performance
        o The 'ONET' RDA module will execute
  
    (2) Network cabling issue
        o The 'Exadata_NetworkCabling' RDA profile will execute

    (3) IB Switch Issues
        o The 'Exadata_IbSwitch' RDA profile will be presented to
          execute with or without root level data
  
    (4) General Network Issues
        o The 'NET' RDA module will execute
            
   

Menu Option: 1  -  [+] Show Command Details
Menu Option: 2  -  [+] Show Command Details
Menu Option: 3  -  [+] Show Menu Details
Menu Option: 4  -  [+] Show Command Details
Menu Option: M  -  [+] Show Menu Details
Menu Option: B  -  [+] Show Menu Details
Top | Back

NetworkIBS - IB Switch
          
  Your IB Switch is about to be diagnosed.

  Select one of the following options:
    (1) Run A New Collection
    (2) Add root level data to the collection

    (M)ain    (B)ack    (H)elp    (E)xit  
   

[+] Toggle Help

Showing Menu Help:
  Network: IB Switch Help

  Explanation of available options:

    (1) Run A New Collection
        o The 'Exadata_IbSwitch' RDA profile will execute without 
          root level data

    (2) Add root level data to the collection
        o The 'Exadata_IbSwitch' RDA profile will execute
        o The following root level data will be included:
            - root access data must be provided
            - root data can only be added to an existing collection
            - option 1 must be run at least once before this command
              will execute successfully

   

Menu Option: 1  -  [+] Show Command Details
Menu Option: 2  -  [+] Show Command Details
Menu Option: M  -  [+] Show Menu Details
Menu Option: B  -  [+] Show Menu Details
Top | Back

DAexaOraErr - Oracle Errors
  

  For Oracle Errors, a 2 step process is needed to gather the incident
  details.  Option 1 will identify the incidents and option 2 will create
  the incident package.
   
  Select one of the following options:
    (1) View incidents, limited by number
    (2) Create incident package
          
    (M)ain    (B)ack    (H)elp    (E)xit
   

[+] Toggle Help

Showing Menu Help:
  Exadata Detailed Diagnostics: Oracle Errors Help

    Select option 1 to find the incident number needed for option 2.

    Explanation of available options:
    
      (1) Show incident, limited by number
          o You will be prompted for MODE
            - The 'MODE' of output includes BASIC, BRIEF or DETAIL where
              each option provides a different level of detail
          o you will be prompted for the number of incidents you wish to view
            - default is 50

      (2) Create package
          o incident id is required
          o this option will create a logical incident package

   

Menu Option: 1  -  [+] Show Command Details
Menu Option: 2  -  [+] Show Command Details
Menu Option: M  -  [+] Show Menu Details
Menu Option: B  -  [+] Show Menu Details
Top | Back

DAexaProcess - Process Issues
          
  Process issues will be diagnosed here.  
   
  Select one of the following options:
    (1) Process Spins and Hangs with Procwatcher
          
    (M)ain    (B)ack    (H)elp    (E)xit  
   

[+] Toggle Help

Showing Menu Help:
  Exadata Detailed Diagnostics: Process Issues Help

  Explanation of available options:

    (1) Process Spins and Hangs with Procwatcher
        o Procwatcher is a tool to examine and monitor Oracle database 
          and/or clusterware processes at an interval
        o for issues or questions encountered while using this option, 
          please refer to My Oracle Support Knowledge Article 459694.1: 
          Script to Monitor and Examine Oracle DB and Clusterware Processes
          (https://support.oracle.com/CSP/main/article?cmd=show&type= 
          NOT&id=459694.1) for assistance

   

Menu Option: 1  -  [+] Show Menu Details
Menu Option: M  -  [+] Show Menu Details
Menu Option: B  -  [+] Show Menu Details
Top | Back

Procwatch - Procwatcher
          
  Procwatcher is an external tool to examine and monitor Oracle 
  database and/or clusterware processes at an interval.

  If you experience problems with Procwatcher, please contact the owner 
  of My Oracle Support Knowledge Article 459694.1: Script to Monitor and 
  Examine Oracle DB and Clusterware Processes (https://support.oracle.com/
  CSP/main/article?cmd=show&type=NOT&id=459694.1) for assistance. 
   
  Select one of the following options:
    (1) Procwatcher status
    (2) Start Procwatcher
    (3) Stop Procwatcher
    (4) Package Procwatcher files
      
    (M)ain    (B)ack    (H)elp    (E)xit  
   

[+] Toggle Help

Showing Menu Help:
  Exadata Process Issues: Procwatcher Help
    
  Procwatcher is an external tool to examine and monitor Oracle 
  database and/or clusterware processes at an interval.

  If you experience problems with Procwatcher, please contact the owner 
  of My Oracle Support Knowledge Article 459694.1 - Procwatcher: Script 
  to Monitor and Examine Oracle DB and Clusterware Processes 
  (https://support.oracle.com/CSP/main/article?cmd=show&type=
  NOT&id=1070954.1)for assistance. 

  The following options will prompt for the directory path where 
  Procwatcher is installed.  Once prompted, the value is stored in 
  config/user.properties: DA_PROCWATCHER_DIR and you will not be 
  prompted again.
      
  Explanation of available options:
          
    (1) Procwatcher status
    (2) Start Procwatcher
    (3) Stop Procwatcher
    (4) Package Procwatcher files

    Additionally, if Procwatcher is deployed, all output will go to 
    GRID_HOME/log/procwatcher but it is possible to run prw.sh standalone  
    from any directory if it is not deployed.

    If installed as standalone, and if prw finds clusterware active, 
    the output will go to GRID_HOME/log/procwatcher

   

Menu Option: 1  -  [+] Show Command Details
Menu Option: 2  -  [+] Show Command Details
Menu Option: 3  -  [+] Show Command Details
Menu Option: 4  -  [+] Show Command Details
Menu Option: M  -  [+] Show Menu Details
Menu Option: B  -  [+] Show Menu Details
Top | Back

DAexaRAC - RAC Issues
   
  This menu targets RCA issues.
  
  Select one of the following options:
    (1) RAC Instance or node Evictions
    (2) Clusterware Startup Problems
          
    (M)ain    (B)ack    (H)elp    (E)xit  
   

[+] Toggle Help

Showing Menu Help:
  Exadata Detailed Diagnostics: RAC Issues Help
  
  Explanation of available options:

    (1) RAC Instance or node Evictions
        o The 'Exadata_RacInstance' RDA profile will execute
          This profile includes the following modules:
            o OS 
            o LOG 
            o PERF 
            o RAC 
            o ASM 
            o EXA

    (2) Clusterware Startup Problems
        o This command collects Clusterware related log files
        o You will be prompted for
            - GRID_HOME path
            - Date you are concerned about in YYY-MM-DD format
        o This command collects Clusterware related log files:
            - Current crs resource state
                \$GRID_HOME/bin/crsctl stat res -t -init
            - Clusterware log files by specific date:
                $GRID_HOME/log/{hostname}/*.log
                \$GRID_HOME/log/{hostname}/ohasd/ohasd.log
                \$GRID_HOME/log/{hostname}/agent/ohasd/oraagent_\$USER/
                    oraagent_\$USER.log
                \$GRID_HOME/log/{hostname}/agent/ohasd/orarootagent_root/
                    orarootagent_root.log
                \$GRID_HOME/log/{hostname}/cssd/ocssd.log
                \$GRID_HOME/log/{hostname}/crsd/crsd.log
                \$GRID_HOME/log/{hostname}/agent/crsd/oraagent_\$USER/
                    oraagent_\$USER.log
                \$GRID_HOME/log/{hostname}/agent/crsd/orarootagent_root/
                    orarootagent_root.log
                \$GRID_HOME/log/{hostname}/diskmon/diskmon.log
                \$GRID_HOME/log/{hostname}/gpnpd/gpnpd.log
  
   

Menu Option: 1  -  [+] Show Command Details
Menu Option: 2  -  [+] Show Command Details
Menu Option: M  -  [+] Show Menu Details
Menu Option: B  -  [+] Show Menu Details
Top | Back

DAexaChk - exachk Collection
  Exachk is designed to audit various important configuration settings
  within an Exadata System - Database Servers, Storage Servers, etc.
  Exachk MUST already be installed as this tool is NOT delivered 
  with the Diagnostic Assistant.

  Select from the following options:
    (1) Verify Exachk Configuration
    (2) Run Exachk (see Help for pre-requisites)
    (3) Run Exachk Without Root Privilege (see Help for pre-requisites)
    (4) Collect Data After Manual DAexaChk Run

    (M)ain    (B)ack    (H)elp    (E)xit
   

[+] Toggle Help

Showing Menu Help:
  Exadata Detailed Diagnostics: Exachk Help

  
  Explanation of available options:

    (1) Verify Exachk Configuration
        o Select this option to see the pre-requisite configuration steps
          that have to be performed to run the Exachk collection from the
          Diagnostic Assistant

    (2) Run Exachk
        o Select this option to run Exachk in non-interactive mode 
          including root privileged checks
            - root privileged checks on database servers will be executed
            - root privileged checks on storage servers will be executed
            - NO Infiniband Switch checks will be performed

          Pre-requisite configuration steps 1-5 must have been performed
          and verified before selecting this collection

    (3) Run Exachk Without Root Privileges
        o Select this option to run Exachk in non-interactive mode without
          root privileged checks
            - No root privileged checks on database servers will be executed
            - No root privileged checks on storage servers will be executed
            - NO Infiniband Switch checks will be performed

    Pre-requisite configuration steps 1-3 must have been performed
    and verified before selecting option 4 collection.

    (4) Collect Data After ManualDAexaChk Run
        o Select this option to collect data from an Exachk run  
          performed outside of the Diagnostic Assistant (i.e. manually 
          or from another tool)
        o If you have performed option 2 or 3, you do not need to select 
          this option since the Exachk data will have been automatically 
          collected

   

Menu Option: 1  -  [+] Show Menu Details
Menu Option: 2  -  [+] Show Command Details
Menu Option: 3  -  [+] Show Command Details
Menu Option: 4  -  [+] Show Command Details
Menu Option: M  -  [+] Show Menu Details
Menu Option: B  -  [+] Show Menu Details
Top | Back

DAexaChkVerify - Verify Exachk Configuration
  To run the Exachk collection from the Diagnostic Assistant,
  please verify the following pre-requisite configuration steps
  have been performed.
  Detailed instructions for each step are available in (H)elp.

    1 configure ssh user equivalence for rdbms software owner (database)
    2 configure ssh user equivalence for rdbms software owner (storage)
    3 Set the environment variable RAT_CELL_SSH_USER
    4 update the sudoers file on each database server
    5 update the sudoers file on each storage server

    (M)ain    (B)ack    (H)elp    (E)xit
   

[+] Toggle Help

Showing Menu Help:
  Exachk: Steps to Verify Exachk Configuration Help

    Instructions and usage documented in the following:
      o Knowledge Article 1070954.1: Oracle Exadata Database Machine Exachk  
        or HealthCheck (https://support.oracle.com/CSP/main/
        article?cmd=show&type=NOT&id=1070954.1) 
    
    (1) configure ssh user equivalence for rdbms software owner
        Example: oracle  on execution database server to oracle user on
                 all other database servers. 
        o To validate proper ssh user equivalence configuration, login 
          as the RDBMS software owner (eg. oracle) and execute the 
          following commands:
            > ssh -o NumberOfPasswordPrompts=0 -o StrictHostKeyChecking=no
              -l oracle dbServerName "echo \"oracle user equivalence is
              setup correctly\""
            where oracle is the RDBMS software owner and dbServerName
            is the database server hostname
        o Repeat for each database server
        o If "Permission denied (publickey,gssapi-with-mic,password)."
          is returned then the ssh user equivalence is not properly 
          configured
             
    (2) configure ssh user equivalence for rdbms software owner
        (eg., oracle) from execution node to cellmonitor user on all
        storage servers. 
        o To validate proper ssh user equivalence configuration login as 
          the RDBMS software owner (eg., oracle) and execute the 
          following commands:
            >\$ ssh -o NumberOfPasswordPrompts=0 -o StrictHostKeyChecking=no
             -l cellmonitor storageServerName "echo \"cellmonitor user
             equivalence is setup correctly\""
            where storageServerName is the storage server hostname.
        o Repeat for each storage server
        o If "Permission denied (publickey,gssapi-with-mic,password)."
          is returned then the ssh user equivalence is not properly 
          configured
             
    (3) Set the environment variable RAT_CELL_SSH_USER to cellmonitor in
        the execution environment of the rdbms software owner (eg oracle)
        # export RAT_CELL_SSH_USER=cellmonitor

    (4) update the sudoers file on each database server
        using visudo command:
            >oracle ALL=(root) NOPASSWD:/tmp/root_Exachk.sh
               
    (5) update the sudoers file on each storage server
        using visudo command:
            >cellmonitor ALL=(root) NOPASSWD:/tmp/Exachk_cells.sh
          
   

Menu Option: M  -  [+] Show Menu Details
Menu Option: B  -  [+] Show Menu Details
Top | Back

COMMAND: adr_130
COMMAND TYPE: Adr
COMMAND TITLE: Incidents, Limit By Number
COMMAND DESCRIPTION:
    Command Summary
      Lists all of the incidents associated with the current ADR home.   
      Output is limited to the number of incidents requested.
      
    Input Requirements
      o Choose an output mode for incidents. BASIC is the default.
        - BASIC displays only basic incident information (the INCIDENT_ID, 
          PROBLEM_ID, and CREATE_TIME fields
          It does not display flood-controlled incidents.
        - BRIEF displays all information related to the incidents
          It includes flood-controlled incidents
        - DETAIL displays all information for the incidents (as with BRIEF 
          mode) as well as information about incident dumps
          It includes flood-controlled incidents
      o A maximun number of incidents to be returned is prompted for 
        - Default is 50	
    
    Ouput Details
      o A list of incidents up to the number requested will be displayed
      
    
command_prompt1:
MODE|Choose BASIC, BRIEF or DETAIL for Mode. Specify the desired number of incidents to report on or specify ALL Mode|BASIC
command_prompt2:
NUM_INC|Number of Incidents|50
COMMAND ARGUMENT(S):
show_incident_all %%MODE%% %%NUM_INC%% 
Top|Back

COMMAND: adr_150
COMMAND TYPE: Adr
COMMAND TITLE: Create Incident Package
COMMAND DESCRIPTION:
    Command Summary
      Creates a new package. ADRCI automatically assigns the package number 
      for the new package.
      
    Input Requirements
      o Incident ID is required
        - Default is 0 which will return an error
    
    Ouput Details
      o Package number created will be displayed
      
    
command_prompt1:
INC_ID|Enter the Incident ID|0
COMMAND ARGUMENT(S):
ips_create %%INC_ID%%
Top|Back

COMMAND: os_213
COMMAND TYPE: OS
COMMAND TITLE: Procwatcher status
COMMAND DESCRIPTION:
    Command Summary
      This command shows the current status of Procwatcher. 
      
    Input Requirements
      o None
    
    Ouput Details
      o Current status of Procwatcher
      
    
COMMAND ARGUMENT(S):
%%DA_PROCWATCHER_DIR%%/prw.sh stat
Top|Back

COMMAND: os_214
COMMAND TYPE: OS
COMMAND TITLE: Start Procwatcher
COMMAND DESCRIPTION:
    Command Summary
      This command starts Procwatcher.
      
    Input Requirements
      o None
    
    Ouput Details
      o Confirmation that Procwatcher was started
      
    
COMMAND ARGUMENT(S):
%%DA_PROCWATCHER_DIR%%/prw.sh start
Top|Back

COMMAND: os_215
COMMAND TYPE: OS
COMMAND TITLE: Stop Procwatcher
COMMAND DESCRIPTION:
    Command Summary
      This command stops Procwatcher. 
      
    Input Requirements
      o None
    
    Ouput Details
      o Confirmation that Procwatcher was terminated
      
    
COMMAND ARGUMENT(S):
%%DA_PROCWATCHER_DIR%%/prw.sh stop
Top|Back

COMMAND: os_216
COMMAND TYPE: OS
COMMAND TITLE: Package Procwatcher files
COMMAND DESCRIPTION:
    Command Summary
      This command packages Procwatcher files. 
      
    Input Requirements
      o None
    
    Ouput Details
      o Confirmation that Procwatcher files have been packaged
      
    
COMMAND ARGUMENT(S):
%%DA_PROCWATCHER_DIR%%/prw.sh pack
    echo DA_FILE_COLLECTED=$PWD/Procwatcher_Files_.*.zip
    
Top|Back

COMMAND: os_218
COMMAND TYPE: OS
COMMAND TITLE: ILOM snapshot data
COMMAND DESCRIPTION:
    Command Summary
      This command collects ILOM snapshot data previously generated.
      
    Input Requirements
      o Full directory path where the ILOM zip file is located
      o Node name (without "-ilom")
    
    Ouput Details
      o Confirms that the snapshot was located
  
    
command_prompt1:
ILOM_ZIPDIR|Enter the directory where the ILOM zip file is found|
command_prompt2:
NODE_NAME|Enter the Node Name (without "-ilom")|
COMMAND ARGUMENT(S):
#!/bin/sh
       if [ "x%%ILOM_ZIPDIR%%" = "x" -o  "x%%NODE_NAME%%" = "x" ]; then
       echo "ERROR: Please enter a value for both ILOM zip directory and Node Name."
       else
         if [ -d %%ILOM_ZIPDIR%% ]; then
         echo DA_FILE_COLLECTED=%%ILOM_ZIPDIR%%/%%NODE_NAME%%-ilom.*.zip
         else
         echo "ERROR: Directory Invalid. You must enter a valid directory path."
         fi
       fi
    
Top|Back

COMMAND: os_220
COMMAND TYPE: OS
COMMAND TITLE: DAexaChk
COMMAND DESCRIPTION:
    Command Summary
      This command runs Exachk in non-interactive mode including root
      privileged checks.
      
    Input Requirements
      o Full directory path where Exachk is installed
      o Full path of the CRS_HOME
    
    Ouput Details
      o Results of the Exachk, including root privileged checks

    
command_prompt1:
Exachk_DIR|Enter the directory where Exachk is installed|
command_prompt2:
CRS_HOME|Enter the CRS_HOME|
COMMAND ARGUMENT(S):
#!/bin/sh
    if [ "x%%Exachk_DIR%%" = "x" -o  "x%%CRS_HOME%%" = "x" ]; then
    echo "ERROR: Please enter a value for both Exachk directory and CRS_HOME."
    else
      if [ -d %%CRS_HOME%% ]; then
        if [ -d %%Exachk_DIR%% ]; then
        export CRS_HOME=%%CRS_HOME%%
        %%Exachk_DIR%%/Exachk -s
        else
        echo "ERROR: Please enter a valid directory where Exachk is installed."
        fi
      else
      echo "ERROR: Please enter a valid CRS_HOME."
      fi
    fi
    
Top|Back

COMMAND: os_221
COMMAND TYPE: OS
COMMAND TITLE: Exachk without root privileges
COMMAND DESCRIPTION:
    Command Summary
      This command runs Exachk in non-interactive mode without root
      privileged checks.
      
    Input Requirements
      o Full directory path where Exachk is installed
      o Full path of the CRS_HOME
    
    Ouput Details
      o Results of the Exachk, excluding root privileged checks

    
command_prompt1:
Exachk_DIR|Enter the directory where Exachk is installed|
command_prompt2:
CRS_HOME|Enter the CRS_HOME|
COMMAND ARGUMENT(S):
#!/bin/sh
    if [ "x%%Exachk_DIR%%" = "x" -o  "x%%CRS_HOME%%" = "x" ]; then
    echo "ERROR: Please enter a value for both Exachk directory and
CRS_HOME."
    else
      if [ -d %%CRS_HOME%% ]; then
        if [ -d %%Exachk_DIR%% ]; then
        export CRS_HOME=%%CRS_HOME%%
        %%Exachk_DIR%%/Exachk -S
        else
        echo "ERROR: Please enter a valid directory where Exachk is installed."
        fi
      else
      echo "ERROR: Please enter a valid CRS_HOME."
      fi
    fi
    
Top|Back

COMMAND: os_222
COMMAND TYPE: OS
COMMAND TITLE: Exachk Data
COMMAND DESCRIPTION:
    Command Summary
      This command collects data from an Exachk execution performed outside
      of the Diagnostic Assistant tool.
      
    Input Requirements
      o Full directory path where Exachk zip file is located
    
    Ouput Details
      o Confirmation that the zip file was found

    
command_prompt1:
Exachk_ZIPDIR|Enter the directory where the Exachk zip file is found|
COMMAND ARGUMENT(S):
#!/bin/sh
     if [ "x%%Exachk_ZIPDIR%%" = "x" ]; then
       echo "ERROR: No Directory. You must enter a valid directory path."
     else
       if [ -d %%Exachk_ZIPDIR%% ]; then
       echo DA_FILE_COLLECTED=%%Exachk_ZIPDIR%%/.*.zip
       else
       echo "ERROR: Directory Invalid. You must enter a valid directory
path."
       fi
     fi
    
Top|Back

COMMAND: os_292
COMMAND TYPE: OS
COMMAND TITLE: Clusterware Startup Problems
COMMAND DESCRIPTION:
    Command Summary
      This command collects Clusterware related log files for a given date.
      
    Input Requirements
      o Full path to the GRID_HOME
      o Date in YYYY-MM-DD format
    
    Ouput Details
      o Zip file of the Clusterware log files for the given date
  
    
command_prompt1:
GRIDHOME|Enter the full path to the GRID_HOME|
command_prompt2:
GREPDATE|Enter the date to grep for in YYYY-MM-DD format|2011-01-31
COMMAND ARGUMENT(S):
#!/bin/sh
    export HOST=`hostname -a`
    export GRID_HOME=%%GRIDHOME%%
    export BASELOGDIR=${GRID_HOME}/log/${HOST}
    export GRIDZIP=gridlogs_${HOST}.zip
     
    if [ -d $BASELOGDIR ]; then
      cd $BASELOGDIR
    else
      echo "ERROR: Invalid directory specified for GRID_HOME: ${GRID_HOME}"
      echo "Please export this environment variable with the correct value."
      exit 1
    fi

    export BASEOUTDIR=/tmp/da_grid
    export OUTDIR=${BASEOUTDIR}/logs_$$
    if [ ! -d ${BASEOUTDIR} ]; then
      mkdir ${BASEOUTDIR}
    fi
     
    if [ ! -d ${OUTDIR} ]; then
      mkdir ${OUTDIR}
    else
      rm -rf ${OUTDIR}/*
    fi
     
    # Need to capture output of:$GRID_HOME/bin/crsctl stat res -t -init
    ${GRID_HOME}/bin/crsctl stat res -t -init > ${OUTDIR}/crsctl.log

    export GREPDATE="%%GREPDATE%%"
         
    ## First, check the alert log
    F1=`grep -c "$GREPDATE" alert${HOST}.log`
    if [ ${F1} -gt 0 ]; then
      cp alert${HOST}.log ${OUTDIR}/
    fi
   
    ## Next, check ohasd/ohasd.*
    for a_log in `ls ohasd/ohasd.l*`
      do
        F2=`grep -c "$GREPDATE" ${a_log}`
        if [ ${F2} -gt 0 ]; then
          cp ${a_log} ${OUTDIR}/
        fi
    done
     
    ## Next, check agent/ohasd/oraagent_${USER}/oraagent_${USER}.l*

    for a_log in `ls agent/ohasd/oraagent_${USER}/oraagent_${USER}.l*`
      do
        F2=`grep -c "$GREPDATE" ${a_log}`
        if [ ${F2} -gt 0 ]; then
          cp ${a_log} ${OUTDIR}/
        fi
    done

    ## Next, check agent/ohasd/orarootagent_root/orarootagent_root.l*

    for a_log in `ls agent/ohasd/orarootagent_root/orarootagent_root.l*`
      do
        F2=`grep -c "$GREPDATE" ${a_log}`
        if [ ${F2} -gt 0 ]; then
          cp ${a_log} ${OUTDIR}/
        fi
    done

    ## Next, check cssd/ocssd.l*

    for a_log in `ls cssd/ocssd.l*`
      do
        F2=`grep -c "$GREPDATE" ${a_log}`
        if [ ${F2} -gt 0 ]; then
          cp ${a_log} ${OUTDIR}/
        fi
    done

    ## Next, check crsd/crsd.l*

    for a_log in `ls crsd/crsd.l*`
      do
        F2=`grep -c "$GREPDATE" ${a_log}`
        if [ ${F2} -gt 0 ]; then
          cp ${a_log} ${OUTDIR}/
        fi
    done

    ## Next, check agent/crsd/oraagent_${USER}/oraagent_${USER}.l*

    for a_log in `ls agent/crsd/oraagent_${USER}/oraagent_${USER}.l*`
      do
        F2=`grep -c "$GREPDATE" ${a_log}`
        if [ ${F2} -gt 0 ]; then
          cp ${a_log} ${OUTDIR}/
        fi
    done

    ## Next, check agent/crsd/orarootagent_root/orarootagent_root.l*

    for a_log in `ls agent/crsd/orarootagent_root/orarootagent_root.l*`
      do
        F2=`grep -c "$GREPDATE" ${a_log}`
        if [ ${F2} -gt 0 ]; then
          cp ${a_log} ${OUTDIR}/
        fi
    done

    ## Next, check diskmon/diskmon.l*

    for a_log in `ls diskmon/diskmon.l*`
      do
        F2=`grep -c "$GREPDATE" ${a_log}`
        if [ ${F2} -gt 0 ]; then
          cp ${a_log} ${OUTDIR}/
        fi
    done

    ## Next, check gpnpd/gpnpd.l*

    for a_log in `ls gpnpd/gpnpd.l*`
      do
        F2=`grep -c "$GREPDATE" ${a_log}`
        if [ ${F2} -gt 0 ]; then
          cp ${a_log} ${OUTDIR}/
        fi
    done

    ## Zip our list of files up, ignoring directories 
    cd ${OUTDIR}
    zip -j "${OUTDIR}/gridlog.zip" *

    echo DA_FILE_COLLECTED=${OUTDIR}/gridlog.zip

    
Top|Back

COMMAND: os_293
COMMAND TYPE: OS
COMMAND TITLE: LSI Battery State
COMMAND DESCRIPTION:
    Command Summary
      This command displays information about LSI battery state.
      
    Input Requirements
      o None
    
    Ouput Details
      o Details about the LSI battery state
  
    
COMMAND ARGUMENT(S):
#!/bin/sh
     if [ ! -x "/opt/MegaRAID/MegaCli/MegaCli64" ]; then
       echo "ERROR: LSI MegaCli64 executable not found"
     else
       /opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -aAll
     fi
    
Top|Back

COMMAND: os_294
COMMAND TYPE: OS
COMMAND TITLE: Virtual Drive(s)
COMMAND DESCRIPTION:
    Command Summary
      This command displays information about Virtual Drive(s).
      
    Input Requirements
      o None
    
    Ouput Details
      o Details about Virtual Drive(s)
  
    
COMMAND ARGUMENT(S):
#!/bin/sh
     if [ ! -x "/opt/MegaRAID/MegaCli/MegaCli64" ]; then
       echo "ERROR: LSI MegaCli64 executable not found"
     else
       /opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -Lall -aAll
     fi
    
Top|Back

COMMAND: os_295
COMMAND TYPE: OS
COMMAND TITLE: Cache State
COMMAND DESCRIPTION:
    Command Summary
      This command verifies the state of the cache in the dbnodes.
      
    Input Requirements
      o None
    
    Ouput Details
      o Verification of the cache state
  
    
COMMAND ARGUMENT(S):
#!/bin/sh
     if [ ! -x "/opt/MegaRAID/MegaCli/MegaCli64" ]; then
       echo "ERROR: LSI MegaCli64 executable not found"
     else
       /opt/MegaRAID/MegaCli/MegaCli64 --CfgDsply -a0 |grep -i writethrough
     fi
    
Top|Back

COMMAND: rda_310
COMMAND TYPE: Rda
COMMAND TITLE: Network Information
COMMAND DESCRIPTION:
    Command Summary
     Runs NET module to collect Network information.
      
    Input Requirements
      o None
    
    Ouput Details
      o Shows a summary of the collection process
      o Generates a collection of files in the "NET" subdirectory under 
        the Diagnostic Assistant output directory
        
    
COMMAND ARGUMENT(S):
run_module=NET
Top|Back

COMMAND: rda_313
COMMAND TYPE: Rda
COMMAND TITLE: Database Trace and Log Files
COMMAND DESCRIPTION:
    Command Summary
      Runs LOG module to collect database trace and log files
    
    Input Requirements
      o None
    
    Ouput Details
      o Shows a summary of the collection process
      o Generates a collection of files in the "LOG" subdirectory under 
        the Diagnostic Assistant output directory

    
COMMAND ARGUMENT(S):
run_module=LOG
Top|Back

COMMAND: rda_314
COMMAND TYPE: Rda
COMMAND TITLE: Alert Logs and Incidents
COMMAND DESCRIPTION:
    Command Summary
      Runs LOG module and shows the Alert Log and Diagnostic Incidents files
    
    Input Requirements
      o None
    
    Ouput Details
      o Shows the Alert log and Diagnostic Incidents log
      
  
COMMAND ARGUMENT(S):
run_module=LOG file_output=DA_LOG_alert_log.txt,DA_LOG_diag_incidents.txt view_existing_output=true
Top|Back

COMMAND: rda_315
COMMAND TYPE: Rda
COMMAND TITLE: Automatic Storage Management
COMMAND DESCRIPTION:
    Command Summary
      Runs ASM module and collects Automatic Storage management (ASM) information.

    Input Requirements
      o Is ASM installed in a separate Oracle home (Y/N)?
      o ASM Oracle Home Directory Path
      o ASM Oracle SID
    
    Ouput Details
      o Shows a summary of the collection process
      o Generates a collection of files in the "ASM" subdirectory under 
        the Diagnostic Assistant output directory

  
COMMAND ARGUMENT(S):
run_module=ASM
Top|Back

COMMAND: rda_318
COMMAND TYPE: Rda
COMMAND TITLE: Oracle Network Information
COMMAND DESCRIPTION:
    Command Summary
      Runs ONET module to collect Oracle Network information.
      
    Input Requirements
      o None
    
    Ouput Details
      o Shows a summary of the collection process
      o Generates a collection of files in the "ONET" subdirectory under 
        the Diagnostic Assistant output directory
        
    
COMMAND ARGUMENT(S):
run_module=ONET
Top|Back

COMMAND: rda_400
COMMAND TYPE: Rda
COMMAND TITLE: Exadata Database Hanging
COMMAND DESCRIPTION:
    Command Summary
      Runs Exadata_DatabaseHang profile to collect information to diagnose 
      database hanging issues.
      
    Input Requirements
      o None
    
    Ouput Details
      o Shows a summary of the collection process
      o Generates a collection of files in the "Exadata_DatabaseHang" 
        subdirectory under the Diagnostic Assistant output directory
  
    
COMMAND ARGUMENT(S):
run_profile=Exadata_DatabaseHang
Top|Back

COMMAND: rda_402
COMMAND TYPE: Rda
COMMAND TITLE: Exadata Failed Drives
COMMAND DESCRIPTION:
    Command Summary
      Runs Exadata_FailedDrives profile to collect information to diagnose 
      failed drive issues.
      
    Input Requirements
      o Enter a specific cell name/IP or a newline for all cells
    
    Ouput Details
      o Shows a summary of the collection process
      o Generates a collection of files in the "Exadata_DatabaseHang" 
        subdirectory under the Diagnostic Assistant output directory
        
    
command_prompt1:
CELL_NODE|You may enter a specific cell name/IP or newline for all cells|ALL
COMMAND ARGUMENT(S):
run_profile=Exadata_FailedDrives CELL_NODE=%%CELL_NODE%%
Top|Back

COMMAND: rda_404
COMMAND TYPE: Rda
COMMAND TITLE: Exadata Sick Cell
COMMAND DESCRIPTION:
    Command Summary
      Runs Exadata_SickCell profile to collect information to diagnose 
      failed cell issues.
      
    Input Requirements
      o Enter a specific cell name/IP or a newline for all cells
    
    Ouput Details
      o Shows a summary of the collection process
      o Generates a collection of files in the "Exadata_Sickcell" 
        subdirectory under the Diagnostic Assistant output directory
        
    
command_prompt1:
CELL_NODE|You may enter a specific cell name/IP or newline for all cells|ALL
COMMAND ARGUMENT(S):
run_profile=Exadata_SickCell CELL_NODE=%%CELL_NODE%%
Top|Back

COMMAND: rda_410
COMMAND TYPE: Rda
COMMAND TITLE: Exadata Flash Drives
COMMAND DESCRIPTION:
    Command Summary
      Runs Exadata_FlashDrives profile to collect information to diagnose  
      flash drive issues.
      
    Input Requirements
      o Enter a specific cell name/IP or a newline for all cells
    
    Ouput Details
      o Shows a summary of the collection process
      o Generates a collection of files in the "Exadata_FlashDrives" 
        subdirectory under the Diagnostic Assistant output directory
        
    
command_prompt1:
CELL_NODE|You may enter a specific cell name/IP or newline for all cells|ALL
COMMAND ARGUMENT(S):
run_profile=Exadata_FlashDrives CELL_NODE=%%CELL_NODE%%
Top|Back

COMMAND: rda_412
COMMAND TYPE: Rda
COMMAND TITLE: Exadata Network Cabling
COMMAND DESCRIPTION:
    Command Summary
      Runs Exadata_NetworkCabling profile to collect information to  
      diagnose Network cabling issues.
      
    Input Requirements
      o None
    
    Ouput Details
      o Shows a summary of the collection process
      o Generates a collection of files in the "Exadata_NetworkCabling" 
        subdirectory under the Diagnostic Assistant output directory
        
    
COMMAND ARGUMENT(S):
run_profile=Exadata_NetworkCabling
Top|Back

COMMAND: rda_414
COMMAND TYPE: Rda
COMMAND TITLE: Exadata Ib Switch
COMMAND DESCRIPTION:
    Command Summary
      Runs Exadata_IbSwitch profile to collect information to diagnose 
      Ib switch issues.
      
    Input Requirements
      o Enter the user login for the remote switches
    
    Ouput Details
      o Shows a summary of the collection process
      o Generates a collection of files in the "Exadata_IbSwitch" 
        subdirectory under the Diagnostic Assistant output directory
        
    
COMMAND ARGUMENT(S):
run_profile=Exadata_IbSwitch
Top|Back

COMMAND: rda_416
COMMAND TYPE: Rda
COMMAND TITLE: Exadata Ib Switch Root
COMMAND DESCRIPTION:
    Command Summary
      Runs Exadata_IbSwitch profile at the root level to collect additional 
      information for the Exadata_IbSwitch collection. You must have run the
      Exadata_IbSwitch profile in normal (non-root) mode prior to this and 
      have an existing "Exadata_IbSwitch" subdirectory under the Diagnostic  
      Assistant output directory to store this additional information.
      
    Input Requirements
      o Enter the root user password
    
    Ouput Details
      o Shows a summary of the collection process
      o Generates a collection of files in the "Exadata_IbSwitch" 
        subdirectory under the Diagnostic Assistant output directory that 
        contain detailed switch data
        
    
COMMAND ARGUMENT(S):
run_profile_root=Exadata_IbSwitch
Top|Back

COMMAND: rda_418
COMMAND TYPE: Rda
COMMAND TITLE: Exadata RAC Instance
COMMAND DESCRIPTION:
    Command Summary
      Runs Exadata_RacInstance profile to collect information to diagnose 
      RAC instance issues.
      
    Input Requirements
      o Is ASM installed in a separate Oracle home (Y/N)?
      o ASM Oracle Home Directory Path
      o ASM Oracle SID
    
    Ouput Details
      o Shows a summary of the collection process
      o Generates a collection of files in the "Exadata_RacInstance" 
        subdirectory under the Diagnostic Assistant output directory
        
    
COMMAND ARGUMENT(S):
run_profile=Exadata_RacInstance
Top|Back

COMMAND: rda_420
COMMAND TYPE: Rda
COMMAND TITLE: Exadata Assessment
COMMAND DESCRIPTION:
    Command Summary
      Runs Exadata_Assessment profile to collect information about the entire 
      Exadata implementation. This collection may take over 15 minutes to complete.
      
    Input Requirements
      o Is ASM installed in a separate Oracle home (Y/N)?
      o ASM Oracle Home Directory Path
      o ASM Oracle SID
    
    Ouput Details
      o Shows a summary of the collection process
      o Generates a collection of files in the "Exadata_Assessment" 
        subdirectory under the Diagnostic Assistant output directory
        
    
COMMAND ARGUMENT(S):
run_profile=Exadata_Assessment
Top|Back

COMMAND: sql_513
COMMAND TYPE: Sqlplus
COMMAND TITLE: Database Assessment
COMMAND DESCRIPTION:
    Command Summary
      Collects the controlfile and redohdr dump data.
      
    Input Requirements
      o None
    
    Ouput Details
      o Shows the controlfile and redohdr dump

    
COMMAND ARGUMENT(S):
connect / as sysdba
      
      set pages 10000
      prompt Collecting the controlfile and redohdr dump...
  
      alter session set events 'immediate trace name controlf level 10' ;
      alter session set events 'immediate trace name redohdr level 10' ; 
    
Top|Back

COMMAND: ssh_700
COMMAND TYPE: OS
COMMAND TITLE: Exadata Cell Patching Step 1
COMMAND DESCRIPTION:
    Command Summary
      This command is step 1 in the 2 step process for the collection of 
      important patching and validation logs.
      
      Step 1 captures from the cell side. Requires "root"
      user creditionals:
        /var/log/cellos/CheckHWnFWProfile.log
        /var/log/cellos/validations.log
        /var/log/cellos/checkconfigs_configuration_summaries.txt
        /var/log/cellos/validations/upgradecbusb.log
        /var/log/cellos/patch.out.place.log
        /root/_patch_hctap_/_p_/wait_out 
        /root/_patch_hctap_/ all .log and .txt files
        /boot/ all .log and .txt files
        imageinfo
        ipmitool event logs 
        /opt/oracle.SupportTools/CheckHWnFWProfile hardware and firmware checks
        /var/log/messages
        /SP/console/history
      
    Input Requirements
      o Cell Node(s)
        - select cell(s) FIRST_NODE, ALL_NODES, enter name/ip 
        - default is ALL NODES
    
    Ouput Details
      o Location of the resulting zip file
  
    
command_prompt2:
CELL_NODE|Please select cell(s) FIRST_NODE, ALL_NODES, entered name/ip 192.168.10.5 |ALL_NODES
COMMAND ARGUMENT(S):
SSHCELL_AS_ROOT %%CELL_NODE%%
    echo "BEGIN GET --------------- /var/log/cellos/CheckHWnFWProfile.log"
    echo " "
    cat /var/log/cellos/CheckHWnFWProfile.log 
    echo " "
    echo "END --------------------- /var/log/cellos/CheckHWnFWProfile.log"
    echo " "
    
    echo "BEGIN GET --------------- /var/log/cellos/validations.log"
    echo " "
    cat /var/log/cellos/validations.log
    echo " "
    echo "END ---------------------  /var/log/cellos/validations.log"
    echo " "
    
    echo "BEGIN GET --------------- /var/log/cellos/checkconfigs_configuration_summaries.txt"
    echo " "
    cat /var/log/cellos/checkconfigs_configuration_summaries.txt
    echo " "
    echo "END ---------------------  /var/log/cellos/checkconfigs_configuration_summaries.txt"
    echo " "
    
    echo "BEGIN GET --------------- /var/log/cellos/upgradecbusb.log"
    echo " "
    cat /var/log/cellos/upgradecbusb.log
    echo " "
    echo "END ---------------------  /var/log/cellos/upgradecbusb.log"
    echo " "
    
    echo "BEGIN GET --------------- /var/log/cellos/patch.out.place.log"
    echo " "
    cat /var/log/cellos/patch.out.place.log
    echo " "
    echo "END ---------------------  /var/log/cellos/patch.out.place.log"
    echo " "
 
    for i in `ls /root/_patch_hctap_/_p_/wait*`
    do
    echo "BEGIN GET --------------- $i"
    echo " "
    cat $i
    echo " "
    echo "END ---------------------  $i"
    echo " "
    done

    for i in `ls /root/_patch_hctap_/*.log`
    do
    echo "BEGIN GET --------------- $i"
    echo " "
    cat $i
    echo " "
    echo "END ---------------------  $i"
    echo " "
    done

    for i in `ls /root/_patch_hctap_/*.txt`
    do
    echo "BEGIN GET --------------- $i"
    echo " "
    cat $i
    echo " "
    echo "END ---------------------  $i"
    echo " "
    done

    for i in `ls /boot/*.log`
    do
    echo "BEGIN GET --------------- $i"
    echo " "
    cat $i
    echo " "
    echo "END ---------------------  $i"
    echo " "
    done

    for i in `ls /boot/*.txt`
    do
    echo "BEGIN GET --------------- $i"
    echo " "
    cat $i
    echo " "
    echo "END ---------------------  $i"
    echo " "
    done

    echo "BEGIN GET --------------- imageinfo"
    echo " "
    imageinfo
    echo " "
    echo "END --------------------- imageinfo"
    echo " "

    echo "BEGIN GET --------------- ipmitool event logs"
    echo " "
    ipmitool sel elist > ipmitool_sel_elist.`hostname -s`.txt
    cat ipmitool_sel_elist.`hostname -s`.txt
    echo " "
    echo "END --------------------- ipmitool event logs"
    echo " "

    echo "BEGIN GET --------------- hardware and firmware checks"
    echo " "
    /opt/oracle.SupportTools/CheckHWnFWProfile -c loose >  hwfw.`hostname`.txt
    cat hwfw.`hostname`.txt
    echo " "
    echo "END --------------------- hardware and firmware checks"
    echo " "

    echo "BEGIN GET --------------- /var/log/messages"
    echo " "
    cat /var/log/messages
    echo " "
    echo "END --------------------- /var/log/messages"
    echo " "

    echo "BEGIN GET --------------- Serial Console"
    echo " "
    ipmitool sunoem cli 'show  /SP/console/history' > serialconsole.`hostname -s`.txt
    cat serialconsole.`hostname -s`.txt
    echo " "
    echo "END --------------------- Serial Console"
    echo " "

    
Top|Back

COMMAND: ssh_701
COMMAND TYPE: OS
COMMAND TITLE: Exadata Cell Patching Step 2
COMMAND DESCRIPTION:
    Command Summary
      This command is step 2 in the 2 step process for the collection of 
      important patching and validation logs.
      
      Step 2 captures patching logs from node location of where the patchmgr
      was last run. Generally requires "root" user creditionals:
            {PATCHMGR_DIR}/patchmgr.stderr
            {PATCHMGR_DIR}/patchmgr.stderr
            {PATCHMGR_DIR}/patchmgr_output.txt
            {PATCHMGR_DIR}/.../...logs
      
    Input Requirements
      o Host
        - enter the node name where the patchmgr ran from
        - no .domain
      o Password
        - ssh password for root user
      o Patch Manager
        - enter the location of where the patchmgr was last run
    
    Ouput Details
      o patching logs from node location of where the patchmgr
        was last run
  
    
command_prompt1:
HOST|Please enter the node name where the patchmgr ran from (no .domain)|hostname
command_prompt2:
PASS|Please enter the ssh password for root user|
command_prompt3:
PATCHMGR|Please enter the location of where the patchmgr was last run|/root/11.2.1.3.1/patch_11.2.1.3.1
COMMAND ARGUMENT(S):
SSH %%PASS%% root@%%HOST%% export LOGDIR=%%PATCHMGR%%
    if [ -d $LOGDIR ]; then
      cd $LOGDIR
    else
      echo "ERROR: Invalid directory specified for PatchMgr Location: %%PATCHMGR%%"
      exit 1
    fi

    echo "BEGIN GET --------------- ${LOGDIR}/patchmgr.stderr"
    echo " "
    cat ${LOGDIR}/patchmgr.stderr
    echo " "
    echo "END ---------------------  ${LOGDIR}/patchmgr.stderr"
    echo " "
    
    echo "BEGIN GET --------------- ${LOGDIR}/patchmgr.stdout"
    echo " "
    cat ${LOGDIR}/patchmgr.stdout
    echo " "
    echo "END ---------------------  ${LOGDIR}/patchmgr.stdout"
    echo " "
    
    echo "BEGIN GET --------------- ${LOGDIR}/patchmgr_output.txt"
    echo " "
    cat ${LOGDIR}/patchmgr_output.txt
    echo " "
    echo "END ---------------------  ${LOGDIR}/patchmgr_output.txt"
    echo " "
    
    for i in `find ${LOGDIR} -name \*.log`
    do
    echo "BEGIN GET --------------- $i"
    echo " "
    cat $i
    echo " "
    echo "END ---------------------  $i"
    echo " "
    done

    
Top|Back

COMMAND: ssh_702
COMMAND TYPE: OS
COMMAND TITLE: Collect Patching Logs using ILOM
COMMAND DESCRIPTION:
    Command Summary
      This option can be used when the cell is not accessible but the ILOM 
      can be accessed. Root user credentials are required.
      
    Input Requirements
      o Host
        - enter the IP address or hostname of the server to connect from
        - no .domain
      o Password
        - ssh password for root user
      o ILOMHost
        - enter the ILOM ip address or hostname for the troubled cell
      o ILOMPassword
        - enter ILOM root user password

    Ouput Details
      o patching logs from the ILOM serial console for the cell
  
    
command_prompt1:
HOST|Please enter the IP address or hostname of the server to connect from|hostname
command_prompt2:
PASS|Please enter the ssh password for root user|
command_prompt3:
ILOMHOST|Please enter the ILOM ip address or hostname for the troubled cell|
command_prompt4:
PASS2|Please enter the ILOM root user password|
COMMAND ARGUMENT(S):
SSH %%PASS%% root@%%HOST%% export IHOST=%%ILOMHOST%% 
    export IPASS=%%PASS2%%
    
    echo "BEGIN GET --------------- Serial Console"
    echo " "
    rm -f serialconsole.${IHOST}.txt
    ipmitool -H ${IHOST} -U root -P ${IPASS} sunoem cli 'show  /SP/console/history' >> serialconsole.${IHOST}.txt
    cat serialconsole.${IHOST}.txt
    echo " "
    echo "END --------------------- Serial Console"

    
Top|Back