Fri Apr 15 17:01:35 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Shared memory segment for instance monitoring created Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskgroups = "" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Fri Apr 15 17:01:35 2011 PMON started with pid=2, OS id=18893 Fri Apr 15 17:01:35 2011 VKTM started with pid=3, OS id=18895 at elevated priority VKTM running at (20)ms precision Fri Apr 15 17:01:35 2011 DIAG started with pid=4, OS id=18899 Fri Apr 15 17:01:35 2011 PING started with pid=5, OS id=18901 Fri Apr 15 17:01:35 2011 PSP0 started with pid=6, OS id=18903 Fri Apr 15 17:01:35 2011 DIA0 started with pid=7, OS id=18905 Fri Apr 15 17:01:35 2011 LMON started with pid=8, OS id=18907 Fri Apr 15 17:01:35 2011 LMD0 started with pid=9, OS id=18909 Fri Apr 15 17:01:35 2011 LMS0 started with pid=10, OS id=18911 at elevated priority Fri Apr 15 17:01:35 2011 MMAN started with pid=11, OS id=18915 Fri Apr 15 17:01:35 2011 DBW0 started with pid=12, OS id=18917 Fri Apr 15 17:01:35 2011 LGWR started with pid=13, OS id=18919 Fri Apr 15 17:01:35 2011 CKPT started with pid=14, OS id=18921 Fri Apr 15 17:01:35 2011 SMON started with pid=15, OS id=18923 Fri Apr 15 17:01:35 2011 RBAL started with pid=16, OS id=18925 Fri Apr 15 17:01:36 2011 GMON started with pid=17, OS id=18927 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 4) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Fri Apr 15 17:01:36 2011 LCK0 started with pid=18, OS id=18932 ORACLE_BASE from environment = /u01/app/oracle Fri Apr 15 17:01:37 2011 SQL> ALTER DISKGROUP ALL MOUNT WARNING::ASMLIB library not found. See trace file for details. Fri Apr 15 17:03:25 2011 * allocate domain 1, invalid = TRUE freeing rdom 1 Received detach msg from node 0 for dom 1 * allocate domain 1, invalid = TRUE Fri Apr 15 17:03:33 2011 SQL> /* OracleOEM */ alter diskgroup ORAFILES MOUNT WARNING: Deprecated privilege SYSDBA for command 'ALTER DISKGROUP MOUNT' NOTE: cache registered group ORAFILES number=1 incarn=0x60b66419 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x60b66419 NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 Fri Apr 15 17:03:34 2011 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x60B66419 (ORAFILES) kjbdomatt send to node 0 Fri Apr 15 17:03:34 2011 NOTE: attached to recovery domain 1 Fri Apr 15 17:03:34 2011 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: ACD expansion required for disk group 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.0 ABA NOTE: seq=2 blk=0 NOTE: cache mounting group 1/0x60B66419 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x60b66419 Fri Apr 15 17:03:37 2011 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: /* OracleOEM */ alter diskgroup ORAFILES MOUNT Fri Apr 15 18:55:50 2011 Received detach msg from node 0 for dom 1 Fri Apr 15 18:55:50 2011 NOTE: PST enabling heartbeating (grp 1) Fri Apr 15 18:55:53 2011 Reconfiguration started (old inc 4, new inc 6) List of nodes: 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Apr 15 18:55:53 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 10 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Fri Apr 15 20:48:42 2011 Reconfiguration started (old inc 6, new inc 8) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Fri Apr 15 20:48:42 2011 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Apr 15 20:48:42 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 10 GCS shadows traversed, 10 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Fri Apr 15 20:49:12 2011 Received detach msg from node 0 for dom 1 Fri Apr 15 20:49:15 2011 Reconfiguration started (old inc 8, new inc 10) List of nodes: 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Apr 15 20:49:15 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 10 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Fri Apr 15 20:56:58 2011 Reconfiguration started (old inc 10, new inc 12) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Fri Apr 15 20:56:58 2011 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Apr 15 20:56:58 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 10 GCS shadows traversed, 10 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Fri Apr 15 20:58:01 2011 Shutting down instance: further logons disabled Shutting down instance (immediate) License high water mark = 2 SQL> ALTER DISKGROUP ALL DISMOUNT NOTE: cache dismounting group 1/0x60B66419 (ORAFILES) Fri Apr 15 20:58:01 2011 kjbdomdet send to node 0 detach from dom 1, sending detach message to node 0 Fri Apr 15 20:58:01 2011 NOTE: detached from domain 1 NOTE: cache dismounted group 1/0x60B66419 (ORAFILES) kfdp_dismount(): 4 Fri Apr 15 20:58:01 2011 kfdp_dismountBg(): 4 NOTE: De-assigning number (1,0) from disk (/dev/mpath/mpath1p1) SUCCESS: diskgroup ORAFILES was dismounted SUCCESS: ALTER DISKGROUP ALL DISMOUNT Fri Apr 15 20:58:01 2011 Stopping background process VKTM: Fri Apr 15 20:58:03 2011 freeing rdom 1 freeing rdom 0 Fri Apr 15 20:58:04 2011 Instance shutdown complete Fri Apr 15 21:05:08 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Fri Apr 15 21:05:09 2011 PMON started with pid=2, OS id=1081 Fri Apr 15 21:05:09 2011 VKTM started with pid=3, OS id=1083 at elevated priority VKTM running at (20)ms precision Fri Apr 15 21:05:09 2011 DIAG started with pid=4, OS id=1087 Fri Apr 15 21:05:09 2011 PING started with pid=5, OS id=1089 Fri Apr 15 21:05:09 2011 PSP0 started with pid=6, OS id=1091 Fri Apr 15 21:05:09 2011 DIA0 started with pid=7, OS id=1093 Fri Apr 15 21:05:09 2011 LMON started with pid=8, OS id=1095 Fri Apr 15 21:05:09 2011 LMD0 started with pid=9, OS id=1097 Fri Apr 15 21:05:09 2011 LMS0 started with pid=10, OS id=1099 at elevated priority Fri Apr 15 21:05:09 2011 MMAN started with pid=11, OS id=1103 Fri Apr 15 21:05:09 2011 DBW0 started with pid=12, OS id=1105 Fri Apr 15 21:05:09 2011 LGWR started with pid=13, OS id=1107 Fri Apr 15 21:05:09 2011 CKPT started with pid=14, OS id=1109 Fri Apr 15 21:05:09 2011 SMON started with pid=15, OS id=1111 Fri Apr 15 21:05:09 2011 RBAL started with pid=16, OS id=1113 Fri Apr 15 21:05:09 2011 GMON started with pid=17, OS id=1133 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 16) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Fri Apr 15 21:05:10 2011 LCK0 started with pid=18, OS id=1174 ORACLE_BASE from environment = /u01/app/oracle Fri Apr 15 21:05:10 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x60bcaf9d NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x60bcaf9d WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x60BCAF9D (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.0 ABA NOTE: seq=3 blk=0 NOTE: cache mounting group 1/0x60BCAF9D (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x60bcaf9d kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Fri Apr 15 21:06:30 2011 Received detach msg from node 0 for dom 1 Fri Apr 15 21:06:30 2011 NOTE: PST enabling heartbeating (grp 1) Fri Apr 15 21:06:33 2011 Reconfiguration started (old inc 16, new inc 18) List of nodes: 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Apr 15 21:06:34 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 10 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Fri Apr 15 21:20:19 2011 Reconfiguration started (old inc 18, new inc 20) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Fri Apr 15 21:20:19 2011 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Apr 15 21:20:19 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 10 GCS shadows traversed, 10 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Reconfiguration started (old inc 20, new inc 22) List of nodes: 1 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 10 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Fri Apr 15 21:20:28 2011 kfdp_query(): 4 Fri Apr 15 21:20:28 2011 kfdp_queryBg(): 4 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=1 ckpt=5.12 group=1 NOTE: advancing ckpt for thread=1 ckpt=5.12 NOTE: smon did instance recovery for domain 1 Fri Apr 15 21:22:43 2011 Reconfiguration started (old inc 22, new inc 24) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Fri Apr 15 21:22:43 2011 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Apr 15 21:22:43 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 10 GCS shadows traversed, 10 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Apr 26 10:48:58 2011 Reconfiguration started (old inc 24, new inc 26) List of nodes: 1 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Apr 26 10:48:58 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 10 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Tue Apr 26 10:48:58 2011 kfdp_query(): 5 Tue Apr 26 10:48:58 2011 kfdp_queryBg(): 5 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=1 ckpt=6.13 group=1 NOTE: advancing ckpt for thread=1 ckpt=6.13 NOTE: smon did instance recovery for domain 1 Tue Apr 26 11:00:24 2011 Reconfiguration started (old inc 26, new inc 28) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Tue Apr 26 11:00:24 2011 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Apr 26 11:00:24 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 10 GCS shadows traversed, 10 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Apr 26 11:08:58 2011 Shutting down instance (abort) License high water mark = 1 USER (ospid: 13651): terminating the instance Instance terminated by USER, pid = 13651 Tue Apr 26 11:09:00 2011 Instance shutdown complete Tue Apr 26 11:20:04 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Tue Apr 26 11:20:06 2011 PMON started with pid=2, OS id=10898 Tue Apr 26 11:20:06 2011 VKTM started with pid=3, OS id=10900 at elevated priority VKTM running at (20)ms precision Tue Apr 26 11:20:06 2011 DIAG started with pid=4, OS id=10905 Tue Apr 26 11:20:06 2011 PING started with pid=5, OS id=10912 Tue Apr 26 11:20:06 2011 PSP0 started with pid=6, OS id=10914 Tue Apr 26 11:20:06 2011 DIA0 started with pid=7, OS id=10916 Tue Apr 26 11:20:06 2011 LMON started with pid=8, OS id=10918 Tue Apr 26 11:20:06 2011 LMD0 started with pid=9, OS id=10920 Tue Apr 26 11:20:06 2011 LMS0 started with pid=10, OS id=10922 at elevated priority Tue Apr 26 11:20:06 2011 MMAN started with pid=11, OS id=10926 Tue Apr 26 11:20:06 2011 DBW0 started with pid=12, OS id=10928 Tue Apr 26 11:20:06 2011 LGWR started with pid=13, OS id=10930 Tue Apr 26 11:20:06 2011 CKPT started with pid=14, OS id=10932 Tue Apr 26 11:20:06 2011 SMON started with pid=15, OS id=10934 Tue Apr 26 11:20:06 2011 RBAL started with pid=16, OS id=10936 Tue Apr 26 11:20:06 2011 GMON started with pid=17, OS id=10938 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 32) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Apr 26 11:20:06 2011 LCK0 started with pid=18, OS id=10943 ORACLE_BASE from environment = /u01/app/oracle Tue Apr 26 11:20:08 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x60b8ce1e NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x60b8ce1e WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x60B8CE1E (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.0 ABA NOTE: seq=4 blk=1 NOTE: cache mounting group 1/0x60B8CE1E (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x60b8ce1e kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Tue Apr 26 21:49:39 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Tue Apr 26 21:49:40 2011 PMON started with pid=2, OS id=11593 Tue Apr 26 21:49:40 2011 VKTM started with pid=3, OS id=11595 at elevated priority VKTM running at (20)ms precision Tue Apr 26 21:49:40 2011 DIAG started with pid=4, OS id=11599 Tue Apr 26 21:49:40 2011 PING started with pid=5, OS id=11607 Tue Apr 26 21:49:40 2011 PSP0 started with pid=6, OS id=11609 Tue Apr 26 21:49:40 2011 DIA0 started with pid=7, OS id=11611 Tue Apr 26 21:49:40 2011 LMON started with pid=8, OS id=11613 Tue Apr 26 21:49:40 2011 LMD0 started with pid=9, OS id=11615 Tue Apr 26 21:49:40 2011 LMS0 started with pid=10, OS id=11617 at elevated priority Tue Apr 26 21:49:40 2011 MMAN started with pid=11, OS id=11621 Tue Apr 26 21:49:40 2011 DBW0 started with pid=12, OS id=11623 Tue Apr 26 21:49:40 2011 LGWR started with pid=13, OS id=11625 Tue Apr 26 21:49:40 2011 CKPT started with pid=14, OS id=11627 Tue Apr 26 21:49:40 2011 SMON started with pid=15, OS id=11629 Tue Apr 26 21:49:40 2011 RBAL started with pid=16, OS id=11631 Tue Apr 26 21:49:40 2011 GMON started with pid=17, OS id=11633 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 4) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Apr 26 21:49:41 2011 LCK0 started with pid=18, OS id=11638 ORACLE_BASE from environment = /u01/app/oracle Tue Apr 26 21:49:42 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x9ed8e5bb NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x9ed8e5bb WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x9ED8E5BB (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.0 ABA NOTE: seq=5 blk=2 NOTE: cache mounting group 1/0x9ED8E5BB (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x9ed8e5bb kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Tue Apr 26 21:52:29 2011 Shutting down instance (abort) License high water mark = 1 USER (ospid: 13738): terminating the instance Instance terminated by USER, pid = 13738 Tue Apr 26 21:52:30 2011 Instance shutdown complete Tue Apr 26 21:55:07 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Tue Apr 26 21:55:08 2011 PMON started with pid=2, OS id=18166 Tue Apr 26 21:55:08 2011 VKTM started with pid=3, OS id=18168 at elevated priority VKTM running at (20)ms precision Tue Apr 26 21:55:08 2011 DIAG started with pid=4, OS id=18172 Tue Apr 26 21:55:08 2011 PING started with pid=5, OS id=18174 Tue Apr 26 21:55:08 2011 PSP0 started with pid=6, OS id=18176 Tue Apr 26 21:55:08 2011 DIA0 started with pid=7, OS id=18178 Tue Apr 26 21:55:08 2011 LMON started with pid=8, OS id=18180 Tue Apr 26 21:55:08 2011 LMD0 started with pid=9, OS id=18182 Tue Apr 26 21:55:08 2011 LMS0 started with pid=10, OS id=18184 at elevated priority Tue Apr 26 21:55:08 2011 MMAN started with pid=11, OS id=18188 Tue Apr 26 21:55:08 2011 DBW0 started with pid=12, OS id=18190 Tue Apr 26 21:55:08 2011 LGWR started with pid=13, OS id=18192 Tue Apr 26 21:55:08 2011 CKPT started with pid=14, OS id=18194 Tue Apr 26 21:55:08 2011 SMON started with pid=15, OS id=18196 Tue Apr 26 21:55:08 2011 RBAL started with pid=16, OS id=18198 Tue Apr 26 21:55:08 2011 GMON started with pid=17, OS id=18200 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 8) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Apr 26 21:55:09 2011 LCK0 started with pid=18, OS id=18236 ORACLE_BASE from environment = /u01/app/oracle Tue Apr 26 21:55:10 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x9ed965ee NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x9ed965ee WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x9ED965EE (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.0 ABA NOTE: seq=6 blk=3 NOTE: cache mounting group 1/0x9ED965EE (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x9ed965ee kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Tue Apr 26 22:10:25 2011 Shutting down instance (abort) License high water mark = 1 USER (ospid: 9060): terminating the instance Instance terminated by USER, pid = 9060 Tue Apr 26 22:10:26 2011 Instance shutdown complete Tue Apr 26 22:28:25 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Tue Apr 26 22:28:26 2011 PMON started with pid=2, OS id=13547 Tue Apr 26 22:28:26 2011 VKTM started with pid=3, OS id=13549 at elevated priority VKTM running at (20)ms precision Tue Apr 26 22:28:26 2011 DIAG started with pid=4, OS id=13553 Tue Apr 26 22:28:26 2011 PING started with pid=5, OS id=13555 Tue Apr 26 22:28:26 2011 PSP0 started with pid=6, OS id=13557 Tue Apr 26 22:28:26 2011 DIA0 started with pid=7, OS id=13559 Tue Apr 26 22:28:26 2011 LMON started with pid=8, OS id=13561 Tue Apr 26 22:28:26 2011 LMD0 started with pid=9, OS id=13563 Tue Apr 26 22:28:27 2011 LMS0 started with pid=10, OS id=13565 at elevated priority Tue Apr 26 22:28:27 2011 MMAN started with pid=11, OS id=13569 Tue Apr 26 22:28:27 2011 DBW0 started with pid=12, OS id=13571 Tue Apr 26 22:28:27 2011 LGWR started with pid=13, OS id=13573 Tue Apr 26 22:28:27 2011 CKPT started with pid=14, OS id=13575 Tue Apr 26 22:28:27 2011 SMON started with pid=15, OS id=13577 Tue Apr 26 22:28:27 2011 RBAL started with pid=16, OS id=13579 Tue Apr 26 22:28:27 2011 GMON started with pid=17, OS id=13581 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 12) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Apr 26 22:28:27 2011 LCK0 started with pid=18, OS id=13586 ORACLE_BASE from environment = /u01/app/oracle Tue Apr 26 22:28:28 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x9ed98cac NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x9ed98cac WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x9ED98CAC (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.0 ABA NOTE: seq=7 blk=4 NOTE: cache mounting group 1/0x9ED98CAC (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x9ed98cac kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Tue Apr 26 22:38:11 2011 Shutting down instance (abort) License high water mark = 1 USER (ospid: 19706): terminating the instance Instance terminated by USER, pid = 19706 Tue Apr 26 22:38:12 2011 Instance shutdown complete Tue Apr 26 22:43:45 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Tue Apr 26 22:43:45 2011 Tue Apr 26 22:43:45 2011 PMON started with pid=2, OS id=29162 VKTM started with pid=3, OS id=29164 at elevated priority Tue Apr 26 22:43:45 2011 DIAG started with pid=4, OS id=29168 Tue Apr 26 22:43:45 2011 Tue Apr 26 22:43:45 2011 PING started with pid=5, OS id=29170 PSP0 started with pid=6, OS id=29172 Tue Apr 26 22:43:45 2011 DIA0 started with pid=7, OS id=29174 Tue Apr 26 22:43:45 2011 LMON started with pid=8, OS id=29176 VKTM running at (20)ms precision Tue Apr 26 22:43:45 2011 LMD0 started with pid=9, OS id=29178 Tue Apr 26 22:43:45 2011 LMS0 started with pid=10, OS id=29180 at elevated priority Tue Apr 26 22:43:45 2011 MMAN started with pid=11, OS id=29185 Tue Apr 26 22:43:45 2011 DBW0 started with pid=12, OS id=29192 Tue Apr 26 22:43:45 2011 LGWR started with pid=13, OS id=29194 Tue Apr 26 22:43:45 2011 CKPT started with pid=14, OS id=29197 Tue Apr 26 22:43:45 2011 SMON started with pid=15, OS id=29199 Tue Apr 26 22:43:45 2011 RBAL started with pid=16, OS id=29201 Tue Apr 26 22:43:45 2011 GMON started with pid=17, OS id=29203 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 16) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Apr 26 22:43:46 2011 LCK0 started with pid=18, OS id=29247 ORACLE_BASE from environment = /u01/app/oracle Tue Apr 26 22:43:47 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x9edaf394 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x9edaf394 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x9EDAF394 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.0 ABA NOTE: seq=8 blk=5 NOTE: cache mounting group 1/0x9EDAF394 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x9edaf394 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Tue Apr 26 22:47:56 2011 Reconfiguration started (old inc 16, new inc 18) List of nodes: 1 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Apr 26 22:47:56 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 10 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Tue Apr 26 22:47:56 2011 kfdp_query(): 4 Tue Apr 26 22:47:56 2011 NOTE: PST enabling heartbeating (grp 1) kfdp_queryBg(): 4 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=1 ckpt=8.16 group=1 NOTE: advancing ckpt for thread=1 ckpt=8.16 NOTE: smon did instance recovery for domain 1 Tue Apr 26 22:49:28 2011 Reconfiguration started (old inc 18, new inc 20) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Tue Apr 26 22:49:28 2011 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Apr 26 22:49:28 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 10 GCS shadows traversed, 10 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Apr 26 22:49:52 2011 Reconfiguration started (old inc 20, new inc 22) List of nodes: 1 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Apr 26 22:49:52 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Tue Apr 26 22:49:52 2011 kfdp_query(): 5 LMS 0: 10 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Tue Apr 26 22:49:52 2011 kfdp_queryBg(): 5 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=1 ckpt=9.17 group=1 NOTE: advancing ckpt for thread=1 ckpt=9.17 NOTE: smon did instance recovery for domain 1 Tue Apr 26 22:52:51 2011 Reconfiguration started (old inc 22, new inc 24) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Tue Apr 26 22:52:51 2011 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Apr 26 22:52:51 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 10 GCS shadows traversed, 10 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Mon May 02 11:27:04 2011 Reconfiguration started (old inc 24, new inc 26) List of nodes: 1 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Mon May 02 11:27:04 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Mon May 02 11:27:04 2011 kfdp_query(): 6 LMS 0: 10 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Mon May 02 11:27:04 2011 kfdp_queryBg(): 6 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=1 ckpt=10.18 group=1 NOTE: advancing ckpt for thread=1 ckpt=10.18 NOTE: smon did instance recovery for domain 1 Mon May 02 11:32:32 2011 Shutting down instance (abort) License high water mark = 1 USER (ospid: 14433): terminating the instance Instance terminated by USER, pid = 14433 Mon May 02 11:32:33 2011 Instance shutdown complete Mon May 02 11:42:55 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Mon May 02 11:42:57 2011 PMON started with pid=2, OS id=11589 Mon May 02 11:42:57 2011 VKTM started with pid=3, OS id=11591 at elevated priority VKTM running at (20)ms precision Mon May 02 11:42:57 2011 DIAG started with pid=4, OS id=11595 Mon May 02 11:42:57 2011 PING started with pid=5, OS id=11597 Mon May 02 11:42:57 2011 PSP0 started with pid=6, OS id=11599 Mon May 02 11:42:57 2011 DIA0 started with pid=7, OS id=11601 Mon May 02 11:42:57 2011 LMON started with pid=8, OS id=11603 Mon May 02 11:42:57 2011 LMD0 started with pid=9, OS id=11605 Mon May 02 11:42:57 2011 LMS0 started with pid=10, OS id=11607 at elevated priority Mon May 02 11:42:57 2011 MMAN started with pid=11, OS id=11611 Mon May 02 11:42:57 2011 DBW0 started with pid=12, OS id=11613 Mon May 02 11:42:57 2011 LGWR started with pid=13, OS id=11615 Mon May 02 11:42:57 2011 CKPT started with pid=14, OS id=11617 Mon May 02 11:42:57 2011 SMON started with pid=15, OS id=11619 Mon May 02 11:42:57 2011 RBAL started with pid=16, OS id=11621 Mon May 02 11:42:57 2011 GMON started with pid=17, OS id=11623 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 4) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Mon May 02 11:42:57 2011 LCK0 started with pid=18, OS id=11630 ORACLE_BASE from environment = /u01/app/oracle Mon May 02 11:42:58 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xe2f8da40 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xe2f8da40 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0xE2F8DA40 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.0 ABA NOTE: seq=9 blk=6 NOTE: cache mounting group 1/0xE2F8DA40 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xe2f8da40 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Mon May 02 11:43:57 2011 NOTE: disk validation pending for group 1/0xe2f8da40 (ORAFILES) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) SUCCESS: validated disks for 1/0xe2f8da40 (ORAFILES) NOTE: membership refresh pending for group 1/0xe2f8da40 (ORAFILES) Mon May 02 11:44:03 2011 kfdp_queryTimeout() kfdp_query(): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 kfdp_query(): 5 kfdp_queryBg(): 5 SUCCESS: refreshed membership for 1/0xe2f8da40 (ORAFILES) Mon May 02 11:44:18 2011 NOTE: membership refresh pending for group 1/0xe2f8da40 (ORAFILES) kfdp_query(): 6 Mon May 02 11:44:18 2011 kfdp_queryBg(): 6 SUCCESS: refreshed membership for 1/0xe2f8da40 (ORAFILES) kfdp_queryTimeout() Thu May 12 21:31:33 2011 Trace dumping is performing id=[cdmp_20110512213133] Thu May 12 21:57:13 2011 Trace dumping is performing id=[cdmp_20110512215712] Fri May 13 17:08:50 2011 Shutting down instance (abort) License high water mark = 1 USER (ospid: 31376): terminating the instance Instance terminated by USER, pid = 31376 Fri May 13 17:08:51 2011 Instance shutdown complete Fri May 13 17:22:32 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Fri May 13 17:22:33 2011 PMON started with pid=2, OS id=7255 Fri May 13 17:22:33 2011 VKTM started with pid=3, OS id=7259 at elevated priority VKTM running at (20)ms precision Fri May 13 17:22:33 2011 DIAG started with pid=4, OS id=7267 Fri May 13 17:22:33 2011 PING started with pid=5, OS id=7269 Fri May 13 17:22:33 2011 PSP0 started with pid=6, OS id=7271 Fri May 13 17:22:33 2011 DIA0 started with pid=7, OS id=7273 Fri May 13 17:22:33 2011 LMON started with pid=8, OS id=7275 Fri May 13 17:22:33 2011 LMD0 started with pid=9, OS id=7277 Fri May 13 17:22:33 2011 LMS0 started with pid=10, OS id=7279 at elevated priority Fri May 13 17:22:33 2011 MMAN started with pid=11, OS id=7283 Fri May 13 17:22:33 2011 DBW0 started with pid=12, OS id=7285 Fri May 13 17:22:33 2011 LGWR started with pid=13, OS id=7287 Fri May 13 17:22:33 2011 CKPT started with pid=14, OS id=7289 Fri May 13 17:22:33 2011 SMON started with pid=15, OS id=7291 Fri May 13 17:22:33 2011 RBAL started with pid=16, OS id=7293 Fri May 13 17:22:33 2011 GMON started with pid=17, OS id=7295 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 2) ASM instance List of nodes: 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Fri May 13 17:22:34 2011 LCK0 started with pid=18, OS id=7300 ORACLE_BASE from environment = /u01/app/oracle Fri May 13 17:22:35 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x2ea8522a NOTE: cache began mount (first) of group ORAFILES number=1 incarn=0x2ea8522a WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: start heartbeating (grp 1) kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (first) group 1/0x2EA8522A (ORAFILES) * allocate domain 1, invalid = TRUE NOTE: attached to recovery domain 1 NOTE: starting recovery of thread=1 ckpt=14.6846 group=1 NOTE: advancing ckpt for thread=1 ckpt=14.6846 NOTE: cache recovered group 1 to fcn 0.993617 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.993617 ABA NOTE: seq=15 blk=6847 NOTE: cache mounting group 1/0x2EA8522A (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x2ea8522a kfdp_query(): 4 kfdp_queryBg(): 4 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Fri May 13 17:23:46 2011 Reconfiguration started (old inc 2, new inc 4) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Fri May 13 17:23:46 2011 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri May 13 17:23:46 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 9 GCS shadows traversed, 9 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Fri May 13 18:58:19 2011 Trace dumping is performing id=[cdmp_20110513185819] Fri May 13 19:06:00 2011 Starting background process ASMB Fri May 13 19:06:00 2011 ASMB started with pid=20, OS id=25274 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Fri May 13 19:14:01 2011 Starting background process ASMB Fri May 13 19:14:01 2011 ASMB started with pid=20, OS id=29801 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Fri May 13 19:35:31 2011 Starting background process ASMB Fri May 13 19:35:31 2011 ASMB started with pid=20, OS id=9716 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon May 16 22:08:44 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Mon May 16 22:08:46 2011 PMON started with pid=2, OS id=7315 Mon May 16 22:08:46 2011 VKTM started with pid=3, OS id=7317 at elevated priority VKTM running at (20)ms precision Mon May 16 22:08:46 2011 DIAG started with pid=4, OS id=7321 Mon May 16 22:08:46 2011 PING started with pid=5, OS id=7323 Mon May 16 22:08:46 2011 PSP0 started with pid=6, OS id=7325 Mon May 16 22:08:46 2011 DIA0 started with pid=7, OS id=7327 Mon May 16 22:08:46 2011 LMON started with pid=8, OS id=7329 Mon May 16 22:08:46 2011 LMD0 started with pid=9, OS id=7331 Mon May 16 22:08:46 2011 LMS0 started with pid=10, OS id=7333 at elevated priority Mon May 16 22:08:46 2011 MMAN started with pid=11, OS id=7337 Mon May 16 22:08:46 2011 DBW0 started with pid=12, OS id=7339 Mon May 16 22:08:46 2011 LGWR started with pid=13, OS id=7341 Mon May 16 22:08:46 2011 CKPT started with pid=14, OS id=7343 Mon May 16 22:08:46 2011 SMON started with pid=15, OS id=7345 Mon May 16 22:08:46 2011 RBAL started with pid=16, OS id=7347 Mon May 16 22:08:46 2011 GMON started with pid=17, OS id=7349 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 8) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Mon May 16 22:08:47 2011 LCK0 started with pid=18, OS id=7360 ORACLE_BASE from environment = /u01/app/oracle Mon May 16 22:08:48 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x2ea85892 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x2ea85892 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x2EA85892 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.1873796 ABA NOTE: seq=17 blk=5608 NOTE: cache mounting group 1/0x2EA85892 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x2ea85892 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Mon May 16 22:08:59 2011 Starting background process ASMB Mon May 16 22:08:59 2011 ASMB started with pid=20, OS id=7562 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Fri May 20 22:02:25 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Fri May 20 22:02:26 2011 PMON started with pid=2, OS id=21198 Fri May 20 22:02:26 2011 VKTM started with pid=3, OS id=21222 at elevated priority VKTM running at (20)ms precision Fri May 20 22:02:26 2011 DIAG started with pid=4, OS id=21283 Fri May 20 22:02:26 2011 PING started with pid=5, OS id=21308 Fri May 20 22:02:26 2011 PSP0 started with pid=6, OS id=21329 Fri May 20 22:02:26 2011 DIA0 started with pid=7, OS id=21355 Fri May 20 22:02:26 2011 LMON started with pid=8, OS id=21381 Fri May 20 22:02:26 2011 LMD0 started with pid=9, OS id=21400 Fri May 20 22:02:26 2011 LMS0 started with pid=10, OS id=21434 at elevated priority Fri May 20 22:02:26 2011 MMAN started with pid=11, OS id=21463 Fri May 20 22:02:26 2011 DBW0 started with pid=12, OS id=21483 Fri May 20 22:02:26 2011 LGWR started with pid=13, OS id=21493 Fri May 20 22:02:26 2011 CKPT started with pid=14, OS id=21495 Fri May 20 22:02:27 2011 SMON started with pid=15, OS id=21501 Fri May 20 22:02:27 2011 RBAL started with pid=16, OS id=21506 Fri May 20 22:02:27 2011 GMON started with pid=17, OS id=21509 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 12) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Fri May 20 22:02:27 2011 LCK0 started with pid=18, OS id=21857 ORACLE_BASE from environment = /u01/app/oracle Fri May 20 22:02:28 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x2eac886f NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x2eac886f WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x2EAC886F (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.2097807 ABA NOTE: seq=18 blk=8338 NOTE: cache mounting group 1/0x2EAC886F (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x2eac886f kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Fri May 20 22:02:38 2011 Starting background process ASMB Fri May 20 22:02:38 2011 ASMB started with pid=20, OS id=30418 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed Jun 08 11:53:16 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Wed Jun 08 11:53:17 2011 PMON started with pid=2, OS id=19804 Wed Jun 08 11:53:17 2011 VKTM started with pid=3, OS id=19806 at elevated priority VKTM running at (20)ms precision Wed Jun 08 11:53:17 2011 DIAG started with pid=4, OS id=19810 Wed Jun 08 11:53:17 2011 PING started with pid=5, OS id=19812 Wed Jun 08 11:53:17 2011 PSP0 started with pid=6, OS id=19814 Wed Jun 08 11:53:17 2011 DIA0 started with pid=7, OS id=19816 Wed Jun 08 11:53:17 2011 LMON started with pid=8, OS id=19818 Wed Jun 08 11:53:17 2011 LMD0 started with pid=9, OS id=19820 Wed Jun 08 11:53:17 2011 LMS0 started with pid=10, OS id=19822 at elevated priority Wed Jun 08 11:53:17 2011 MMAN started with pid=11, OS id=19826 Wed Jun 08 11:53:17 2011 DBW0 started with pid=12, OS id=19828 Wed Jun 08 11:53:17 2011 LGWR started with pid=13, OS id=19830 Wed Jun 08 11:53:17 2011 CKPT started with pid=14, OS id=19832 Wed Jun 08 11:53:18 2011 SMON started with pid=15, OS id=19834 Wed Jun 08 11:53:18 2011 RBAL started with pid=16, OS id=19836 Wed Jun 08 11:53:18 2011 GMON started with pid=17, OS id=19838 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 16) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Jun 08 11:53:19 2011 LCK0 started with pid=18, OS id=19855 ORACLE_BASE from environment = /u01/app/oracle Wed Jun 08 11:53:21 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x2eafaaa5 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x2eafaaa5 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x2EAFAAA5 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.3211657 ABA NOTE: seq=20 blk=167 NOTE: cache mounting group 1/0x2EAFAAA5 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x2eafaaa5 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Wed Jun 08 11:53:32 2011 Starting background process ASMB Wed Jun 08 11:53:32 2011 ASMB started with pid=20, OS id=20392 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Jun 09 10:17:59 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Thu Jun 09 10:18:00 2011 PMON started with pid=2, OS id=21780 Thu Jun 09 10:18:00 2011 VKTM started with pid=3, OS id=21782 at elevated priority VKTM running at (20)ms precision Thu Jun 09 10:18:00 2011 DIAG started with pid=4, OS id=21786 Thu Jun 09 10:18:00 2011 PING started with pid=5, OS id=21788 Thu Jun 09 10:18:00 2011 PSP0 started with pid=6, OS id=21790 Thu Jun 09 10:18:00 2011 DIA0 started with pid=7, OS id=21792 Thu Jun 09 10:18:00 2011 LMON started with pid=8, OS id=21794 Thu Jun 09 10:18:00 2011 LMD0 started with pid=9, OS id=21796 Thu Jun 09 10:18:00 2011 LMS0 started with pid=10, OS id=21798 at elevated priority Thu Jun 09 10:18:00 2011 MMAN started with pid=11, OS id=21802 Thu Jun 09 10:18:00 2011 DBW0 started with pid=12, OS id=21804 Thu Jun 09 10:18:00 2011 LGWR started with pid=13, OS id=21806 Thu Jun 09 10:18:00 2011 CKPT started with pid=14, OS id=21808 Thu Jun 09 10:18:00 2011 SMON started with pid=15, OS id=21810 Thu Jun 09 10:18:00 2011 RBAL started with pid=16, OS id=21812 Thu Jun 09 10:18:00 2011 GMON started with pid=17, OS id=21817 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 20) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Thu Jun 09 10:18:01 2011 LCK0 started with pid=18, OS id=21830 ORACLE_BASE from environment = /u01/app/oracle Thu Jun 09 10:18:02 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x2eaa49eb NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x2eaa49eb WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x2EAA49EB (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.8196112 ABA NOTE: seq=21 blk=485 NOTE: cache mounting group 1/0x2EAA49EB (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x2eaa49eb kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Thu Jun 09 10:18:08 2011 Starting background process ASMB Thu Jun 09 10:18:08 2011 ASMB started with pid=20, OS id=21965 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Jun 09 10:27:14 2011 Starting background process ASMB Thu Jun 09 10:27:14 2011 ASMB started with pid=20, OS id=28187 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Jun 09 10:58:26 2011 Starting background process ASMB Thu Jun 09 10:58:26 2011 ASMB started with pid=20, OS id=17788 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Jun 09 16:03:08 2011 Shutting down instance: further logons disabled Shutting down instance (immediate) License high water mark = 5 SQL> ALTER DISKGROUP ALL DISMOUNT NOTE: cache dismounting group 1/0x2EAA49EB (ORAFILES) Thu Jun 09 16:03:09 2011 kjbdomdet send to node 0 detach from dom 1, sending detach message to node 0 Thu Jun 09 16:03:09 2011 NOTE: detached from domain 1 NOTE: cache dismounted group 1/0x2EAA49EB (ORAFILES) kfdp_dismount(): 4 Thu Jun 09 16:03:09 2011 kfdp_dismountBg(): 4 NOTE: De-assigning number (1,0) from disk (/dev/mpath/mpath1p1) NOTE: De-assigning number (1,1) from disk (/dev/mpath/mpath2p1) SUCCESS: diskgroup ORAFILES was dismounted SUCCESS: ALTER DISKGROUP ALL DISMOUNT Thu Jun 09 16:03:10 2011 Stopping background process VKTM: Thu Jun 09 16:03:12 2011 freeing rdom 1 freeing rdom 0 Thu Jun 09 16:03:13 2011 Instance shutdown complete Thu Jun 09 16:19:48 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Thu Jun 09 16:19:50 2011 PMON started with pid=2, OS id=8133 Thu Jun 09 16:19:50 2011 VKTM started with pid=3, OS id=8135 at elevated priority VKTM running at (20)ms precision Thu Jun 09 16:19:50 2011 DIAG started with pid=4, OS id=8139 Thu Jun 09 16:19:50 2011 PING started with pid=5, OS id=8141 Thu Jun 09 16:19:50 2011 PSP0 started with pid=6, OS id=8143 Thu Jun 09 16:19:50 2011 DIA0 started with pid=7, OS id=8145 Thu Jun 09 16:19:50 2011 LMON started with pid=8, OS id=8147 Thu Jun 09 16:19:50 2011 LMD0 started with pid=9, OS id=8150 Thu Jun 09 16:19:50 2011 LMS0 started with pid=10, OS id=8157 at elevated priority Thu Jun 09 16:19:50 2011 MMAN started with pid=11, OS id=8161 Thu Jun 09 16:19:50 2011 DBW0 started with pid=12, OS id=8163 Thu Jun 09 16:19:50 2011 LGWR started with pid=13, OS id=8165 Thu Jun 09 16:19:50 2011 CKPT started with pid=14, OS id=8167 Thu Jun 09 16:19:50 2011 SMON started with pid=15, OS id=8169 Thu Jun 09 16:19:50 2011 RBAL started with pid=16, OS id=8171 Thu Jun 09 16:19:50 2011 GMON started with pid=17, OS id=8173 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 2) ASM instance List of nodes: 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Thu Jun 09 16:19:51 2011 LCK0 started with pid=18, OS id=8178 ORACLE_BASE from environment = /u01/app/oracle Thu Jun 09 16:19:51 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58e864c6 NOTE: cache began mount (first) of group ORAFILES number=1 incarn=0x58e864c6 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: start heartbeating (grp 1) kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (first) group 1/0x58E864C6 (ORAFILES) * allocate domain 1, invalid = TRUE NOTE: attached to recovery domain 1 NOTE: starting recovery of thread=2 ckpt=36.4810 group=1 NOTE: advancing ckpt for thread=2 ckpt=36.4810 NOTE: cache recovered group 1 to fcn 0.8974608 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.8974006 ABA NOTE: seq=22 blk=1424 NOTE: cache mounting group 1/0x58E864C6 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58e864c6 kfdp_query(): 4 kfdp_queryBg(): 4 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Thu Jun 09 16:20:07 2011 Starting background process ASMB Thu Jun 09 16:20:07 2011 ASMB started with pid=20, OS id=8643 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Jun 09 16:30:12 2011 Reconfiguration started (old inc 2, new inc 4) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Thu Jun 09 16:30:12 2011 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Thu Jun 09 16:30:12 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 2468 GCS shadows traversed, 1496 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Jun 11 03:48:45 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sat Jun 11 03:48:47 2011 PMON started with pid=2, OS id=5011 Sat Jun 11 03:48:47 2011 VKTM started with pid=3, OS id=5013 at elevated priority VKTM running at (20)ms precision Sat Jun 11 03:48:47 2011 DIAG started with pid=4, OS id=5017 Sat Jun 11 03:48:47 2011 PING started with pid=5, OS id=5019 Sat Jun 11 03:48:47 2011 PSP0 started with pid=6, OS id=5021 Sat Jun 11 03:48:47 2011 DIA0 started with pid=7, OS id=5023 Sat Jun 11 03:48:47 2011 LMON started with pid=8, OS id=5025 Sat Jun 11 03:48:47 2011 LMD0 started with pid=9, OS id=5027 Sat Jun 11 03:48:47 2011 LMS0 started with pid=10, OS id=5029 at elevated priority Sat Jun 11 03:48:47 2011 MMAN started with pid=11, OS id=5033 Sat Jun 11 03:48:47 2011 DBW0 started with pid=12, OS id=5035 Sat Jun 11 03:48:47 2011 LGWR started with pid=13, OS id=5037 Sat Jun 11 03:48:47 2011 CKPT started with pid=14, OS id=5039 Sat Jun 11 03:48:47 2011 SMON started with pid=15, OS id=5041 Sat Jun 11 03:48:47 2011 RBAL started with pid=16, OS id=5043 Sat Jun 11 03:48:47 2011 GMON started with pid=17, OS id=5045 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 8) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Jun 11 03:48:47 2011 LCK0 started with pid=18, OS id=5060 ORACLE_BASE from environment = /u01/app/oracle Sat Jun 11 03:48:48 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58e59031 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58e59031 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58E59031 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.9102479 ABA NOTE: seq=23 blk=1674 NOTE: cache mounting group 1/0x58E59031 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58e59031 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sat Jun 11 03:48:56 2011 Starting background process ASMB Sat Jun 11 03:48:56 2011 ASMB started with pid=19, OS id=5353 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Jun 12 03:15:01 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sun Jun 12 03:15:03 2011 PMON started with pid=2, OS id=29010 Sun Jun 12 03:15:03 2011 VKTM started with pid=3, OS id=29012 at elevated priority VKTM running at (20)ms precision Sun Jun 12 03:15:03 2011 DIAG started with pid=4, OS id=29020 Sun Jun 12 03:15:03 2011 PING started with pid=5, OS id=29022 Sun Jun 12 03:15:03 2011 PSP0 started with pid=6, OS id=29024 Sun Jun 12 03:15:03 2011 DIA0 started with pid=7, OS id=29026 Sun Jun 12 03:15:03 2011 LMON started with pid=8, OS id=29028 Sun Jun 12 03:15:03 2011 LMD0 started with pid=9, OS id=29030 Sun Jun 12 03:15:03 2011 LMS0 started with pid=10, OS id=29032 at elevated priority Sun Jun 12 03:15:03 2011 MMAN started with pid=11, OS id=29036 Sun Jun 12 03:15:03 2011 DBW0 started with pid=12, OS id=29038 Sun Jun 12 03:15:03 2011 LGWR started with pid=13, OS id=29040 Sun Jun 12 03:15:03 2011 CKPT started with pid=14, OS id=29042 Sun Jun 12 03:15:03 2011 SMON started with pid=15, OS id=29044 Sun Jun 12 03:15:03 2011 RBAL started with pid=16, OS id=29046 Sun Jun 12 03:15:03 2011 GMON started with pid=17, OS id=29048 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 12) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sun Jun 12 03:15:03 2011 LCK0 started with pid=18, OS id=29064 ORACLE_BASE from environment = /u01/app/oracle Sun Jun 12 03:15:05 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58ea1653 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58ea1653 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58EA1653 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.10239211 ABA NOTE: seq=25 blk=9029 NOTE: cache mounting group 1/0x58EA1653 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58ea1653 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sun Jun 12 03:15:14 2011 Starting background process ASMB Sun Jun 12 03:15:14 2011 ASMB started with pid=20, OS id=29213 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Jun 13 05:29:17 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Mon Jun 13 05:29:18 2011 PMON started with pid=2, OS id=12701 Mon Jun 13 05:29:18 2011 VKTM started with pid=3, OS id=12703 at elevated priority VKTM running at (20)ms precision Mon Jun 13 05:29:18 2011 DIAG started with pid=4, OS id=12707 Mon Jun 13 05:29:18 2011 PING started with pid=5, OS id=12709 Mon Jun 13 05:29:18 2011 PSP0 started with pid=6, OS id=12711 Mon Jun 13 05:29:18 2011 DIA0 started with pid=7, OS id=12713 Mon Jun 13 05:29:18 2011 LMON started with pid=8, OS id=12715 Mon Jun 13 05:29:18 2011 LMD0 started with pid=9, OS id=12717 Mon Jun 13 05:29:18 2011 LMS0 started with pid=10, OS id=12719 at elevated priority Mon Jun 13 05:29:19 2011 MMAN started with pid=11, OS id=12723 Mon Jun 13 05:29:19 2011 DBW0 started with pid=12, OS id=12725 Mon Jun 13 05:29:19 2011 LGWR started with pid=13, OS id=12728 Mon Jun 13 05:29:19 2011 CKPT started with pid=14, OS id=12730 Mon Jun 13 05:29:19 2011 SMON started with pid=15, OS id=12732 Mon Jun 13 05:29:19 2011 RBAL started with pid=16, OS id=12734 Mon Jun 13 05:29:19 2011 GMON started with pid=17, OS id=12736 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 16) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Mon Jun 13 05:29:19 2011 LCK0 started with pid=18, OS id=12752 ORACLE_BASE from environment = /u01/app/oracle Mon Jun 13 05:29:20 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58e5c505 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58e5c505 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58E5C505 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.10519150 ABA NOTE: seq=27 blk=4310 NOTE: cache mounting group 1/0x58E5C505 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58e5c505 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Mon Jun 13 05:29:29 2011 Starting background process ASMB Mon Jun 13 05:29:29 2011 ASMB started with pid=20, OS id=12904 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Jun 14 23:11:04 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Tue Jun 14 23:11:05 2011 PMON started with pid=2, OS id=28813 Tue Jun 14 23:11:05 2011 VKTM started with pid=3, OS id=28815 at elevated priority VKTM running at (20)ms precision Tue Jun 14 23:11:05 2011 DIAG started with pid=4, OS id=28819 Tue Jun 14 23:11:05 2011 PING started with pid=5, OS id=28821 Tue Jun 14 23:11:05 2011 PSP0 started with pid=6, OS id=28823 Tue Jun 14 23:11:05 2011 DIA0 started with pid=7, OS id=28825 Tue Jun 14 23:11:05 2011 LMON started with pid=8, OS id=28827 Tue Jun 14 23:11:05 2011 LMD0 started with pid=9, OS id=28829 Tue Jun 14 23:11:05 2011 LMS0 started with pid=10, OS id=28831 at elevated priority Tue Jun 14 23:11:05 2011 MMAN started with pid=11, OS id=28835 Tue Jun 14 23:11:05 2011 DBW0 started with pid=12, OS id=28837 Tue Jun 14 23:11:05 2011 LGWR started with pid=13, OS id=28839 Tue Jun 14 23:11:05 2011 CKPT started with pid=14, OS id=28841 Tue Jun 14 23:11:05 2011 SMON started with pid=15, OS id=28843 Tue Jun 14 23:11:05 2011 RBAL started with pid=16, OS id=28845 Tue Jun 14 23:11:05 2011 GMON started with pid=17, OS id=28847 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 20) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Jun 14 23:11:06 2011 LCK0 started with pid=18, OS id=28863 ORACLE_BASE from environment = /u01/app/oracle Tue Jun 14 23:11:07 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58ec4a66 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58ec4a66 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58EC4A66 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.11547851 ABA NOTE: seq=28 blk=8768 NOTE: cache mounting group 1/0x58EC4A66 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58ec4a66 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Tue Jun 14 23:11:16 2011 Starting background process ASMB Tue Jun 14 23:11:17 2011 ASMB started with pid=20, OS id=29013 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Fri Jun 17 20:58:22 2011 Reconfiguration started (old inc 20, new inc 22) List of nodes: 1 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Jun 17 20:58:22 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5438 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Fri Jun 17 20:58:22 2011 kfdp_query(): 4 Fri Jun 17 20:58:22 2011 NOTE: PST enabling heartbeating (grp 1) kfdp_queryBg(): 4 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=2 ckpt=49.1471 group=1 NOTE: advancing ckpt for thread=2 ckpt=49.1471 NOTE: smon did instance recovery for domain 1 Fri Jun 17 21:15:39 2011 Reconfiguration started (old inc 22, new inc 24) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Fri Jun 17 21:15:40 2011 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Jun 17 21:15:40 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5438 GCS shadows traversed, 2737 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Jun 21 23:20:58 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Tue Jun 21 23:20:59 2011 PMON started with pid=2, OS id=28680 Tue Jun 21 23:20:59 2011 VKTM started with pid=3, OS id=28682 at elevated priority Tue Jun 21 23:20:59 2011 DIAG started with pid=4, OS id=28686 VKTM running at (20)ms precision Tue Jun 21 23:20:59 2011 PING started with pid=5, OS id=28688 Tue Jun 21 23:20:59 2011 PSP0 started with pid=6, OS id=28690 Tue Jun 21 23:20:59 2011 DIA0 started with pid=7, OS id=28692 Tue Jun 21 23:20:59 2011 LMON started with pid=8, OS id=28695 Tue Jun 21 23:20:59 2011 LMD0 started with pid=9, OS id=28697 Tue Jun 21 23:20:59 2011 LMS0 started with pid=10, OS id=28699 at elevated priority Tue Jun 21 23:20:59 2011 MMAN started with pid=11, OS id=28703 Tue Jun 21 23:20:59 2011 DBW0 started with pid=12, OS id=28705 Tue Jun 21 23:20:59 2011 LGWR started with pid=13, OS id=28707 Tue Jun 21 23:20:59 2011 CKPT started with pid=14, OS id=28709 Tue Jun 21 23:20:59 2011 SMON started with pid=15, OS id=28711 Tue Jun 21 23:20:59 2011 RBAL started with pid=16, OS id=28713 Tue Jun 21 23:20:59 2011 GMON started with pid=17, OS id=28715 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 28) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Jun 21 23:21:00 2011 LCK0 started with pid=18, OS id=28731 ORACLE_BASE from environment = /u01/app/oracle Tue Jun 21 23:21:01 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58ec4b83 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58ec4b83 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58EC4B83 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.14321757 ABA NOTE: seq=35 blk=339 NOTE: cache mounting group 1/0x58EC4B83 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58ec4b83 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Tue Jun 21 23:21:10 2011 Starting background process ASMB Tue Jun 21 23:21:10 2011 ASMB started with pid=20, OS id=28880 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed Jun 22 16:04:51 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Wed Jun 22 16:04:52 2011 PMON started with pid=2, OS id=21897 Wed Jun 22 16:04:52 2011 VKTM started with pid=3, OS id=21899 at elevated priority VKTM running at (20)ms precision Wed Jun 22 16:04:52 2011 DIAG started with pid=4, OS id=21903 Wed Jun 22 16:04:52 2011 PING started with pid=5, OS id=21905 Wed Jun 22 16:04:52 2011 PSP0 started with pid=6, OS id=21907 Wed Jun 22 16:04:52 2011 DIA0 started with pid=7, OS id=21909 Wed Jun 22 16:04:52 2011 LMON started with pid=8, OS id=21911 Wed Jun 22 16:04:52 2011 LMD0 started with pid=9, OS id=21913 Wed Jun 22 16:04:52 2011 LMS0 started with pid=10, OS id=21918 at elevated priority Wed Jun 22 16:04:52 2011 MMAN started with pid=11, OS id=21924 Wed Jun 22 16:04:52 2011 DBW0 started with pid=12, OS id=21926 Wed Jun 22 16:04:52 2011 LGWR started with pid=13, OS id=21928 Wed Jun 22 16:04:52 2011 CKPT started with pid=14, OS id=21930 Wed Jun 22 16:04:53 2011 SMON started with pid=15, OS id=21932 Wed Jun 22 16:04:53 2011 RBAL started with pid=16, OS id=21934 Wed Jun 22 16:04:53 2011 GMON started with pid=17, OS id=21936 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 32) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Jun 22 16:04:53 2011 LCK0 started with pid=18, OS id=21947 ORACLE_BASE from environment = /u01/app/oracle Wed Jun 22 16:04:54 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58eb99f4 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58eb99f4 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58EB99F4 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.14677997 ABA NOTE: seq=36 blk=3775 NOTE: cache mounting group 1/0x58EB99F4 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58eb99f4 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Wed Jun 22 16:05:04 2011 Starting background process ASMB Wed Jun 22 16:05:04 2011 ASMB started with pid=20, OS id=22095 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed Jun 22 20:31:13 2011 Reconfiguration started (old inc 32, new inc 34) List of nodes: 1 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Jun 22 20:31:13 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 3018 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Wed Jun 22 20:31:13 2011 kfdp_query(): 4 Wed Jun 22 20:31:13 2011 NOTE: PST enabling heartbeating (grp 1) kfdp_queryBg(): 4 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=2 ckpt=54.6428 group=1 NOTE: advancing ckpt for thread=2 ckpt=54.6428 NOTE: smon did instance recovery for domain 1 Wed Jun 22 20:36:11 2011 Reconfiguration started (old inc 34, new inc 36) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Wed Jun 22 20:36:11 2011 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Jun 22 20:36:11 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 3018 GCS shadows traversed, 1748 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Thu Jun 23 16:37:46 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Thu Jun 23 16:37:47 2011 PMON started with pid=2, OS id=23937 Thu Jun 23 16:37:47 2011 VKTM started with pid=3, OS id=23939 at elevated priority VKTM running at (20)ms precision Thu Jun 23 16:37:47 2011 DIAG started with pid=4, OS id=23943 Thu Jun 23 16:37:47 2011 PING started with pid=5, OS id=23945 Thu Jun 23 16:37:47 2011 PSP0 started with pid=6, OS id=23947 Thu Jun 23 16:37:47 2011 DIA0 started with pid=7, OS id=23949 Thu Jun 23 16:37:47 2011 LMON started with pid=8, OS id=23951 Thu Jun 23 16:37:47 2011 LMD0 started with pid=9, OS id=23953 Thu Jun 23 16:37:47 2011 LMS0 started with pid=10, OS id=23955 at elevated priority Thu Jun 23 16:37:47 2011 MMAN started with pid=11, OS id=23962 Thu Jun 23 16:37:47 2011 DBW0 started with pid=12, OS id=23966 Thu Jun 23 16:37:47 2011 LGWR started with pid=13, OS id=23968 Thu Jun 23 16:37:47 2011 CKPT started with pid=14, OS id=23970 Thu Jun 23 16:37:47 2011 SMON started with pid=15, OS id=23972 Thu Jun 23 16:37:47 2011 RBAL started with pid=16, OS id=23974 Thu Jun 23 16:37:47 2011 GMON started with pid=17, OS id=23976 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 40) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Thu Jun 23 16:37:48 2011 LCK0 started with pid=18, OS id=23988 ORACLE_BASE from environment = /u01/app/oracle Thu Jun 23 16:37:49 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58ebd4e5 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58ebd4e5 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58EBD4E5 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.15020338 ABA NOTE: seq=37 blk=7504 NOTE: cache mounting group 1/0x58EBD4E5 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58ebd4e5 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Thu Jun 23 16:37:59 2011 Starting background process ASMB Thu Jun 23 16:37:59 2011 ASMB started with pid=20, OS id=24141 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Jun 23 21:41:08 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Thu Jun 23 21:41:09 2011 PMON started with pid=2, OS id=20061 Thu Jun 23 21:41:09 2011 VKTM started with pid=3, OS id=20063 at elevated priority VKTM running at (20)ms precision Thu Jun 23 21:41:09 2011 DIAG started with pid=4, OS id=20067 Thu Jun 23 21:41:09 2011 PING started with pid=5, OS id=20069 Thu Jun 23 21:41:09 2011 PSP0 started with pid=6, OS id=20071 Thu Jun 23 21:41:09 2011 DIA0 started with pid=7, OS id=20073 Thu Jun 23 21:41:09 2011 LMON started with pid=8, OS id=20075 Thu Jun 23 21:41:09 2011 LMD0 started with pid=9, OS id=20077 Thu Jun 23 21:41:09 2011 LMS0 started with pid=10, OS id=20079 at elevated priority Thu Jun 23 21:41:09 2011 MMAN started with pid=11, OS id=20083 Thu Jun 23 21:41:09 2011 DBW0 started with pid=12, OS id=20085 Thu Jun 23 21:41:09 2011 LGWR started with pid=13, OS id=20087 Thu Jun 23 21:41:09 2011 CKPT started with pid=14, OS id=20089 Thu Jun 23 21:41:09 2011 SMON started with pid=15, OS id=20091 Thu Jun 23 21:41:09 2011 RBAL started with pid=16, OS id=20093 Thu Jun 23 21:41:10 2011 GMON started with pid=17, OS id=20095 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 44) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Thu Jun 23 21:41:10 2011 LCK0 started with pid=18, OS id=20117 ORACLE_BASE from environment = /u01/app/oracle Thu Jun 23 21:41:11 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58e940d8 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58e940d8 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58E940D8 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.15024224 ABA NOTE: seq=38 blk=7577 NOTE: cache mounting group 1/0x58E940D8 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58e940d8 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Thu Jun 23 21:41:22 2011 Starting background process ASMB Thu Jun 23 21:41:22 2011 ASMB started with pid=20, OS id=20615 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Fri Jun 24 22:25:19 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Fri Jun 24 22:25:21 2011 PMON started with pid=2, OS id=8716 Fri Jun 24 22:25:21 2011 VKTM started with pid=3, OS id=8718 at elevated priority Fri Jun 24 22:25:21 2011 DIAG started with pid=4, OS id=8722 Fri Jun 24 22:25:21 2011 PING started with pid=5, OS id=8724 VKTM running at (20)ms precision Fri Jun 24 22:25:21 2011 PSP0 started with pid=6, OS id=8726 Fri Jun 24 22:25:21 2011 DIA0 started with pid=7, OS id=8728 Fri Jun 24 22:25:21 2011 LMON started with pid=8, OS id=8730 Fri Jun 24 22:25:21 2011 LMD0 started with pid=9, OS id=8732 Fri Jun 24 22:25:21 2011 LMS0 started with pid=10, OS id=8734 at elevated priority Fri Jun 24 22:25:21 2011 MMAN started with pid=11, OS id=8738 Fri Jun 24 22:25:21 2011 DBW0 started with pid=12, OS id=8740 Fri Jun 24 22:25:21 2011 LGWR started with pid=13, OS id=8742 Fri Jun 24 22:25:21 2011 CKPT started with pid=14, OS id=8744 Fri Jun 24 22:25:21 2011 SMON started with pid=15, OS id=8746 Fri Jun 24 22:25:21 2011 RBAL started with pid=16, OS id=8748 Fri Jun 24 22:25:21 2011 GMON started with pid=17, OS id=8750 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 48) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Fri Jun 24 22:25:22 2011 LCK0 started with pid=18, OS id=8802 ORACLE_BASE from environment = /u01/app/oracle Fri Jun 24 22:25:23 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58e86450 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58e86450 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58E86450 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.15419752 ABA NOTE: seq=39 blk=10533 NOTE: cache mounting group 1/0x58E86450 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58e86450 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Fri Jun 24 22:25:33 2011 Starting background process ASMB Fri Jun 24 22:25:33 2011 ASMB started with pid=20, OS id=9102 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Jun 25 06:31:01 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sat Jun 25 06:31:03 2011 PMON started with pid=2, OS id=8754 Sat Jun 25 06:31:04 2011 VKTM started with pid=3, OS id=8765 at elevated priority Sat Jun 25 06:31:04 2011 DIAG started with pid=4, OS id=8785 VKTM running at (20)ms precision Sat Jun 25 06:31:04 2011 PING started with pid=5, OS id=8787 Sat Jun 25 06:31:04 2011 PSP0 started with pid=6, OS id=8789 Sat Jun 25 06:31:04 2011 DIA0 started with pid=7, OS id=8792 Sat Jun 25 06:31:04 2011 LMON started with pid=8, OS id=8794 Sat Jun 25 06:31:04 2011 LMD0 started with pid=9, OS id=8796 Sat Jun 25 06:31:04 2011 LMS0 started with pid=10, OS id=8798 at elevated priority Sat Jun 25 06:31:04 2011 MMAN started with pid=11, OS id=8802 Sat Jun 25 06:31:04 2011 DBW0 started with pid=12, OS id=8804 Sat Jun 25 06:31:04 2011 LGWR started with pid=13, OS id=8806 Sat Jun 25 06:31:04 2011 CKPT started with pid=14, OS id=8808 Sat Jun 25 06:31:04 2011 SMON started with pid=15, OS id=8810 Sat Jun 25 06:31:04 2011 RBAL started with pid=16, OS id=8812 Sat Jun 25 06:31:04 2011 GMON started with pid=17, OS id=8814 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 52) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Jun 25 06:31:04 2011 LCK0 started with pid=18, OS id=8864 ORACLE_BASE from environment = /u01/app/oracle Sat Jun 25 06:31:06 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58e862f8 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58e862f8 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58E862F8 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.15720045 ABA NOTE: seq=41 blk=4371 NOTE: cache mounting group 1/0x58E862F8 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58e862f8 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sat Jun 25 06:31:18 2011 Starting background process ASMB Sat Jun 25 06:31:18 2011 ASMB started with pid=20, OS id=9360 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Jun 27 12:29:51 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Mon Jun 27 12:29:52 2011 PMON started with pid=2, OS id=23771 Mon Jun 27 12:29:52 2011 VKTM started with pid=3, OS id=23773 at elevated priority VKTM running at (20)ms precision Mon Jun 27 12:29:52 2011 DIAG started with pid=4, OS id=23777 Mon Jun 27 12:29:52 2011 PING started with pid=5, OS id=23779 Mon Jun 27 12:29:52 2011 PSP0 started with pid=6, OS id=23781 Mon Jun 27 12:29:52 2011 DIA0 started with pid=7, OS id=23783 Mon Jun 27 12:29:52 2011 LMON started with pid=8, OS id=23785 Mon Jun 27 12:29:52 2011 LMD0 started with pid=9, OS id=23787 Mon Jun 27 12:29:52 2011 LMS0 started with pid=10, OS id=23789 at elevated priority Mon Jun 27 12:29:53 2011 MMAN started with pid=11, OS id=23793 Mon Jun 27 12:29:53 2011 DBW0 started with pid=12, OS id=23795 Mon Jun 27 12:29:53 2011 LGWR started with pid=13, OS id=23797 Mon Jun 27 12:29:53 2011 CKPT started with pid=14, OS id=23799 Mon Jun 27 12:29:53 2011 SMON started with pid=15, OS id=23801 Mon Jun 27 12:29:53 2011 RBAL started with pid=16, OS id=23803 Mon Jun 27 12:29:53 2011 GMON started with pid=17, OS id=23805 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 56) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Mon Jun 27 12:29:53 2011 LCK0 started with pid=18, OS id=23821 ORACLE_BASE from environment = /u01/app/oracle Mon Jun 27 12:29:54 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58efc17b NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58efc17b WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58EFC17B (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.16437044 ABA NOTE: seq=43 blk=2815 NOTE: cache mounting group 1/0x58EFC17B (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58efc17b kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Mon Jun 27 12:30:05 2011 Starting background process ASMB Mon Jun 27 12:30:05 2011 ASMB started with pid=20, OS id=24101 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Jun 27 17:37:50 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Mon Jun 27 17:37:51 2011 PMON started with pid=2, OS id=28087 Mon Jun 27 17:37:51 2011 VKTM started with pid=3, OS id=28089 at elevated priority VKTM running at (20)ms precision Mon Jun 27 17:37:51 2011 DIAG started with pid=4, OS id=28099 Mon Jun 27 17:37:51 2011 PING started with pid=5, OS id=28101 Mon Jun 27 17:37:51 2011 PSP0 started with pid=6, OS id=28103 Mon Jun 27 17:37:51 2011 DIA0 started with pid=7, OS id=28105 Mon Jun 27 17:37:51 2011 LMON started with pid=8, OS id=28107 Mon Jun 27 17:37:51 2011 LMD0 started with pid=9, OS id=28109 Mon Jun 27 17:37:51 2011 LMS0 started with pid=10, OS id=28111 at elevated priority Mon Jun 27 17:37:51 2011 MMAN started with pid=11, OS id=28115 Mon Jun 27 17:37:51 2011 DBW0 started with pid=12, OS id=28117 Mon Jun 27 17:37:51 2011 LGWR started with pid=13, OS id=28119 Mon Jun 27 17:37:51 2011 CKPT started with pid=14, OS id=28121 Mon Jun 27 17:37:51 2011 SMON started with pid=15, OS id=28123 Mon Jun 27 17:37:51 2011 RBAL started with pid=16, OS id=28125 Mon Jun 27 17:37:51 2011 GMON started with pid=17, OS id=28127 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 60) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Mon Jun 27 17:37:52 2011 LCK0 started with pid=18, OS id=28137 ORACLE_BASE from environment = /u01/app/oracle Mon Jun 27 17:37:53 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58e2336b NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58e2336b WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58E2336B (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.16456799 ABA NOTE: seq=44 blk=2922 NOTE: cache mounting group 1/0x58E2336B (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58e2336b kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Mon Jun 27 17:38:02 2011 Starting background process ASMB Mon Jun 27 17:38:02 2011 ASMB started with pid=20, OS id=28288 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Jun 27 18:06:06 2011 Reconfiguration started (old inc 60, new inc 62) List of nodes: 1 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Mon Jun 27 18:06:06 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 2760 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Mon Jun 27 18:06:06 2011 kfdp_query(): 4 Mon Jun 27 18:06:06 2011 NOTE: PST enabling heartbeating (grp 1) kfdp_queryBg(): 4 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=2 ckpt=60.1923 group=1 NOTE: advancing ckpt for thread=2 ckpt=60.1923 NOTE: smon did instance recovery for domain 1 Mon Jun 27 19:06:28 2011 Reconfiguration started (old inc 62, new inc 64) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Mon Jun 27 19:06:28 2011 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Mon Jun 27 19:06:28 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 2760 GCS shadows traversed, 1616 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Jun 28 15:26:01 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Tue Jun 28 15:26:02 2011 PMON started with pid=2, OS id=8931 Tue Jun 28 15:26:02 2011 VKTM started with pid=3, OS id=8933 at elevated priority Tue Jun 28 15:26:02 2011 DIAG started with pid=4, OS id=8937 VKTM running at (20)ms precision Tue Jun 28 15:26:02 2011 PING started with pid=5, OS id=8939 Tue Jun 28 15:26:02 2011 PSP0 started with pid=6, OS id=8941 Tue Jun 28 15:26:02 2011 DIA0 started with pid=7, OS id=8943 Tue Jun 28 15:26:02 2011 LMON started with pid=8, OS id=8945 Tue Jun 28 15:26:02 2011 LMD0 started with pid=9, OS id=8947 Tue Jun 28 15:26:02 2011 LMS0 started with pid=10, OS id=8949 at elevated priority Tue Jun 28 15:26:02 2011 MMAN started with pid=11, OS id=8953 Tue Jun 28 15:26:02 2011 DBW0 started with pid=12, OS id=8955 Tue Jun 28 15:26:02 2011 LGWR started with pid=13, OS id=8957 Tue Jun 28 15:26:02 2011 CKPT started with pid=14, OS id=8959 Tue Jun 28 15:26:02 2011 SMON started with pid=15, OS id=8961 Tue Jun 28 15:26:02 2011 RBAL started with pid=16, OS id=8963 Tue Jun 28 15:26:02 2011 GMON started with pid=17, OS id=8965 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 68) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Jun 28 15:26:03 2011 LCK0 started with pid=18, OS id=8977 ORACLE_BASE from environment = /u01/app/oracle Tue Jun 28 15:26:04 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58e865a1 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58e865a1 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58E865A1 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.16802495 ABA NOTE: seq=45 blk=6195 NOTE: cache mounting group 1/0x58E865A1 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58e865a1 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Tue Jun 28 15:26:17 2011 Starting background process ASMB Tue Jun 28 15:26:17 2011 ASMB started with pid=20, OS id=9306 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Jun 28 22:50:51 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Tue Jun 28 22:50:53 2011 PMON started with pid=2, OS id=21909 Tue Jun 28 22:50:53 2011 VKTM started with pid=3, OS id=21911 at elevated priority VKTM running at (20)ms precision Tue Jun 28 22:50:53 2011 DIAG started with pid=4, OS id=21915 Tue Jun 28 22:50:53 2011 PING started with pid=5, OS id=21917 Tue Jun 28 22:50:53 2011 PSP0 started with pid=6, OS id=21919 Tue Jun 28 22:50:53 2011 DIA0 started with pid=7, OS id=21921 Tue Jun 28 22:50:53 2011 LMON started with pid=8, OS id=21923 Tue Jun 28 22:50:53 2011 LMD0 started with pid=9, OS id=21925 Tue Jun 28 22:50:53 2011 LMS0 started with pid=10, OS id=21927 at elevated priority Tue Jun 28 22:50:53 2011 MMAN started with pid=11, OS id=21931 Tue Jun 28 22:50:53 2011 DBW0 started with pid=12, OS id=21933 Tue Jun 28 22:50:53 2011 LGWR started with pid=13, OS id=21935 Tue Jun 28 22:50:53 2011 CKPT started with pid=14, OS id=21937 Tue Jun 28 22:50:53 2011 SMON started with pid=15, OS id=21939 Tue Jun 28 22:50:53 2011 RBAL started with pid=16, OS id=21941 Tue Jun 28 22:50:53 2011 GMON started with pid=17, OS id=21943 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 72) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Jun 28 22:50:54 2011 LCK0 started with pid=18, OS id=21960 ORACLE_BASE from environment = /u01/app/oracle Tue Jun 28 22:50:55 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58e9f4b0 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58e9f4b0 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58E9F4B0 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.16833037 ABA NOTE: seq=46 blk=6284 NOTE: cache mounting group 1/0x58E9F4B0 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58e9f4b0 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Tue Jun 28 22:51:04 2011 Starting background process ASMB Tue Jun 28 22:51:04 2011 ASMB started with pid=20, OS id=22106 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed Jun 29 14:22:21 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Wed Jun 29 14:22:23 2011 PMON started with pid=2, OS id=8601 Wed Jun 29 14:22:23 2011 VKTM started with pid=3, OS id=8603 at elevated priority VKTM running at (20)ms precision Wed Jun 29 14:22:23 2011 DIAG started with pid=4, OS id=8607 Wed Jun 29 14:22:23 2011 PING started with pid=5, OS id=8613 Wed Jun 29 14:22:23 2011 PSP0 started with pid=6, OS id=8627 Wed Jun 29 14:22:23 2011 DIA0 started with pid=7, OS id=8629 Wed Jun 29 14:22:23 2011 LMON started with pid=8, OS id=8631 Wed Jun 29 14:22:23 2011 LMD0 started with pid=9, OS id=8633 Wed Jun 29 14:22:23 2011 LMS0 started with pid=10, OS id=8635 at elevated priority Wed Jun 29 14:22:23 2011 MMAN started with pid=11, OS id=8639 Wed Jun 29 14:22:23 2011 DBW0 started with pid=12, OS id=8641 Wed Jun 29 14:22:23 2011 LGWR started with pid=13, OS id=8643 Wed Jun 29 14:22:23 2011 CKPT started with pid=14, OS id=8645 Wed Jun 29 14:22:23 2011 SMON started with pid=15, OS id=8647 Wed Jun 29 14:22:23 2011 RBAL started with pid=16, OS id=8649 Wed Jun 29 14:22:23 2011 GMON started with pid=17, OS id=8651 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 76) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Jun 29 14:22:24 2011 LCK0 started with pid=18, OS id=8703 ORACLE_BASE from environment = /u01/app/oracle Wed Jun 29 14:22:25 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58e8643b NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58e8643b WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58E8643B (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.17158344 ABA NOTE: seq=48 blk=104 NOTE: cache mounting group 1/0x58E8643B (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58e8643b kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Wed Jun 29 14:22:36 2011 Starting background process ASMB Wed Jun 29 14:22:36 2011 ASMB started with pid=20, OS id=9174 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed Jun 29 21:43:04 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Wed Jun 29 21:43:05 2011 PMON started with pid=2, OS id=27715 Wed Jun 29 21:43:05 2011 VKTM started with pid=3, OS id=27717 at elevated priority VKTM running at (20)ms precision Wed Jun 29 21:43:05 2011 DIAG started with pid=4, OS id=27721 Wed Jun 29 21:43:05 2011 PING started with pid=5, OS id=27723 Wed Jun 29 21:43:05 2011 PSP0 started with pid=6, OS id=27725 Wed Jun 29 21:43:05 2011 DIA0 started with pid=7, OS id=27727 Wed Jun 29 21:43:05 2011 LMON started with pid=8, OS id=27729 Wed Jun 29 21:43:05 2011 LMD0 started with pid=9, OS id=27731 Wed Jun 29 21:43:05 2011 LMS0 started with pid=10, OS id=27733 at elevated priority Wed Jun 29 21:43:06 2011 MMAN started with pid=11, OS id=27737 Wed Jun 29 21:43:06 2011 DBW0 started with pid=12, OS id=27739 Wed Jun 29 21:43:06 2011 LGWR started with pid=13, OS id=27741 Wed Jun 29 21:43:06 2011 CKPT started with pid=14, OS id=27743 Wed Jun 29 21:43:06 2011 SMON started with pid=15, OS id=27745 Wed Jun 29 21:43:06 2011 RBAL started with pid=16, OS id=27747 Wed Jun 29 21:43:06 2011 GMON started with pid=17, OS id=27749 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 80) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Jun 29 21:43:06 2011 LCK0 started with pid=18, OS id=27765 ORACLE_BASE from environment = /u01/app/oracle Wed Jun 29 21:43:07 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58ecaad5 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58ecaad5 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58ECAAD5 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.17185779 ABA NOTE: seq=49 blk=190 NOTE: cache mounting group 1/0x58ECAAD5 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58ecaad5 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Wed Jun 29 21:43:17 2011 Starting background process ASMB Wed Jun 29 21:43:17 2011 ASMB started with pid=20, OS id=27938 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Jun 30 22:33:26 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Thu Jun 30 22:33:28 2011 PMON started with pid=2, OS id=8861 Thu Jun 30 22:33:28 2011 VKTM started with pid=3, OS id=8863 at elevated priority Thu Jun 30 22:33:28 2011 DIAG started with pid=4, OS id=8867 VKTM running at (20)ms precision Thu Jun 30 22:33:28 2011 PING started with pid=5, OS id=8869 Thu Jun 30 22:33:28 2011 PSP0 started with pid=6, OS id=8871 Thu Jun 30 22:33:28 2011 DIA0 started with pid=7, OS id=8873 Thu Jun 30 22:33:28 2011 LMON started with pid=8, OS id=8875 Thu Jun 30 22:33:28 2011 LMD0 started with pid=9, OS id=8877 Thu Jun 30 22:33:28 2011 LMS0 started with pid=10, OS id=8879 at elevated priority Thu Jun 30 22:33:28 2011 MMAN started with pid=11, OS id=8883 Thu Jun 30 22:33:29 2011 DBW0 started with pid=12, OS id=8885 Thu Jun 30 22:33:29 2011 LGWR started with pid=13, OS id=8887 Thu Jun 30 22:33:29 2011 CKPT started with pid=14, OS id=8889 Thu Jun 30 22:33:29 2011 SMON started with pid=15, OS id=8891 Thu Jun 30 22:33:29 2011 RBAL started with pid=16, OS id=8893 Thu Jun 30 22:33:29 2011 GMON started with pid=17, OS id=8895 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 84) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Thu Jun 30 22:33:29 2011 LCK0 started with pid=18, OS id=8906 ORACLE_BASE from environment = /u01/app/oracle Thu Jun 30 22:33:30 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58e864e4 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58e864e4 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58E864E4 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.17559695 ABA NOTE: seq=50 blk=5327 NOTE: cache mounting group 1/0x58E864E4 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58e864e4 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Thu Jun 30 22:33:41 2011 Starting background process ASMB Thu Jun 30 22:33:41 2011 ASMB started with pid=20, OS id=9247 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Jul 02 09:11:38 2011 Starting ORACLE instance (normal) WARNING: Deprecated privilege SYSDBA for command 'STARTUP' LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_21761.trc: ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:skgxnqtsz failed with status: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: SKGXN not av clsssinit ret = 21 interconnect information is not available from OCR WARNING: No cluster interconnect has been specified. Depending on the communication driver configured Oracle cluster traffic may be directed to the public interface of this machine. Oracle recommends that RAC clustered databases be configured with a private interconnect for enhanced security and performance. Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Tue Jul 05 18:07:02 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Tue Jul 05 18:07:04 2011 PMON started with pid=2, OS id=9089 Tue Jul 05 18:07:04 2011 VKTM started with pid=3, OS id=9093 at elevated priority VKTM running at (20)ms precision Tue Jul 05 18:07:04 2011 DIAG started with pid=4, OS id=9099 Tue Jul 05 18:07:04 2011 PING started with pid=5, OS id=9101 Tue Jul 05 18:07:04 2011 PSP0 started with pid=6, OS id=9105 Tue Jul 05 18:07:04 2011 DIA0 started with pid=7, OS id=9109 Tue Jul 05 18:07:04 2011 LMON started with pid=8, OS id=9111 Tue Jul 05 18:07:04 2011 LMD0 started with pid=9, OS id=9113 Tue Jul 05 18:07:04 2011 LMS0 started with pid=10, OS id=9115 at elevated priority Tue Jul 05 18:07:04 2011 MMAN started with pid=11, OS id=9119 Tue Jul 05 18:07:04 2011 DBW0 started with pid=12, OS id=9121 Tue Jul 05 18:07:04 2011 LGWR started with pid=13, OS id=9123 Tue Jul 05 18:07:04 2011 CKPT started with pid=14, OS id=9125 Tue Jul 05 18:07:04 2011 SMON started with pid=15, OS id=9127 Tue Jul 05 18:07:04 2011 RBAL started with pid=16, OS id=9129 Tue Jul 05 18:07:04 2011 GMON started with pid=17, OS id=9131 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 2) ASM instance List of nodes: 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Tue Jul 05 18:07:05 2011 LCK0 started with pid=18, OS id=9153 ORACLE_BASE from environment = /u01/app/oracle Tue Jul 05 18:07:06 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc6088b72 NOTE: cache began mount (first) of group ORAFILES number=1 incarn=0xc6088b72 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: start heartbeating (grp 1) kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (first) group 1/0xC6088B72 (ORAFILES) * allocate domain 1, invalid = TRUE NOTE: attached to recovery domain 1 NOTE: starting recovery of thread=2 ckpt=92.6531 group=1 NOTE: advancing ckpt for thread=2 ckpt=92.6531 NOTE: cache recovered group 1 to fcn 0.20046928 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.18234261 ABA NOTE: seq=52 blk=5607 NOTE: cache mounting group 1/0xC6088B72 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc6088b72 kfdp_query(): 4 kfdp_queryBg(): 4 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Tue Jul 05 18:07:22 2011 Starting background process ASMB Tue Jul 05 18:07:22 2011 ASMB started with pid=20, OS id=9831 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Jul 05 18:08:31 2011 Reconfiguration started (old inc 2, new inc 4) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Tue Jul 05 18:08:31 2011 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Jul 05 18:08:31 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 1059 GCS shadows traversed, 812 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Jul 05 18:12:36 2011 Starting background process ASMB Tue Jul 05 18:12:36 2011 ASMB started with pid=20, OS id=14445 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Aug 16 09:02:29 2011 * allocate domain 2, invalid = TRUE freeing rdom 2 Received detach msg from node 0 for dom 2 * allocate domain 2, invalid = TRUE Tue Aug 16 09:02:40 2011 SQL> /* OracleOEM */ alter diskgroup FLASH MOUNT WARNING: Deprecated privilege SYSDBA for command 'ALTER DISKGROUP MOUNT' NOTE: cache registered group FLASH number=2 incarn=0x3a288b76 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a288b76 NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) kfdp_query(): 6 Tue Aug 16 09:02:40 2011 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A288B76 (FLASH) Tue Aug 16 09:02:41 2011 kjbdomatt send to node 0 Tue Aug 16 09:02:41 2011 NOTE: attached to recovery domain 2 Tue Aug 16 09:02:41 2011 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: ACD expansion required for disk group 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.0 ABA NOTE: seq=2 blk=0 NOTE: cache mounting group 2/0x3A288B76 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a288b76 Tue Aug 16 09:02:44 2011 kfdp_query(): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted SUCCESS: /* OracleOEM */ alter diskgroup FLASH MOUNT Mon Aug 29 16:07:28 2011 NOTE: ASM client dwh12:dwh1 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_14549.trc Mon Aug 29 16:07:29 2011 Trace dumping is performing id=[cdmp_20110829160729] Mon Aug 29 16:07:59 2011 Starting background process ASMB Mon Aug 29 16:07:59 2011 ASMB started with pid=20, OS id=27391 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 18 18:27:36 2011 Received detach msg from node 0 for dom 1 Received detach msg from node 0 for dom 2 Sun Sep 18 18:27:37 2011 NOTE: PST enabling heartbeating (grp 2) Sun Sep 18 18:27:41 2011 Reconfiguration started (old inc 4, new inc 6) List of nodes: 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sun Sep 18 18:27:41 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5405 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sun Sep 18 18:49:31 2011 Reconfiguration started (old inc 6, new inc 8) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Sun Sep 18 18:49:31 2011 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 * domain 2 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sun Sep 18 18:49:31 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5440 GCS shadows traversed, 2757 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sun Sep 18 19:23:54 2011 Shutting down instance: further logons disabled Shutting down instance (immediate) License high water mark = 13 SQL> ALTER DISKGROUP ALL DISMOUNT NOTE: cache dismounting group 1/0xC6088B72 (ORAFILES) Sun Sep 18 19:23:54 2011 kjbdomdet send to node 0 detach from dom 1, sending detach message to node 0 Sun Sep 18 19:23:55 2011 NOTE: detached from domain 1 NOTE: cache dismounted group 1/0xC6088B72 (ORAFILES) kfdp_dismount(): 8 Sun Sep 18 19:23:55 2011 kfdp_dismountBg(): 8 NOTE: De-assigning number (1,0) from disk (/dev/mpath/mpath1p1) NOTE: De-assigning number (1,1) from disk (/dev/mpath/mpath2p1) SUCCESS: diskgroup ORAFILES was dismounted NOTE: cache dismounting group 2/0x3A288B76 (FLASH) kjbdomdet send to node 0 detach from dom 2, sending detach message to node 0 NOTE: detached from domain 2 NOTE: cache dismounted group 2/0x3A288B76 (FLASH) kfdp_dismount(): 9 kfdp_dismountBg(): 9 NOTE: De-assigning number (2,0) from disk (/dev/mpath/mpath3p1) SUCCESS: diskgroup FLASH was dismounted SUCCESS: ALTER DISKGROUP ALL DISMOUNT Sun Sep 18 19:23:58 2011 Stopping background process VKTM: Sun Sep 18 19:24:00 2011 freeing rdom 2 freeing rdom 1 freeing rdom 0 Sun Sep 18 19:24:01 2011 Instance shutdown complete Sun Sep 18 19:44:53 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sun Sep 18 19:44:55 2011 PMON started with pid=2, OS id=9779 Sun Sep 18 19:44:55 2011 VKTM started with pid=3, OS id=9781 at elevated priority Sun Sep 18 19:44:55 2011 DIAG started with pid=4, OS id=9785 VKTM running at (20)ms precision Sun Sep 18 19:44:55 2011 PING started with pid=5, OS id=9787 Sun Sep 18 19:44:55 2011 PSP0 started with pid=6, OS id=9789 Sun Sep 18 19:44:55 2011 DIA0 started with pid=7, OS id=9791 Sun Sep 18 19:44:55 2011 LMON started with pid=8, OS id=9793 Sun Sep 18 19:44:55 2011 LMD0 started with pid=9, OS id=9795 Sun Sep 18 19:44:55 2011 LMS0 started with pid=10, OS id=9797 at elevated priority Sun Sep 18 19:44:55 2011 MMAN started with pid=11, OS id=9801 Sun Sep 18 19:44:55 2011 DBW0 started with pid=12, OS id=9803 Sun Sep 18 19:44:55 2011 LGWR started with pid=13, OS id=9811 Sun Sep 18 19:44:55 2011 CKPT started with pid=14, OS id=9813 Sun Sep 18 19:44:55 2011 SMON started with pid=15, OS id=9815 Sun Sep 18 19:44:55 2011 RBAL started with pid=16, OS id=9817 Sun Sep 18 19:44:55 2011 GMON started with pid=17, OS id=9819 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 12) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sun Sep 18 19:44:56 2011 LCK0 started with pid=18, OS id=9824 ORACLE_BASE from environment = /u01/app/oracle Sun Sep 18 19:44:57 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc60881c9 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc60881c9 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0xC60881C9 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.37583097 ABA NOTE: seq=76 blk=7962 NOTE: cache mounting group 1/0xC60881C9 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc60881c9 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sun Sep 18 19:45:07 2011 Starting background process ASMB Sun Sep 18 19:45:07 2011 ASMB started with pid=20, OS id=10199 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Sep 19 02:30:42 2011 SQL> alter diskgroup flash mount WARNING: Deprecated privilege SYSDBA for command 'ALTER DISKGROUP MOUNT' NOTE: cache registered group FLASH number=2 incarn=0x3a2881d1 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a2881d1 NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) kfdp_query(): 5 Mon Sep 19 02:30:43 2011 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A2881D1 (FLASH) Mon Sep 19 02:30:43 2011 kjbdomatt send to node 0 Mon Sep 19 02:30:43 2011 NOTE: attached to recovery domain 2 Mon Sep 19 02:30:43 2011 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.13535918 ABA NOTE: seq=16 blk=7692 NOTE: cache mounting group 2/0x3A2881D1 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a2881d1 Mon Sep 19 02:30:43 2011 kfdp_query(): 6 kfdp_queryBg(): 6 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted SUCCESS: alter diskgroup flash mount Mon Sep 19 21:19:15 2011 * allocate domain 3, invalid = TRUE freeing rdom 3 Received detach msg from node 0 for dom 3 * allocate domain 3, invalid = TRUE Mon Sep 19 21:19:48 2011 SQL> alter diskgroup PRD01 mount NOTE: cache registered group PRD01 number=3 incarn=0xd3f881d7 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f881d7 NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(): 8 Mon Sep 19 21:19:48 2011 kfdp_queryBg(): 8 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F881D7 (PRD01) Mon Sep 19 21:19:48 2011 kjbdomatt send to node 0 Mon Sep 19 21:19:49 2011 NOTE: attached to recovery domain 3 Mon Sep 19 21:19:49 2011 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: ACD expansion required for disk group 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.0 ABA NOTE: seq=2 blk=0 NOTE: cache mounting group 3/0xD3F881D7 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f881d7 Mon Sep 19 21:19:51 2011 kfdp_query(): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: alter diskgroup PRD01 mount Mon Sep 19 22:11:46 2011 Starting background process ASMB Mon Sep 19 22:11:46 2011 ASMB started with pid=22, OS id=7306 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Sep 19 22:27:43 2011 Trace dumping is performing id=[cdmp_20110919222743] Mon Sep 19 22:33:46 2011 Starting background process ASMB Mon Sep 19 22:33:47 2011 ASMB started with pid=21, OS id=26747 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 12:09:03 2011 Starting background process ASMB Tue Sep 20 12:09:03 2011 ASMB started with pid=21, OS id=13674 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 14:05:09 2011 Starting background process ASMB Tue Sep 20 14:05:09 2011 ASMB started with pid=22, OS id=31983 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 14:10:07 2011 Starting background process ASMB Tue Sep 20 14:10:07 2011 ASMB started with pid=21, OS id=3891 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 14:35:51 2011 Starting background process ASMB Tue Sep 20 14:35:51 2011 ASMB started with pid=21, OS id=24256 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 14:58:50 2011 Trace dumping is performing id=[cdmp_20110920145850] Tue Sep 20 15:01:30 2011 Trace dumping is performing id=[cdmp_20110920150130] Tue Sep 20 15:03:24 2011 Trace dumping is performing id=[cdmp_20110920150324] Tue Sep 20 15:10:02 2011 Starting background process ASMB Tue Sep 20 15:10:02 2011 ASMB started with pid=21, OS id=31612 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 15:12:00 2011 Trace dumping is performing id=[cdmp_20110920151159] Tue Sep 20 15:16:24 2011 Trace dumping is performing id=[cdmp_20110920151624] Tue Sep 20 15:19:03 2011 Trace dumping is performing id=[cdmp_20110920151902] Tue Sep 20 15:21:33 2011 Trace dumping is performing id=[cdmp_20110920152133] Tue Sep 20 15:23:32 2011 Trace dumping is performing id=[cdmp_20110920152332] Tue Sep 20 15:25:14 2011 Trace dumping is performing id=[cdmp_20110920152514] Tue Sep 20 15:29:05 2011 SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' Tue Sep 20 15:30:09 2011 Trace dumping is performing id=[cdmp_20110920153009] Tue Sep 20 15:35:19 2011 Trace dumping is performing id=[cdmp_20110920153519] Tue Sep 20 15:52:25 2011 Trace dumping is performing id=[cdmp_20110920155224] Tue Sep 20 15:57:34 2011 Trace dumping is performing id=[cdmp_20110920155734] Tue Sep 20 16:03:17 2011 Trace dumping is performing id=[cdmp_20110920160317] Tue Sep 20 16:07:27 2011 SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' Tue Sep 20 16:08:18 2011 Trace dumping is performing id=[cdmp_20110920160817] Tue Sep 20 16:12:11 2011 SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' Tue Sep 20 16:15:17 2011 SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' Tue Sep 20 16:16:08 2011 Trace dumping is performing id=[cdmp_20110920161607] Tue Sep 20 16:16:29 2011 Trace dumping is performing id=[cdmp_20110920161629] Tue Sep 20 16:17:18 2011 SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' Tue Sep 20 16:18:34 2011 Trace dumping is performing id=[cdmp_20110920161833] Tue Sep 20 16:19:26 2011 SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' Tue Sep 20 16:20:50 2011 Starting background process ASMB Tue Sep 20 16:20:50 2011 ASMB started with pid=22, OS id=2135 Tue Sep 20 16:20:54 2011 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_2280.trc Tue Sep 20 16:20:54 2011 Trace dumping is performing id=[cdmp_20110920162054] NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:22:45 2011 Starting background process ASMB Tue Sep 20 16:22:45 2011 ASMB started with pid=22, OS id=3789 Tue Sep 20 16:22:49 2011 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_3991.trc Tue Sep 20 16:22:49 2011 Trace dumping is performing id=[cdmp_20110920162249] NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:25:05 2011 Starting background process ASMB Tue Sep 20 16:25:05 2011 ASMB started with pid=21, OS id=5918 Tue Sep 20 16:25:09 2011 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_6059.trc Tue Sep 20 16:25:10 2011 Trace dumping is performing id=[cdmp_20110920162510] NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Trace dumping is performing id=[cdmp_20110920162515] Tue Sep 20 16:30:36 2011 Trace dumping is performing id=[cdmp_20110920163036] Tue Sep 20 16:35:37 2011 Trace dumping is performing id=[cdmp_20110920163537] Tue Sep 20 16:40:47 2011 Trace dumping is performing id=[cdmp_20110920164047] Tue Sep 20 16:50:11 2011 Starting background process ASMB Tue Sep 20 16:50:11 2011 ASMB started with pid=21, OS id=25608 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:50:28 2011 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_25798.trc Tue Sep 20 16:50:28 2011 Trace dumping is performing id=[cdmp_20110920165028] Tue Sep 20 16:51:20 2011 Starting background process ASMB Tue Sep 20 16:51:20 2011 ASMB started with pid=21, OS id=26521 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:51:37 2011 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_26671.trc Tue Sep 20 16:51:37 2011 Trace dumping is performing id=[cdmp_20110920165137] Tue Sep 20 16:55:16 2011 Trace dumping is performing id=[cdmp_20110920165515] Tue Sep 20 16:57:18 2011 Trace dumping is performing id=[cdmp_20110920165717] Tue Sep 20 17:02:28 2011 Trace dumping is performing id=[cdmp_20110920170227] Tue Sep 20 17:49:47 2011 Trace dumping is performing id=[cdmp_20110920174947] Tue Sep 20 17:50:11 2011 Trace dumping is performing id=[cdmp_20110920175010] Tue Sep 20 17:54:18 2011 Starting background process ASMB Tue Sep 20 17:54:18 2011 ASMB started with pid=21, OS id=23727 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 17:54:25 2011 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_23959.trc Tue Sep 20 17:54:25 2011 Trace dumping is performing id=[cdmp_20110920175425] Tue Sep 20 17:57:45 2011 Trace dumping is performing id=[cdmp_20110920175745] Tue Sep 20 18:04:17 2011 Starting background process ASMB Tue Sep 20 18:04:17 2011 ASMB started with pid=21, OS id=12378 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 18:04:31 2011 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_12545.trc Tue Sep 20 18:04:31 2011 Trace dumping is performing id=[cdmp_20110920180431] Tue Sep 20 18:18:09 2011 Starting background process ASMB Tue Sep 20 18:18:09 2011 ASMB started with pid=21, OS id=23866 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 18:18:24 2011 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_24072.trc Tue Sep 20 18:18:24 2011 Trace dumping is performing id=[cdmp_20110920181824] Tue Sep 20 18:23:21 2011 Starting background process ASMB Tue Sep 20 18:23:21 2011 ASMB started with pid=21, OS id=28442 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 18:23:36 2011 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_28630.trc Tue Sep 20 18:23:36 2011 Trace dumping is performing id=[cdmp_20110920182336] Tue Sep 20 18:27:00 2011 Starting background process ASMB Tue Sep 20 18:27:00 2011 ASMB started with pid=21, OS id=31018 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 18:27:15 2011 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_31561.trc Tue Sep 20 18:27:15 2011 Trace dumping is performing id=[cdmp_20110920182715] Tue Sep 20 18:27:27 2011 Trace dumping is performing id=[cdmp_20110920182727] Tue Sep 20 18:27:52 2011 Trace dumping is performing id=[cdmp_20110920182752] Tue Sep 20 18:32:12 2011 Starting background process ASMB Tue Sep 20 18:32:12 2011 ASMB started with pid=21, OS id=3290 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 18:32:27 2011 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_3463.trc Tue Sep 20 18:32:27 2011 Trace dumping is performing id=[cdmp_20110920183227] Tue Sep 20 18:37:02 2011 Trace dumping is performing id=[cdmp_20110920183702] Tue Sep 20 18:38:14 2011 Trace dumping is performing id=[cdmp_20110920183814] Tue Sep 20 18:38:38 2011 Trace dumping is performing id=[cdmp_20110920183838] Tue Sep 20 18:55:05 2011 Trace dumping is performing id=[cdmp_20110920185505] Tue Sep 20 18:55:20 2011 Trace dumping is performing id=[cdmp_20110920185520] Tue Sep 20 19:08:59 2011 Starting background process ASMB Tue Sep 20 19:08:59 2011 ASMB started with pid=21, OS id=12670 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 19:09:10 2011 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_12806.trc Tue Sep 20 19:09:11 2011 Trace dumping is performing id=[cdmp_20110920190911] Tue Sep 20 19:14:07 2011 Starting background process ASMB Tue Sep 20 19:14:07 2011 ASMB started with pid=20, OS id=16928 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 19:14:19 2011 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_17111.trc Tue Sep 20 19:14:19 2011 Trace dumping is performing id=[cdmp_20110920191419] Sun Sep 25 18:01:07 2011 Trace dumping is performing id=[cdmp_20110925180106] Sun Sep 25 18:32:29 2011 Received detach msg from node 0 for dom 1 Sun Sep 25 18:32:29 2011 NOTE: PST enabling heartbeating (grp 1) Received detach msg from node 0 for dom 2 NOTE: PST enabling heartbeating (grp 2) Received detach msg from node 0 for dom 3 NOTE: PST enabling heartbeating (grp 3) Sun Sep 25 18:32:33 2011 Reconfiguration started (old inc 12, new inc 14) List of nodes: 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sun Sep 25 18:32:33 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5405 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sun Sep 25 18:39:41 2011 Reconfiguration started (old inc 14, new inc 16) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Sun Sep 25 18:39:41 2011 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 * domain 2 valid = 1 according to instance 0 * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sun Sep 25 18:39:41 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5405 GCS shadows traversed, 2732 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sun Sep 25 18:41:30 2011 Shutting down instance: further logons disabled Shutting down instance (immediate) License high water mark = 11 SQL> ALTER DISKGROUP ALL DISMOUNT NOTE: cache dismounting group 1/0xC60881C9 (ORAFILES) Sun Sep 25 18:41:30 2011 kjbdomdet send to node 0 detach from dom 1, sending detach message to node 0 Sun Sep 25 18:41:30 2011 NOTE: detached from domain 1 NOTE: cache dismounted group 1/0xC60881C9 (ORAFILES) kfdp_dismount(): 10 Sun Sep 25 18:41:30 2011 kfdp_dismountBg(): 10 NOTE: De-assigning number (1,0) from disk (/dev/mpath/mpath1p1) NOTE: De-assigning number (1,1) from disk (/dev/mpath/mpath2p1) SUCCESS: diskgroup ORAFILES was dismounted NOTE: cache dismounting group 2/0x3A2881D1 (FLASH) kjbdomdet send to node 0 detach from dom 2, sending detach message to node 0 NOTE: detached from domain 2 NOTE: cache dismounted group 2/0x3A2881D1 (FLASH) kfdp_dismount(): 11 kfdp_dismountBg(): 11 NOTE: De-assigning number (2,0) from disk (/dev/mpath/mpath3p1) SUCCESS: diskgroup FLASH was dismounted NOTE: cache dismounting group 3/0xD3F881D7 (PRD01) kjbdomdet send to node 0 detach from dom 3, sending detach message to node 0 NOTE: detached from domain 3 NOTE: cache dismounted group 3/0xD3F881D7 (PRD01) kfdp_dismount(): 12 kfdp_dismountBg(): 12 NOTE: De-assigning number (3,0) from disk (/dev/mpath/mpath4p1) SUCCESS: diskgroup PRD01 was dismounted SUCCESS: ALTER DISKGROUP ALL DISMOUNT Sun Sep 25 18:41:32 2011 Stopping background process VKTM: Sun Sep 25 18:41:34 2011 freeing rdom 3 freeing rdom 2 freeing rdom 1 freeing rdom 0 Sun Sep 25 18:41:35 2011 Instance shutdown complete Sun Sep 25 18:44:32 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sun Sep 25 18:44:33 2011 PMON started with pid=2, OS id=5473 Sun Sep 25 18:44:33 2011 VKTM started with pid=3, OS id=5475 at elevated priority VKTM running at (20)ms precision Sun Sep 25 18:44:33 2011 DIAG started with pid=4, OS id=5479 Sun Sep 25 18:44:33 2011 PING started with pid=5, OS id=5481 Sun Sep 25 18:44:33 2011 PSP0 started with pid=6, OS id=5483 Sun Sep 25 18:44:33 2011 DIA0 started with pid=7, OS id=5485 Sun Sep 25 18:44:33 2011 LMON started with pid=8, OS id=5487 Sun Sep 25 18:44:33 2011 LMD0 started with pid=9, OS id=5490 Sun Sep 25 18:44:33 2011 LMS0 started with pid=10, OS id=5494 at elevated priority Sun Sep 25 18:44:33 2011 MMAN started with pid=11, OS id=5499 Sun Sep 25 18:44:33 2011 DBW0 started with pid=12, OS id=5503 Sun Sep 25 18:44:33 2011 LGWR started with pid=13, OS id=5509 Sun Sep 25 18:44:33 2011 CKPT started with pid=14, OS id=5528 Sun Sep 25 18:44:33 2011 SMON started with pid=15, OS id=5544 Sun Sep 25 18:44:33 2011 RBAL started with pid=16, OS id=5546 Sun Sep 25 18:44:33 2011 GMON started with pid=17, OS id=5548 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 20) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sun Sep 25 18:44:33 2011 LCK0 started with pid=18, OS id=5553 ORACLE_BASE from environment = /u01/app/oracle Sun Sep 25 18:44:34 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc60db01c NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc60db01c NOTE: cache registered group FLASH number=2 incarn=0x3a2db01b NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a2db01b NOTE: cache registered group PRD01 number=3 incarn=0xd3fdb01d NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3fdb01d WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0xC60DB01C (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.37637207 ABA NOTE: seq=77 blk=8964 NOTE: cache mounting group 1/0xC60DB01C (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc60db01c kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A2DB01B (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.16550319 ABA NOTE: seq=20 blk=8321 NOTE: cache mounting group 2/0x3A2DB01B (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a2db01b kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3FDB01D (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.9933 ABA NOTE: seq=3 blk=103 NOTE: cache mounting group 3/0xD3FDB01D (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3fdb01d kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sun Sep 25 18:45:02 2011 Received detach msg from node 0 for dom 1 Sun Sep 25 18:45:02 2011 NOTE: PST enabling heartbeating (grp 1) Received detach msg from node 0 for dom 2 NOTE: PST enabling heartbeating (grp 2) Received detach msg from node 0 for dom 3 NOTE: PST enabling heartbeating (grp 3) Sun Sep 25 18:45:06 2011 Reconfiguration started (old inc 20, new inc 22) List of nodes: 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sun Sep 25 18:45:06 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 29 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sun Sep 25 18:47:26 2011 Reconfiguration started (old inc 22, new inc 24) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Sun Sep 25 18:47:26 2011 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 * domain 2 valid = 1 according to instance 0 * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sun Sep 25 18:47:26 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 29 GCS shadows traversed, 29 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sun Sep 25 18:47:57 2011 Shutting down instance: further logons disabled Shutting down instance (immediate) License high water mark = 1 SQL> ALTER DISKGROUP ALL DISMOUNT NOTE: cache dismounting group 1/0xC60DB01C (ORAFILES) Sun Sep 25 18:47:57 2011 kjbdomdet send to node 0 detach from dom 1, sending detach message to node 0 Sun Sep 25 18:47:57 2011 NOTE: detached from domain 1 NOTE: cache dismounted group 1/0xC60DB01C (ORAFILES) kfdp_dismount(): 10 Sun Sep 25 18:47:57 2011 kfdp_dismountBg(): 10 NOTE: De-assigning number (1,0) from disk (/dev/mpath/mpath1p1) NOTE: De-assigning number (1,1) from disk (/dev/mpath/mpath2p1) SUCCESS: diskgroup ORAFILES was dismounted NOTE: cache dismounting group 2/0x3A2DB01B (FLASH) kjbdomdet send to node 0 detach from dom 2, sending detach message to node 0 NOTE: detached from domain 2 NOTE: cache dismounted group 2/0x3A2DB01B (FLASH) kfdp_dismount(): 11 kfdp_dismountBg(): 11 NOTE: De-assigning number (2,0) from disk (/dev/mpath/mpath3p1) SUCCESS: diskgroup FLASH was dismounted NOTE: cache dismounting group 3/0xD3FDB01D (PRD01) kjbdomdet send to node 0 detach from dom 3, sending detach message to node 0 NOTE: detached from domain 3 NOTE: cache dismounted group 3/0xD3FDB01D (PRD01) kfdp_dismount(): 12 kfdp_dismountBg(): 12 NOTE: De-assigning number (3,0) from disk (/dev/mpath/mpath4p1) SUCCESS: diskgroup PRD01 was dismounted SUCCESS: ALTER DISKGROUP ALL DISMOUNT Sun Sep 25 18:47:59 2011 Stopping background process VKTM: Sun Sep 25 18:48:00 2011 freeing rdom 3 freeing rdom 2 freeing rdom 1 freeing rdom 0 Sun Sep 25 18:48:02 2011 Instance shutdown complete Sun Sep 25 18:49:16 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sun Sep 25 18:49:17 2011 PMON started with pid=2, OS id=10130 Sun Sep 25 18:49:17 2011 VKTM started with pid=3, OS id=10132 at elevated priority VKTM running at (20)ms precision Sun Sep 25 18:49:17 2011 DIAG started with pid=4, OS id=10136 Sun Sep 25 18:49:17 2011 PING started with pid=5, OS id=10138 Sun Sep 25 18:49:17 2011 PSP0 started with pid=6, OS id=10140 Sun Sep 25 18:49:17 2011 DIA0 started with pid=7, OS id=10142 Sun Sep 25 18:49:17 2011 LMON started with pid=8, OS id=10144 Sun Sep 25 18:49:17 2011 LMD0 started with pid=9, OS id=10146 Sun Sep 25 18:49:17 2011 LMS0 started with pid=10, OS id=10148 at elevated priority Sun Sep 25 18:49:17 2011 MMAN started with pid=11, OS id=10152 Sun Sep 25 18:49:17 2011 DBW0 started with pid=12, OS id=10154 Sun Sep 25 18:49:17 2011 LGWR started with pid=13, OS id=10156 Sun Sep 25 18:49:17 2011 CKPT started with pid=14, OS id=10158 Sun Sep 25 18:49:17 2011 SMON started with pid=15, OS id=10160 Sun Sep 25 18:49:17 2011 RBAL started with pid=16, OS id=10162 Sun Sep 25 18:49:17 2011 GMON started with pid=17, OS id=10164 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 28) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sun Sep 25 18:49:18 2011 LCK0 started with pid=18, OS id=10175 ORACLE_BASE from environment = /u01/app/oracle Sun Sep 25 18:49:18 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc60e1f22 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc60e1f22 NOTE: cache registered group FLASH number=2 incarn=0x3a2e1f21 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a2e1f21 NOTE: cache registered group PRD01 number=3 incarn=0xd3fe1f23 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3fe1f23 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0xC60E1F22 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.37637207 ABA NOTE: seq=78 blk=8964 NOTE: cache mounting group 1/0xC60E1F22 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc60e1f22 kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A2E1F21 (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.16550319 ABA NOTE: seq=21 blk=8321 NOTE: cache mounting group 2/0x3A2E1F21 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a2e1f21 kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3FE1F23 (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.9933 ABA NOTE: seq=4 blk=103 NOTE: cache mounting group 3/0xD3FE1F23 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3fe1f23 kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sun Sep 25 18:57:22 2011 Trace dumping is performing id=[cdmp_20110925185722] Sun Sep 25 19:06:12 2011 Starting background process ASMB Sun Sep 25 19:06:12 2011 ASMB started with pid=21, OS id=8972 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 19:28:09 2011 Starting background process ASMB Sun Sep 25 19:28:09 2011 ASMB started with pid=20, OS id=28694 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 19:34:37 2011 Starting background process ASMB Sun Sep 25 19:34:38 2011 ASMB started with pid=20, OS id=1563 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 19:41:08 2011 Trace dumping is performing id=[cdmp_20110925194108] Sun Sep 25 20:05:55 2011 Trace dumping is performing id=[cdmp_20110925200555] Sun Sep 25 20:48:55 2011 Starting background process ASMB Sun Sep 25 20:48:55 2011 ASMB started with pid=21, OS id=12471 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 21:10:10 2011 Starting background process ASMB Sun Sep 25 21:10:10 2011 ASMB started with pid=20, OS id=11189 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 21:37:06 2011 Starting background process ASMB Sun Sep 25 21:37:06 2011 ASMB started with pid=21, OS id=1378 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Sep 26 11:08:23 2011 Trace dumping is performing id=[cdmp_20110926110823] Mon Sep 26 11:11:53 2011 Reconfiguration started (old inc 28, new inc 30) List of nodes: 1 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Mon Sep 26 11:11:54 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5458 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Mon Sep 26 11:11:54 2011 kfdp_query(ORAFILES): 10 Reconfiguration complete Mon Sep 26 11:11:54 2011 Trace dumping is performing id=[cdmp_20110926111153] Mon Sep 26 11:11:54 2011 NOTE: PST enabling heartbeating (grp 2) NOTE: PST enabling heartbeating (grp 3) NOTE: PST enabling heartbeating (grp 1) kfdp_queryBg(): 10 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=2 ckpt=165.3435 group=1 NOTE: advancing ckpt for thread=2 ckpt=165.3435 NOTE: smon did instance recovery for domain 1 kfdp_query(FLASH): 11 kfdp_queryBg(): 11 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=1 ckpt=45.9561 group=2 NOTE: advancing ckpt for thread=1 ckpt=45.9561 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 12 kfdp_queryBg(): 12 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=1 ckpt=5.521 group=3 NOTE: advancing ckpt for thread=1 ckpt=5.521 NOTE: smon did instance recovery for domain 3 Mon Sep 26 11:12:24 2011 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_11372.trc Mon Sep 26 11:12:24 2011 Trace dumping is performing id=[cdmp_20110926111224] Mon Sep 26 11:17:40 2011 Reconfiguration started (old inc 30, new inc 32) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Mon Sep 26 11:17:40 2011 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 * domain 2 valid = 1 according to instance 0 * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Mon Sep 26 11:17:40 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5458 GCS shadows traversed, 2800 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Mon Sep 26 11:27:02 2011 Starting background process ASMB Mon Sep 26 11:27:02 2011 ASMB started with pid=22, OS id=29995 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Sep 26 11:27:09 2011 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_30276.trc Mon Sep 26 11:27:09 2011 Trace dumping is performing id=[cdmp_20110926112709] Mon Sep 26 11:33:40 2011 Starting background process ASMB Mon Sep 26 11:33:40 2011 ASMB started with pid=21, OS id=3578 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Sep 26 11:33:47 2011 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_3743.trc Mon Sep 26 11:33:47 2011 Trace dumping is performing id=[cdmp_20110926113347] Mon Sep 26 11:35:32 2011 Starting background process ASMB Mon Sep 26 11:35:32 2011 ASMB started with pid=21, OS id=5031 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Sep 26 11:35:39 2011 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_5230.trc Mon Sep 26 11:35:39 2011 Trace dumping is performing id=[cdmp_20110926113539] Mon Sep 26 11:41:23 2011 Starting background process ASMB Mon Sep 26 11:41:23 2011 ASMB started with pid=21, OS id=9991 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Oct 23 02:58:56 2011 *********************************************************************** Fatal NI connect error 12560, connecting to: (LOCAL=NO) VERSION INFORMATION: TNS for Linux: Version 11.1.0.7.0 - Production Oracle Bequeath NT Protocol Adapter for Linux: Version 11.1.0.7.0 - Production Time: 23-OCT-2011 02:58:56 Tracing not turned on. Tns error struct: ns main err code: 12560 TNS-12560: TNS:protocol adapter error ns secondary err code: 0 nt main err code: 0 nt secondary err code: 0 nt OS err code: 0 ORA-609 : opiodr aborting process unknown ospid (26247_140064597522144) Thu Dec 08 15:54:56 2011 NOTE: disk validation pending for group 1/0xc60e1f22 (ORAFILES) Thu Dec 08 15:54:57 2011 WARNING: RDBMS client (dwh12:dwh1°) with version (10.2.0.5.0) is not compatible with a pending operation NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) SUCCESS: validated disks for 1/0xc60e1f22 (ORAFILES) WARNING: RDBMS client (dwh12:dwh1°) with version (10.2.0.5.0) is not compatible with a pending operation NOTE: membership refresh pending for group 1/0xc60e1f22 (ORAFILES) kfdp_query(ORAFILES): 13 Thu Dec 08 15:55:00 2011 kfdp_queryBg(): 13 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 kfdp_queryTimeout(ORAFILES) kfdp_query(ORAFILES): 14 kfdp_queryBg(): 14 SUCCESS: refreshed membership for 1/0xc60e1f22 (ORAFILES) Thu Dec 08 20:24:33 2011 NOTE: membership refresh pending for group 1/0xc60e1f22 (ORAFILES) Thu Dec 08 20:24:33 2011 kfdp_queryTimeout(ORAFILES) kfdp_query(ORAFILES): 15 kfdp_queryBg(): 15 SUCCESS: refreshed membership for 1/0xc60e1f22 (ORAFILES) Wed Feb 15 18:51:56 2012 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_28269.trc Wed Feb 15 18:52:07 2012 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_10136.trc Wed Feb 15 20:34:07 2012 Shutting down instance: further logons disabled Shutting down instance (immediate) License high water mark = 16 SQL> ALTER DISKGROUP ALL DISMOUNT NOTE: cache dismounting group 1/0xC60E1F22 (ORAFILES) Wed Feb 15 20:34:07 2012 kjbdomdet send to node 0 detach from dom 1, sending detach message to node 0 Wed Feb 15 20:34:08 2012 NOTE: detached from domain 1 NOTE: cache dismounted group 1/0xC60E1F22 (ORAFILES) kfdp_dismount(): 16 Wed Feb 15 20:34:08 2012 kfdp_dismountBg(): 16 NOTE: De-assigning number (1,0) from disk (/dev/mpath/mpath1p1) NOTE: De-assigning number (1,1) from disk (/dev/mpath/mpath2p1) NOTE: De-assigning number (1,2) from disk (/dev/mpath/mpath5p1) SUCCESS: diskgroup ORAFILES was dismounted NOTE: cache dismounting group 2/0x3A2E1F21 (FLASH) kjbdomdet send to node 0 detach from dom 2, sending detach message to node 0 NOTE: detached from domain 2 NOTE: cache dismounted group 2/0x3A2E1F21 (FLASH) kfdp_dismount(): 17 kfdp_dismountBg(): 17 NOTE: De-assigning number (2,0) from disk (/dev/mpath/mpath3p1) SUCCESS: diskgroup FLASH was dismounted NOTE: cache dismounting group 3/0xD3FE1F23 (PRD01) kjbdomdet send to node 0 detach from dom 3, sending detach message to node 0 NOTE: detached from domain 3 NOTE: cache dismounted group 3/0xD3FE1F23 (PRD01) kfdp_dismount(): 18 kfdp_dismountBg(): 18 NOTE: De-assigning number (3,0) from disk (/dev/mpath/mpath4p1) SUCCESS: diskgroup PRD01 was dismounted SUCCESS: ALTER DISKGROUP ALL DISMOUNT Wed Feb 15 20:34:10 2012 Stopping background process VKTM: Wed Feb 15 20:34:12 2012 freeing rdom 3 freeing rdom 2 freeing rdom 1 freeing rdom 0 Wed Feb 15 20:34:13 2012 Instance shutdown complete Wed Feb 15 20:43:59 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Wed Feb 15 20:44:02 2012 PMON started with pid=2, OS id=10468 Wed Feb 15 20:44:02 2012 VKTM started with pid=3, OS id=10470 at elevated priority Wed Feb 15 20:44:02 2012 DIAG started with pid=4, OS id=10474 Wed Feb 15 20:44:02 2012 PING started with pid=5, OS id=10476 Wed Feb 15 20:44:02 2012 PSP0 started with pid=6, OS id=10478 VKTM running at (20)ms precision Wed Feb 15 20:44:02 2012 DIA0 started with pid=7, OS id=10480 Wed Feb 15 20:44:02 2012 LMON started with pid=8, OS id=10482 Wed Feb 15 20:44:02 2012 LMD0 started with pid=9, OS id=10484 Wed Feb 15 20:44:02 2012 LMS0 started with pid=10, OS id=10491 at elevated priority Wed Feb 15 20:44:02 2012 MMAN started with pid=11, OS id=10496 Wed Feb 15 20:44:02 2012 DBW0 started with pid=12, OS id=10498 Wed Feb 15 20:44:02 2012 LGWR started with pid=13, OS id=10500 Wed Feb 15 20:44:02 2012 CKPT started with pid=14, OS id=10502 Wed Feb 15 20:44:02 2012 SMON started with pid=15, OS id=10504 Wed Feb 15 20:44:02 2012 RBAL started with pid=16, OS id=10506 Wed Feb 15 20:44:02 2012 GMON started with pid=17, OS id=10508 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 36) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Feb 15 20:44:03 2012 LCK0 started with pid=18, OS id=10516 ORACLE_BASE from environment = /u01/app/oracle Wed Feb 15 20:44:04 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc6087c1f NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc6087c1f NOTE: cache registered group FLASH number=2 incarn=0x3a287c1e NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a287c1e NOTE: cache registered group PRD01 number=3 incarn=0xd3f87c20 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f87c20 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (not first) group 1/0xC6087C1F (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.38467254 ABA NOTE: seq=83 blk=5304 NOTE: cache mounting group 1/0xC6087C1F (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc6087c1f kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A287C1E (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.48871141 ABA NOTE: seq=53 blk=3802 NOTE: cache mounting group 2/0x3A287C1E (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a287c1e kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F87C20 (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3066550 ABA NOTE: seq=5 blk=4710 NOTE: cache mounting group 3/0xD3F87C20 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f87c20 kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Wed Feb 15 20:44:16 2012 Starting background process ASMB Wed Feb 15 20:44:16 2012 ASMB started with pid=20, OS id=10782 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed Feb 15 20:45:26 2012 Starting background process ASMB Wed Feb 15 20:45:26 2012 ASMB started with pid=25, OS id=12352 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Feb 16 00:07:02 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Thu Feb 16 00:07:04 2012 PMON started with pid=2, OS id=9838 Thu Feb 16 00:07:04 2012 VKTM started with pid=3, OS id=9843 at elevated priority VKTM running at (20)ms precision Thu Feb 16 00:07:04 2012 DIAG started with pid=4, OS id=9862 Thu Feb 16 00:07:04 2012 PING started with pid=5, OS id=9867 Thu Feb 16 00:07:04 2012 PSP0 started with pid=6, OS id=9869 Thu Feb 16 00:07:04 2012 DIA0 started with pid=7, OS id=9871 Thu Feb 16 00:07:04 2012 LMON started with pid=8, OS id=9880 Thu Feb 16 00:07:04 2012 LMD0 started with pid=9, OS id=9882 Thu Feb 16 00:07:04 2012 LMS0 started with pid=10, OS id=9884 at elevated priority Thu Feb 16 00:07:04 2012 MMAN started with pid=11, OS id=9889 Thu Feb 16 00:07:04 2012 DBW0 started with pid=12, OS id=9899 Thu Feb 16 00:07:04 2012 LGWR started with pid=13, OS id=9901 Thu Feb 16 00:07:04 2012 CKPT started with pid=14, OS id=9905 Thu Feb 16 00:07:04 2012 SMON started with pid=15, OS id=9909 Thu Feb 16 00:07:04 2012 RBAL started with pid=16, OS id=9913 Thu Feb 16 00:07:04 2012 GMON started with pid=17, OS id=9915 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 40) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Thu Feb 16 00:07:05 2012 LCK0 started with pid=18, OS id=10024 ORACLE_BASE from environment = /u01/app/oracle Thu Feb 16 00:07:07 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc60863b9 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc60863b9 NOTE: cache registered group FLASH number=2 incarn=0x3a2863b8 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a2863b8 NOTE: cache registered group PRD01 number=3 incarn=0xd3f863ba NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f863ba WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (not first) group 1/0xC60863B9 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.38467256 ABA NOTE: seq=84 blk=5306 NOTE: cache mounting group 1/0xC60863B9 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc60863b9 kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A2863B8 (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.48871141 ABA NOTE: seq=54 blk=3803 NOTE: cache mounting group 2/0x3A2863B8 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a2863b8 kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F863BA (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3066552 ABA NOTE: seq=6 blk=4712 NOTE: cache mounting group 3/0xD3F863BA (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f863ba kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Thu Feb 16 00:07:18 2012 Starting background process ASMB Thu Feb 16 00:07:18 2012 ASMB started with pid=20, OS id=10651 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Feb 16 00:23:04 2012 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_10917.trc Thu Feb 16 00:23:04 2012 Trace dumping is performing id=[cdmp_20120216002304] Thu Feb 16 00:28:21 2012 Starting background process ASMB Thu Feb 16 00:28:21 2012 ASMB started with pid=21, OS id=28834 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Feb 16 03:23:41 2012 Starting background process ASMB Thu Feb 16 03:23:41 2012 ASMB started with pid=21, OS id=10142 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Feb 18 14:44:09 2012 ksimpoll: Cluster Synchronization Service is shutting down Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_lck0_10024.trc: ORA-29746: Cluster Synchronization Service is being shut down. LCK0 (ospid: 10024): terminating the instance due to error 29746 Instance terminated by LCK0, pid = 10024 Sat Feb 18 16:14:04 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sat Feb 18 16:14:45 2012 PMON started with pid=2, OS id=11146 Sat Feb 18 16:14:45 2012 VKTM started with pid=3, OS id=11148 at elevated priority VKTM running at (20)ms precision Sat Feb 18 16:14:45 2012 DIAG started with pid=4, OS id=11152 Sat Feb 18 16:14:45 2012 PING started with pid=5, OS id=11154 Sat Feb 18 16:14:45 2012 PSP0 started with pid=6, OS id=11156 Sat Feb 18 16:14:45 2012 DIA0 started with pid=7, OS id=11158 Sat Feb 18 16:14:45 2012 LMON started with pid=8, OS id=11160 Sat Feb 18 16:14:45 2012 LMD0 started with pid=9, OS id=11162 Sat Feb 18 16:14:45 2012 LMS0 started with pid=10, OS id=11164 at elevated priority Sat Feb 18 16:14:45 2012 MMAN started with pid=11, OS id=11168 Sat Feb 18 16:14:45 2012 DBW0 started with pid=12, OS id=11170 Sat Feb 18 16:14:46 2012 LGWR started with pid=13, OS id=11172 Sat Feb 18 16:14:46 2012 CKPT started with pid=14, OS id=11174 Sat Feb 18 16:14:46 2012 SMON started with pid=15, OS id=11176 Sat Feb 18 16:14:46 2012 RBAL started with pid=16, OS id=11178 Sat Feb 18 16:14:46 2012 GMON started with pid=17, OS id=11180 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 44) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Feb 18 16:14:46 2012 LCK0 started with pid=18, OS id=11191 Sat Feb 18 16:14:47 2012 ORACLE_BASE from environment = /u01/app/oracle Sat Feb 18 16:14:48 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc6088e77 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc6088e77 NOTE: cache registered group FLASH number=2 incarn=0x3a288e76 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a288e76 NOTE: cache registered group PRD01 number=3 incarn=0xd3f88e78 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f88e78 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (not first) group 1/0xC6088E77 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.38474559 ABA NOTE: seq=85 blk=5682 NOTE: cache mounting group 1/0xC6088E77 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc6088e77 kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A288E76 (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.49486066 ABA NOTE: seq=56 blk=1355 NOTE: cache mounting group 2/0x3A288E76 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a288e76 kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F88E78 (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3070685 ABA NOTE: seq=7 blk=4782 NOTE: cache mounting group 3/0xD3F88E78 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f88e78 kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sat Feb 18 16:14:58 2012 Starting background process ASMB Sat Feb 18 16:14:58 2012 ASMB started with pid=20, OS id=11372 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Feb 18 16:20:24 2012 Starting background process ASMB Sat Feb 18 16:20:24 2012 ASMB started with pid=22, OS id=16529 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Feb 18 18:48:52 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sat Feb 18 18:49:33 2012 PMON started with pid=2, OS id=18296 Sat Feb 18 18:49:34 2012 VKTM started with pid=3, OS id=18304 at elevated priority Sat Feb 18 18:49:34 2012 DIAG started with pid=4, OS id=18309 VKTM running at (20)ms precision Sat Feb 18 18:49:34 2012 PING started with pid=5, OS id=18311 Sat Feb 18 18:49:34 2012 PSP0 started with pid=6, OS id=18313 Sat Feb 18 18:49:34 2012 DIA0 started with pid=7, OS id=18317 Sat Feb 18 18:49:34 2012 LMON started with pid=8, OS id=18319 Sat Feb 18 18:49:34 2012 LMD0 started with pid=9, OS id=18321 Sat Feb 18 18:49:34 2012 LMS0 started with pid=10, OS id=18323 at elevated priority Sat Feb 18 18:49:34 2012 MMAN started with pid=11, OS id=18327 Sat Feb 18 18:49:34 2012 DBW0 started with pid=12, OS id=18329 Sat Feb 18 18:49:34 2012 LGWR started with pid=13, OS id=18331 Sat Feb 18 18:49:34 2012 CKPT started with pid=14, OS id=18334 Sat Feb 18 18:49:34 2012 SMON started with pid=15, OS id=18336 Sat Feb 18 18:49:34 2012 RBAL started with pid=16, OS id=18338 Sat Feb 18 18:49:34 2012 GMON started with pid=17, OS id=18340 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 48) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Feb 18 18:49:35 2012 LCK0 started with pid=18, OS id=18349 Sat Feb 18 18:49:35 2012 ORACLE_BASE from environment = /u01/app/oracle Sat Feb 18 18:49:36 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc60880d8 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc60880d8 NOTE: cache registered group FLASH number=2 incarn=0x3a2880d7 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a2880d7 NOTE: cache registered group PRD01 number=3 incarn=0xd3f880d9 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f880d9 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (not first) group 1/0xC60880D8 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.38474733 ABA NOTE: seq=86 blk=5770 NOTE: cache mounting group 1/0xC60880D8 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc60880d8 kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A2880D7 (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.49486066 ABA NOTE: seq=57 blk=1356 NOTE: cache mounting group 2/0x3A2880D7 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a2880d7 kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F880D9 (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3071989 ABA NOTE: seq=8 blk=4809 NOTE: cache mounting group 3/0xD3F880D9 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f880d9 kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sat Feb 18 18:49:46 2012 Starting background process ASMB Sat Feb 18 18:49:46 2012 ASMB started with pid=20, OS id=18544 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Feb 18 19:06:19 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sat Feb 18 19:07:01 2012 PMON started with pid=2, OS id=10932 Sat Feb 18 19:07:01 2012 VKTM started with pid=3, OS id=10934 at elevated priority VKTM running at (20)ms precision Sat Feb 18 19:07:01 2012 DIAG started with pid=4, OS id=10938 Sat Feb 18 19:07:01 2012 PING started with pid=5, OS id=10940 Sat Feb 18 19:07:01 2012 PSP0 started with pid=6, OS id=10942 Sat Feb 18 19:07:01 2012 DIA0 started with pid=7, OS id=10944 Sat Feb 18 19:07:01 2012 LMON started with pid=8, OS id=10946 Sat Feb 18 19:07:01 2012 LMD0 started with pid=9, OS id=10948 Sat Feb 18 19:07:01 2012 LMS0 started with pid=10, OS id=10950 at elevated priority Sat Feb 18 19:07:02 2012 MMAN started with pid=11, OS id=10954 Sat Feb 18 19:07:02 2012 DBW0 started with pid=12, OS id=10956 Sat Feb 18 19:07:02 2012 LGWR started with pid=13, OS id=10958 Sat Feb 18 19:07:02 2012 CKPT started with pid=14, OS id=10960 Sat Feb 18 19:07:02 2012 SMON started with pid=15, OS id=10962 Sat Feb 18 19:07:02 2012 RBAL started with pid=16, OS id=10964 Sat Feb 18 19:07:02 2012 GMON started with pid=17, OS id=10966 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 52) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Feb 18 19:07:02 2012 LCK0 started with pid=18, OS id=11018 Sat Feb 18 19:07:03 2012 ORACLE_BASE from environment = /u01/app/oracle Sat Feb 18 19:07:03 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc60881e5 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc60881e5 NOTE: cache registered group FLASH number=2 incarn=0x3a2881e4 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a2881e4 NOTE: cache registered group PRD01 number=3 incarn=0xd3f881e6 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f881e6 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (not first) group 1/0xC60881E5 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.38474733 ABA NOTE: seq=87 blk=5771 NOTE: cache mounting group 1/0xC60881E5 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc60881e5 kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A2881E4 (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.49486066 ABA NOTE: seq=58 blk=1357 NOTE: cache mounting group 2/0x3A2881E4 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a2881e4 kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F881E6 (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3072195 ABA NOTE: seq=9 blk=4811 NOTE: cache mounting group 3/0xD3F881E6 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f881e6 kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sat Feb 18 19:07:13 2012 Starting background process ASMB Sat Feb 18 19:07:13 2012 ASMB started with pid=20, OS id=11178 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Feb 18 19:16:47 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sat Feb 18 19:17:28 2012 PMON started with pid=2, OS id=18257 Sat Feb 18 19:17:28 2012 VKTM started with pid=3, OS id=18259 at elevated priority VKTM running at (20)ms precision Sat Feb 18 19:17:28 2012 DIAG started with pid=4, OS id=18263 Sat Feb 18 19:17:28 2012 PING started with pid=5, OS id=18265 Sat Feb 18 19:17:28 2012 PSP0 started with pid=6, OS id=18267 Sat Feb 18 19:17:28 2012 DIA0 started with pid=7, OS id=18269 Sat Feb 18 19:17:28 2012 LMON started with pid=8, OS id=18271 Sat Feb 18 19:17:28 2012 LMD0 started with pid=9, OS id=18273 Sat Feb 18 19:17:28 2012 LMS0 started with pid=10, OS id=18275 at elevated priority Sat Feb 18 19:17:28 2012 MMAN started with pid=11, OS id=18279 Sat Feb 18 19:17:28 2012 DBW0 started with pid=12, OS id=18281 Sat Feb 18 19:17:28 2012 LGWR started with pid=13, OS id=18283 Sat Feb 18 19:17:29 2012 CKPT started with pid=14, OS id=18285 Sat Feb 18 19:17:29 2012 SMON started with pid=15, OS id=18287 Sat Feb 18 19:17:29 2012 RBAL started with pid=16, OS id=18289 Sat Feb 18 19:17:29 2012 GMON started with pid=17, OS id=18291 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 56) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Feb 18 19:17:29 2012 LCK0 started with pid=18, OS id=18302 Sat Feb 18 19:17:30 2012 ORACLE_BASE from environment = /u01/app/oracle Sat Feb 18 19:17:30 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc6088a1d NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc6088a1d NOTE: cache registered group FLASH number=2 incarn=0x3a288a1c NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a288a1c NOTE: cache registered group PRD01 number=3 incarn=0xd3f88a1e NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f88a1e WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (not first) group 1/0xC6088A1D (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.38474733 ABA NOTE: seq=88 blk=5772 NOTE: cache mounting group 1/0xC6088A1D (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc6088a1d kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A288A1C (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.49486066 ABA NOTE: seq=59 blk=1358 NOTE: cache mounting group 2/0x3A288A1C (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a288a1c kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F88A1E (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3072197 ABA NOTE: seq=10 blk=4813 NOTE: cache mounting group 3/0xD3F88A1E (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f88a1e kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sat Feb 18 19:17:41 2012 Starting background process ASMB Sat Feb 18 19:17:41 2012 ASMB started with pid=20, OS id=18503 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Feb 18 19:23:02 2012 Starting background process ASMB Sat Feb 18 19:23:02 2012 ASMB started with pid=23, OS id=22402 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Feb 18 19:23:53 2012 NOTE: ASM client dwh12:dwh1 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_22526.trc Sat Feb 18 19:23:53 2012 Trace dumping is performing id=[cdmp_20120218192353] Sat Feb 18 19:23:58 2012 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_21883.trc Trace dumping is performing id=[cdmp_20120218192358] Sat Feb 18 19:24:00 2012 Shutting down instance (abort) License high water mark = 9 USER (ospid: 22958): terminating the instance Instance terminated by USER, pid = 22958 Sat Feb 18 19:24:01 2012 Instance shutdown complete Sat Feb 18 18:28:49 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sat Feb 18 18:28:51 2012 PMON started with pid=2, OS id=9888 Sat Feb 18 18:28:51 2012 VKTM started with pid=3, OS id=9890 at elevated priority Sat Feb 18 18:28:51 2012 DIAG started with pid=4, OS id=9894 VKTM running at (20)ms precision Sat Feb 18 18:28:51 2012 PING started with pid=5, OS id=9896 Sat Feb 18 18:28:51 2012 PSP0 started with pid=6, OS id=9898 Sat Feb 18 18:28:51 2012 DIA0 started with pid=7, OS id=9900 Sat Feb 18 18:28:51 2012 LMON started with pid=8, OS id=9902 Sat Feb 18 18:28:51 2012 LMD0 started with pid=9, OS id=9904 Sat Feb 18 18:28:52 2012 LMS0 started with pid=10, OS id=9906 at elevated priority Sat Feb 18 18:28:52 2012 MMAN started with pid=11, OS id=9910 Sat Feb 18 18:28:52 2012 DBW0 started with pid=12, OS id=9912 Sat Feb 18 18:28:52 2012 LGWR started with pid=13, OS id=9914 Sat Feb 18 18:28:52 2012 CKPT started with pid=14, OS id=9916 Sat Feb 18 18:28:52 2012 SMON started with pid=15, OS id=9921 Sat Feb 18 18:28:52 2012 RBAL started with pid=16, OS id=9923 Sat Feb 18 18:28:52 2012 GMON started with pid=17, OS id=9925 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 60) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Feb 18 18:28:52 2012 LCK0 started with pid=18, OS id=9936 ORACLE_BASE from environment = /u01/app/oracle Sat Feb 18 18:28:53 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc60864b0 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc60864b0 NOTE: cache registered group FLASH number=2 incarn=0x3a2864af NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a2864af NOTE: cache registered group PRD01 number=3 incarn=0xd3f864b1 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f864b1 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) ERROR: no PST quorum in group 1: required 1, found 0 NOTE: cache dismounting group 1/0xC60864B0 (ORAFILES) NOTE: dbwr not being msg'd to dismount NOTE: lgwr not being msg'd to dismount NOTE: cache dismounted group 1/0xC60864B0 (ORAFILES) NOTE: cache ending mount (fail) of group ORAFILES number=1 incarn=0xc60864b0 kfdp_dismount(): 2 kfdp_dismountBg(): 2 NOTE: De-assigning number (1,2) from disk (/dev/mpath/mpath5p1) ERROR: diskgroup ORAFILES was not mounted NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A2864AF (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.49512961 ABA NOTE: seq=60 blk=1367 NOTE: cache mounting group 2/0x3A2864AF (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a2864af kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F864B1 (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3072233 ABA NOTE: seq=11 blk=4832 NOTE: cache mounting group 3/0xD3F864B1 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f864b1 kfdp_query(FLASH): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted ORA-15032: not all alterations performed ORA-15063: ASM discovered an insufficient number of disks for diskgroup "ORAFILES" ERROR: ALTER DISKGROUP ALL MOUNT Sat Feb 18 18:29:04 2012 Starting background process ASMB Sat Feb 18 18:29:04 2012 ASMB started with pid=20, OS id=10345 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Feb 18 19:07:47 2012 SQL> ALTER DISKGROUP ORAFILES mount WARNING: Deprecated privilege SYSDBA for command 'ALTER DISKGROUP MOUNT' NOTE: cache registered group ORAFILES number=1 incarn=0xc60864b4 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc60864b4 NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) Sat Feb 18 19:07:47 2012 ERROR: no PST quorum in group 1: required 1, found 0 NOTE: cache dismounting group 1/0xC60864B4 (ORAFILES) NOTE: dbwr not being msg'd to dismount NOTE: lgwr not being msg'd to dismount NOTE: cache dismounted group 1/0xC60864B4 (ORAFILES) NOTE: cache ending mount (fail) of group ORAFILES number=1 incarn=0xc60864b4 kfdp_dismount(): 10 kfdp_dismountBg(): 10 NOTE: De-assigning number (1,2) from disk (/dev/mpath/mpath5p1) ERROR: diskgroup ORAFILES was not mounted ORA-15032: not all alterations performed ORA-15063: ASM discovered an insufficient number of disks for diskgroup "ORAFILES" ERROR: ALTER DISKGROUP ORAFILES mount Sat Feb 18 19:33:26 2012 SQL> ALTER DISKGROUP ORAFILES MOUNT WARNING: Deprecated privilege SYSDBA for command 'ALTER DISKGROUP MOUNT' NOTE: cache registered group ORAFILES number=1 incarn=0xc60864bc NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc60864bc NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) Sat Feb 18 19:33:26 2012 ERROR: no PST quorum in group 1: required 1, found 0 NOTE: cache dismounting group 1/0xC60864BC (ORAFILES) NOTE: dbwr not being msg'd to dismount NOTE: lgwr not being msg'd to dismount NOTE: cache dismounted group 1/0xC60864BC (ORAFILES) NOTE: cache ending mount (fail) of group ORAFILES number=1 incarn=0xc60864bc kfdp_dismount(): 12 kfdp_dismountBg(): 12 NOTE: De-assigning number (1,2) from disk (/dev/mpath/mpath5p1) ERROR: diskgroup ORAFILES was not mounted ORA-15032: not all alterations performed ORA-15063: ASM discovered an insufficient number of disks for diskgroup "ORAFILES" ERROR: ALTER DISKGROUP ORAFILES MOUNT Sat Feb 18 19:35:22 2012 SQL> alter diskgroup ORAFILES mount WARNING: Deprecated privilege SYSDBA for command 'ALTER DISKGROUP MOUNT' NOTE: cache registered group ORAFILES number=1 incarn=0xc60864bf NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc60864bf NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) Sat Feb 18 19:35:22 2012 ERROR: no PST quorum in group 1: required 1, found 0 NOTE: cache dismounting group 1/0xC60864BF (ORAFILES) NOTE: dbwr not being msg'd to dismount NOTE: lgwr not being msg'd to dismount NOTE: cache dismounted group 1/0xC60864BF (ORAFILES) NOTE: cache ending mount (fail) of group ORAFILES number=1 incarn=0xc60864bf kfdp_dismount(): 14 kfdp_dismountBg(): 14 NOTE: De-assigning number (1,2) from disk (/dev/mpath/mpath5p1) ERROR: diskgroup ORAFILES was not mounted ORA-15032: not all alterations performed ORA-15063: ASM discovered an insufficient number of disks for diskgroup "ORAFILES" ERROR: alter diskgroup ORAFILES mount Sat Feb 18 19:40:35 2012 SQL> ALTER DISKGROUP ORAFILES MOUNT WARNING: Deprecated privilege SYSDBA for command 'ALTER DISKGROUP MOUNT' NOTE: cache registered group ORAFILES number=1 incarn=0xc60864c2 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc60864c2 NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) Sat Feb 18 19:40:36 2012 ERROR: no PST quorum in group 1: required 1, found 0 NOTE: cache dismounting group 1/0xC60864C2 (ORAFILES) NOTE: dbwr not being msg'd to dismount NOTE: lgwr not being msg'd to dismount NOTE: cache dismounted group 1/0xC60864C2 (ORAFILES) NOTE: cache ending mount (fail) of group ORAFILES number=1 incarn=0xc60864c2 kfdp_dismount(): 16 kfdp_dismountBg(): 16 NOTE: De-assigning number (1,2) from disk (/dev/mpath/mpath5p1) ERROR: diskgroup ORAFILES was not mounted ORA-15032: not all alterations performed ORA-15063: ASM discovered an insufficient number of disks for diskgroup "ORAFILES" ERROR: ALTER DISKGROUP ORAFILES MOUNT Sat Feb 18 20:17:09 2012 SQL> alter diskgroup ORAFILES mount WARNING: Deprecated privilege SYSDBA for command 'ALTER DISKGROUP MOUNT' NOTE: cache registered group ORAFILES number=1 incarn=0xc60864c5 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc60864c5 NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) Sat Feb 18 20:17:09 2012 ERROR: no PST quorum in group 1: required 1, found 0 NOTE: cache dismounting group 1/0xC60864C5 (ORAFILES) NOTE: dbwr not being msg'd to dismount NOTE: lgwr not being msg'd to dismount NOTE: cache dismounted group 1/0xC60864C5 (ORAFILES) NOTE: cache ending mount (fail) of group ORAFILES number=1 incarn=0xc60864c5 kfdp_dismount(): 18 kfdp_dismountBg(): 18 NOTE: De-assigning number (1,2) from disk (/dev/mpath/mpath5p1) ERROR: diskgroup ORAFILES was not mounted ORA-15032: not all alterations performed ORA-15063: ASM discovered an insufficient number of disks for diskgroup "ORAFILES" ERROR: alter diskgroup ORAFILES mount Sat Feb 18 20:42:06 2012 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_10697.trc Sat Feb 18 20:42:06 2012 Trace dumping is performing id=[cdmp_20120218204206] Sat Feb 18 20:42:10 2012 Shutting down instance (abort) License high water mark = 5 USER (ospid: 27566): terminating the instance Instance terminated by USER, pid = 27566 Sat Feb 18 20:42:11 2012 Instance shutdown complete Sat Feb 18 20:55:46 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sat Feb 18 20:55:48 2012 PMON started with pid=2, OS id=9500 Sat Feb 18 20:55:48 2012 VKTM started with pid=3, OS id=9502 at elevated priority Sat Feb 18 20:55:48 2012 DIAG started with pid=4, OS id=9507 Sat Feb 18 20:55:48 2012 PING started with pid=5, OS id=9509 VKTM running at (20)ms precision Sat Feb 18 20:55:48 2012 PSP0 started with pid=6, OS id=9511 Sat Feb 18 20:55:48 2012 DIA0 started with pid=7, OS id=9520 Sat Feb 18 20:55:48 2012 LMON started with pid=8, OS id=9534 Sat Feb 18 20:55:48 2012 LMD0 started with pid=9, OS id=9540 Sat Feb 18 20:55:48 2012 LMS0 started with pid=10, OS id=9545 at elevated priority Sat Feb 18 20:55:48 2012 MMAN started with pid=11, OS id=9555 Sat Feb 18 20:55:48 2012 DBW0 started with pid=12, OS id=9558 Sat Feb 18 20:55:48 2012 LGWR started with pid=13, OS id=9560 Sat Feb 18 20:55:48 2012 CKPT started with pid=14, OS id=9562 Sat Feb 18 20:55:48 2012 SMON started with pid=15, OS id=9564 Sat Feb 18 20:55:48 2012 RBAL started with pid=16, OS id=9572 Sat Feb 18 20:55:48 2012 GMON started with pid=17, OS id=9574 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 64) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Feb 18 20:55:49 2012 LCK0 started with pid=18, OS id=9657 ORACLE_BASE from environment = /u01/app/oracle Sat Feb 18 20:55:50 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group FLASH number=2 incarn=0x3a2860f3 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a2860f3 NOTE: cache registered group PRD01 number=3 incarn=0xd3f860f4 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f860f4 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(FLASH): 3 kfdp_queryBg(): 3 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A2860F3 (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.49512961 ABA NOTE: seq=61 blk=1368 NOTE: cache mounting group 2/0x3A2860F3 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a2860f3 kfdp_query(PRD01): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F860F4 (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3072283 ABA NOTE: seq=12 blk=4850 NOTE: cache mounting group 3/0xD3F860F4 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f860f4 kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sat Feb 18 20:56:01 2012 Starting background process ASMB Sat Feb 18 20:56:01 2012 ASMB started with pid=20, OS id=10164 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Feb 18 20:59:55 2012 SQL> alter diskgroup ORAFILES mount WARNING: Deprecated privilege SYSDBA for command 'ALTER DISKGROUP MOUNT' NOTE: cache registered group ORAFILES number=1 incarn=0xc60860f7 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc60860f7 NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) kfdp_query(ORAFILES): 8 Sat Feb 18 20:59:55 2012 kfdp_queryBg(): 8 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (not first) group 1/0xC60860F7 (ORAFILES) Sat Feb 18 20:59:55 2012 kjbdomatt send to node 0 Sat Feb 18 20:59:55 2012 NOTE: attached to recovery domain 1 Sat Feb 18 20:59:55 2012 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.38474907 ABA NOTE: seq=89 blk=5860 NOTE: cache mounting group 1/0xC60860F7 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc60860f7 Sat Feb 18 20:59:56 2012 kfdp_query(ORAFILES): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: alter diskgroup ORAFILES mount Sat Feb 18 21:00:43 2012 Starting background process ASMB Sat Feb 18 21:00:43 2012 ASMB started with pid=26, OS id=14909 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Feb 18 21:30:00 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sat Feb 18 21:30:02 2012 PMON started with pid=2, OS id=9682 Sat Feb 18 21:30:02 2012 VKTM started with pid=3, OS id=9686 at elevated priority VKTM running at (20)ms precision Sat Feb 18 21:30:02 2012 DIAG started with pid=4, OS id=9698 Sat Feb 18 21:30:02 2012 PING started with pid=5, OS id=9702 Sat Feb 18 21:30:02 2012 PSP0 started with pid=6, OS id=9716 Sat Feb 18 21:30:02 2012 DIA0 started with pid=7, OS id=9732 Sat Feb 18 21:30:02 2012 LMON started with pid=8, OS id=9747 Sat Feb 18 21:30:02 2012 LMD0 started with pid=9, OS id=9762 Sat Feb 18 21:30:03 2012 LMS0 started with pid=10, OS id=9781 at elevated priority Sat Feb 18 21:30:03 2012 MMAN started with pid=11, OS id=9801 Sat Feb 18 21:30:03 2012 DBW0 started with pid=12, OS id=9818 Sat Feb 18 21:30:03 2012 LGWR started with pid=13, OS id=9844 Sat Feb 18 21:30:03 2012 CKPT started with pid=14, OS id=9860 Sat Feb 18 21:30:03 2012 SMON started with pid=15, OS id=9863 Sat Feb 18 21:30:03 2012 RBAL started with pid=16, OS id=9866 Sat Feb 18 21:30:03 2012 GMON started with pid=17, OS id=9868 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 68) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Feb 18 21:30:03 2012 LCK0 started with pid=18, OS id=9928 ORACLE_BASE from environment = /u01/app/oracle Sat Feb 18 21:30:05 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc6086a71 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc6086a71 NOTE: cache registered group FLASH number=2 incarn=0x3a286a70 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a286a70 NOTE: cache registered group PRD01 number=3 incarn=0xd3f86a72 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f86a72 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (not first) group 1/0xC6086A71 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.38475077 ABA NOTE: seq=90 blk=5946 NOTE: cache mounting group 1/0xC6086A71 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc6086a71 kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A286A70 (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.49513750 ABA NOTE: seq=62 blk=1374 NOTE: cache mounting group 2/0x3A286A70 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a286a70 kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F86A72 (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3072336 ABA NOTE: seq=13 blk=4868 NOTE: cache mounting group 3/0xD3F86A72 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f86a72 kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sat Feb 18 21:30:16 2012 Starting background process ASMB Sat Feb 18 21:30:16 2012 ASMB started with pid=20, OS id=10681 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Feb 18 21:31:11 2012 Starting background process ASMB Sat Feb 18 21:31:11 2012 ASMB started with pid=25, OS id=12130 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Feb 25 16:17:44 2012 Shutting down instance: further logons disabled Shutting down instance (immediate) License high water mark = 10 SQL> ALTER DISKGROUP ALL DISMOUNT NOTE: cache dismounting group 1/0xC6086A71 (ORAFILES) Sat Feb 25 16:17:44 2012 kjbdomdet send to node 0 detach from dom 1, sending detach message to node 0 Sat Feb 25 16:17:44 2012 NOTE: detached from domain 1 NOTE: cache dismounted group 1/0xC6086A71 (ORAFILES) kfdp_dismount(): 10 Sat Feb 25 16:17:44 2012 kfdp_dismountBg(): 10 NOTE: De-assigning number (1,0) from disk (/dev/mpath/mpath1p1) NOTE: De-assigning number (1,1) from disk (/dev/mpath/mpath2p1) NOTE: De-assigning number (1,2) from disk (/dev/mpath/mpath5p1) SUCCESS: diskgroup ORAFILES was dismounted NOTE: cache dismounting group 2/0x3A286A70 (FLASH) kjbdomdet send to node 0 detach from dom 2, sending detach message to node 0 NOTE: detached from domain 2 NOTE: cache dismounted group 2/0x3A286A70 (FLASH) kfdp_dismount(): 11 kfdp_dismountBg(): 11 NOTE: De-assigning number (2,0) from disk (/dev/mpath/mpath3p1) SUCCESS: diskgroup FLASH was dismounted NOTE: cache dismounting group 3/0xD3F86A72 (PRD01) kjbdomdet send to node 0 detach from dom 3, sending detach message to node 0 NOTE: detached from domain 3 NOTE: cache dismounted group 3/0xD3F86A72 (PRD01) kfdp_dismount(): 12 kfdp_dismountBg(): 12 NOTE: De-assigning number (3,0) from disk (/dev/mpath/mpath4p1) SUCCESS: diskgroup PRD01 was dismounted SUCCESS: ALTER DISKGROUP ALL DISMOUNT Sat Feb 25 16:17:46 2012 Stopping background process VKTM: Sat Feb 25 16:17:48 2012 freeing rdom 3 freeing rdom 2 freeing rdom 1 freeing rdom 0 Sat Feb 25 16:17:49 2012 Instance shutdown complete Sat Feb 25 17:03:01 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sat Feb 25 17:03:43 2012 PMON started with pid=2, OS id=11155 Sat Feb 25 17:03:43 2012 VKTM started with pid=3, OS id=11157 at elevated priority Sat Feb 25 17:03:43 2012 DIAG started with pid=4, OS id=11161 VKTM running at (20)ms precision Sat Feb 25 17:03:43 2012 PING started with pid=5, OS id=11163 Sat Feb 25 17:03:43 2012 PSP0 started with pid=6, OS id=11165 Sat Feb 25 17:03:43 2012 DIA0 started with pid=7, OS id=11167 Sat Feb 25 17:03:43 2012 LMON started with pid=8, OS id=11169 Sat Feb 25 17:03:43 2012 LMD0 started with pid=9, OS id=11171 Sat Feb 25 17:03:43 2012 LMS0 started with pid=10, OS id=11173 at elevated priority Sat Feb 25 17:03:43 2012 MMAN started with pid=11, OS id=11177 Sat Feb 25 17:03:43 2012 DBW0 started with pid=12, OS id=11179 Sat Feb 25 17:03:43 2012 LGWR started with pid=13, OS id=11181 Sat Feb 25 17:03:43 2012 CKPT started with pid=14, OS id=11183 Sat Feb 25 17:03:43 2012 SMON started with pid=15, OS id=11185 Sat Feb 25 17:03:43 2012 RBAL started with pid=16, OS id=11187 Sat Feb 25 17:03:43 2012 GMON started with pid=17, OS id=11189 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 72) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Feb 25 17:03:44 2012 LCK0 started with pid=18, OS id=11200 Sat Feb 25 17:03:44 2012 ORACLE_BASE from environment = /u01/app/oracle Sat Feb 25 17:03:45 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc608825b NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc608825b NOTE: cache registered group FLASH number=2 incarn=0x3a28825a NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a28825a NOTE: cache registered group PRD01 number=3 incarn=0xd3f8825c NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f8825c WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (not first) group 1/0xC608825B (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.38477330 ABA NOTE: seq=91 blk=6731 NOTE: cache mounting group 1/0xC608825B (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc608825b kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A28825A (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.51203938 ABA NOTE: seq=65 blk=387 NOTE: cache mounting group 2/0x3A28825A (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a28825a kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F8825C (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3085143 ABA NOTE: seq=14 blk=5102 NOTE: cache mounting group 3/0xD3F8825C (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f8825c kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sat Feb 25 17:03:56 2012 Starting background process ASMB Sat Feb 25 17:03:56 2012 ASMB started with pid=20, OS id=11374 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Feb 25 17:09:20 2012 Starting background process ASMB Sat Feb 25 17:09:20 2012 ASMB started with pid=23, OS id=15678 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Feb 25 17:31:57 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sat Feb 25 17:31:59 2012 PMON started with pid=2, OS id=9783 Sat Feb 25 17:31:59 2012 VKTM started with pid=3, OS id=9788 at elevated priority VKTM running at (20)ms precision Sat Feb 25 17:31:59 2012 DIAG started with pid=4, OS id=9815 Sat Feb 25 17:31:59 2012 PING started with pid=5, OS id=9817 Sat Feb 25 17:31:59 2012 PSP0 started with pid=6, OS id=9825 Sat Feb 25 17:31:59 2012 DIA0 started with pid=7, OS id=9827 Sat Feb 25 17:31:59 2012 LMON started with pid=8, OS id=9829 Sat Feb 25 17:31:59 2012 LMD0 started with pid=9, OS id=9831 Sat Feb 25 17:31:59 2012 LMS0 started with pid=10, OS id=9833 at elevated priority Sat Feb 25 17:31:59 2012 MMAN started with pid=11, OS id=9837 Sat Feb 25 17:31:59 2012 DBW0 started with pid=12, OS id=9839 Sat Feb 25 17:31:59 2012 LGWR started with pid=13, OS id=9841 Sat Feb 25 17:31:59 2012 CKPT started with pid=14, OS id=9846 Sat Feb 25 17:31:59 2012 SMON started with pid=15, OS id=9848 Sat Feb 25 17:31:59 2012 RBAL started with pid=16, OS id=9850 Sat Feb 25 17:31:59 2012 GMON started with pid=17, OS id=9852 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 76) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Feb 25 17:32:00 2012 LCK0 started with pid=18, OS id=9928 ORACLE_BASE from environment = /u01/app/oracle Sat Feb 25 17:32:02 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc60868cb NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc60868cb NOTE: cache registered group FLASH number=2 incarn=0x3a2868ca NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a2868ca NOTE: cache registered group PRD01 number=3 incarn=0xd3f868cc NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f868cc WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (not first) group 1/0xC60868CB (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.38477504 ABA NOTE: seq=92 blk=6819 NOTE: cache mounting group 1/0xC60868CB (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc60868cb kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A2868CA (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.51203938 ABA NOTE: seq=66 blk=388 NOTE: cache mounting group 2/0x3A2868CA (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a2868ca kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F868CC (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3085181 ABA NOTE: seq=15 blk=5122 NOTE: cache mounting group 3/0xD3F868CC (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f868cc kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sat Feb 25 17:32:13 2012 Starting background process ASMB Sat Feb 25 17:32:13 2012 ASMB started with pid=20, OS id=10583 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Feb 25 17:42:45 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sat Feb 25 17:42:47 2012 PMON started with pid=2, OS id=9812 Sat Feb 25 17:42:47 2012 VKTM started with pid=3, OS id=9814 at elevated priority Sat Feb 25 17:42:47 2012 DIAG started with pid=4, OS id=9824 VKTM running at (20)ms precision Sat Feb 25 17:42:47 2012 PING started with pid=5, OS id=9826 Sat Feb 25 17:42:47 2012 PSP0 started with pid=6, OS id=9828 Sat Feb 25 17:42:47 2012 Sat Feb 25 17:42:47 2012 DIA0 started with pid=7, OS id=9830 LMON started with pid=8, OS id=9832 Sat Feb 25 17:42:47 2012 LMD0 started with pid=9, OS id=9834 Sat Feb 25 17:42:47 2012 LMS0 started with pid=10, OS id=9836 at elevated priority Sat Feb 25 17:42:47 2012 MMAN started with pid=11, OS id=9840 Sat Feb 25 17:42:47 2012 DBW0 started with pid=12, OS id=9846 Sat Feb 25 17:42:47 2012 LGWR started with pid=13, OS id=9869 Sat Feb 25 17:42:47 2012 CKPT started with pid=14, OS id=9871 Sat Feb 25 17:42:47 2012 SMON started with pid=15, OS id=9876 Sat Feb 25 17:42:47 2012 RBAL started with pid=16, OS id=9878 Sat Feb 25 17:42:47 2012 GMON started with pid=17, OS id=9886 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 80) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Feb 25 17:42:48 2012 LCK0 started with pid=18, OS id=10002 ORACLE_BASE from environment = /u01/app/oracle Sat Feb 25 17:42:50 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc6086a06 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc6086a06 NOTE: cache registered group FLASH number=2 incarn=0x3a286a05 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a286a05 NOTE: cache registered group PRD01 number=3 incarn=0xd3f86a07 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f86a07 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (not first) group 1/0xC6086A06 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.38477504 ABA NOTE: seq=93 blk=6820 NOTE: cache mounting group 1/0xC6086A06 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc6086a06 kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A286A05 (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.51203938 ABA NOTE: seq=67 blk=389 NOTE: cache mounting group 2/0x3A286A05 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a286a05 kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F86A07 (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3085181 ABA NOTE: seq=16 blk=5123 NOTE: cache mounting group 3/0xD3F86A07 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f86a07 kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sat Feb 25 17:43:00 2012 Starting background process ASMB Sat Feb 25 17:43:00 2012 ASMB started with pid=20, OS id=10563 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Feb 25 17:43:47 2012 Starting background process ASMB Sat Feb 25 17:43:47 2012 ASMB started with pid=25, OS id=12101 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Fri Mar 16 22:19:55 2012 NOTE: ASM client dwh12:dwh1 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_12494.trc Fri Mar 16 22:19:55 2012 Trace dumping is performing id=[cdmp_20120316221955] Fri Mar 16 22:20:00 2012 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_11000.trc Trace dumping is performing id=[cdmp_20120316222000] Fri Mar 16 22:20:04 2012 Shutting down instance (abort) License high water mark = 10 USER (ospid: 13511): terminating the instance Instance terminated by USER, pid = 13511 Fri Mar 16 22:20:05 2012 Instance shutdown complete Fri Mar 16 22:26:22 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Fri Mar 16 22:26:24 2012 PMON started with pid=2, OS id=9645 Fri Mar 16 22:26:24 2012 VKTM started with pid=3, OS id=9647 at elevated priority Fri Mar 16 22:26:24 2012 DIAG started with pid=4, OS id=9651 VKTM running at (20)ms precision Fri Mar 16 22:26:24 2012 PING started with pid=5, OS id=9653 Fri Mar 16 22:26:25 2012 PSP0 started with pid=6, OS id=9655 Fri Mar 16 22:26:25 2012 DIA0 started with pid=7, OS id=9657 Fri Mar 16 22:26:25 2012 LMON started with pid=8, OS id=9659 Fri Mar 16 22:26:25 2012 LMD0 started with pid=9, OS id=9661 Fri Mar 16 22:26:25 2012 LMS0 started with pid=10, OS id=9663 at elevated priority Fri Mar 16 22:26:25 2012 MMAN started with pid=11, OS id=9667 Fri Mar 16 22:26:25 2012 DBW0 started with pid=12, OS id=9669 Fri Mar 16 22:26:25 2012 LGWR started with pid=13, OS id=9671 Fri Mar 16 22:26:25 2012 CKPT started with pid=14, OS id=9673 Fri Mar 16 22:26:25 2012 SMON started with pid=15, OS id=9675 Fri Mar 16 22:26:25 2012 RBAL started with pid=16, OS id=9677 Fri Mar 16 22:26:25 2012 GMON started with pid=17, OS id=9679 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 84) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Fri Mar 16 22:26:25 2012 LCK0 started with pid=18, OS id=9693 ORACLE_BASE from environment = /u01/app/oracle Fri Mar 16 22:26:27 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc6086343 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc6086343 NOTE: cache registered group FLASH number=2 incarn=0x3a286342 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a286342 NOTE: cache registered group PRD01 number=3 incarn=0xd3f86344 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f86344 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (not first) group 1/0xC6086343 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.38584256 ABA NOTE: seq=94 blk=8582 NOTE: cache mounting group 1/0xC6086343 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc6086343 kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A286342 (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.55866320 ABA NOTE: seq=72 blk=6581 NOTE: cache mounting group 2/0x3A286342 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a286342 kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F86344 (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3157675 ABA NOTE: seq=17 blk=5713 NOTE: cache mounting group 3/0xD3F86344 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f86344 kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Fri Mar 16 22:26:37 2012 Starting background process ASMB Fri Mar 16 22:26:37 2012 ASMB started with pid=20, OS id=9906 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Fri Mar 16 22:28:04 2012 Starting background process ASMB Fri Mar 16 22:28:04 2012 ASMB started with pid=25, OS id=13049 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Fri Mar 16 22:33:53 2012 NOTE: ASM client dwh12:dwh1 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_13251.trc Fri Mar 16 22:33:53 2012 Trace dumping is performing id=[cdmp_20120316223353] Fri Mar 16 22:33:58 2012 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_10402.trc Trace dumping is performing id=[cdmp_20120316223358] Fri Mar 16 22:34:02 2012 Shutting down instance (abort) License high water mark = 11 USER (ospid: 18166): terminating the instance Instance terminated by USER, pid = 18166 Fri Mar 16 22:34:03 2012 Instance shutdown complete Sat Mar 17 04:05:10 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sat Mar 17 04:05:11 2012 PMON started with pid=2, OS id=26505 Sat Mar 17 04:05:11 2012 VKTM started with pid=3, OS id=26507 at elevated priority VKTM running at (20)ms precision Sat Mar 17 04:05:11 2012 DIAG started with pid=4, OS id=26511 Sat Mar 17 04:05:11 2012 PING started with pid=5, OS id=26513 Sat Mar 17 04:05:11 2012 PSP0 started with pid=6, OS id=26515 Sat Mar 17 04:05:11 2012 DIA0 started with pid=7, OS id=26517 Sat Mar 17 04:05:11 2012 LMON started with pid=8, OS id=26519 Sat Mar 17 04:05:11 2012 LMD0 started with pid=9, OS id=26521 Sat Mar 17 04:05:11 2012 LMS0 started with pid=10, OS id=26528 at elevated priority Sat Mar 17 04:05:11 2012 MMAN started with pid=11, OS id=26532 Sat Mar 17 04:05:11 2012 DBW0 started with pid=12, OS id=26534 Sat Mar 17 04:05:11 2012 LGWR started with pid=13, OS id=26536 Sat Mar 17 04:05:11 2012 CKPT started with pid=14, OS id=26538 Sat Mar 17 04:05:11 2012 SMON started with pid=15, OS id=26540 Sat Mar 17 04:05:11 2012 RBAL started with pid=16, OS id=26542 Sat Mar 17 04:05:11 2012 GMON started with pid=17, OS id=26544 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 88) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Mar 17 04:05:12 2012 LCK0 started with pid=18, OS id=26555 ORACLE_BASE from environment = /u01/app/oracle Sat Mar 17 04:05:13 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc60f00c3 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc60f00c3 NOTE: cache registered group FLASH number=2 incarn=0x3a2f00c2 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a2f00c2 NOTE: cache registered group PRD01 number=3 incarn=0xd3ff00c4 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3ff00c4 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (not first) group 1/0xC60F00C3 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.38584434 ABA NOTE: seq=95 blk=8672 NOTE: cache mounting group 1/0xC60F00C3 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc60f00c3 kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A2F00C2 (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.55868227 ABA NOTE: seq=73 blk=6596 NOTE: cache mounting group 2/0x3A2F00C2 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a2f00c2 kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3FF00C4 (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3157896 ABA NOTE: seq=18 blk=5731 NOTE: cache mounting group 3/0xD3FF00C4 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3ff00c4 kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sat Mar 17 04:05:21 2012 Starting background process ASMB Sat Mar 17 04:05:21 2012 ASMB started with pid=20, OS id=26693 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Mar 17 04:06:22 2012 Starting background process ASMB Sat Mar 17 04:06:22 2012 ASMB started with pid=24, OS id=28033 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Mar 25 02:17:59 2012 Starting background process ASMB Sun Mar 25 02:17:59 2012 ASMB started with pid=22, OS id=3910 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Fri Apr 20 01:53:44 2012 Immediate Kill Session#: 81, Serial#: 46 Immediate Kill Session: sess: 0x70f96b78 OS pid: 4588 Fri Apr 20 01:53:44 2012 kfclDelLop: test event triggering SO deletion action=3 Fri Apr 20 02:36:00 2012 Immediate Kill Session#: 74, Serial#: 12 Immediate Kill Session: sess: 0x70f83c80 OS pid: 16212 Fri Apr 20 03:07:56 2012 Reconfiguration started (old inc 88, new inc 90) List of nodes: 1 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Apr 20 03:07:56 2012 LMS 0: 2 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5449 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Fri Apr 20 03:07:56 2012 NOTE: waiting for instance recovery of group 2 Fri Apr 20 03:07:56 2012 NOTE: waiting for instance recovery of group 2 Fri Apr 20 03:07:56 2012 NOTE: waiting for instance recovery of group 2 Fri Apr 20 03:07:56 2012 NOTE: waiting for instance recovery of group 2 Fri Apr 20 03:07:56 2012 NOTE: waiting for instance recovery of group 2 Fri Apr 20 03:07:56 2012 NOTE: waiting for instance recovery of group 2 Fri Apr 20 03:07:56 2012 NOTE: waiting for instance recovery of group 2 Fri Apr 20 03:07:56 2012 NOTE: waiting for instance recovery of group 2 Fri Apr 20 03:07:56 2012 NOTE: waiting for instance recovery of group 2 Fri Apr 20 03:07:56 2012 NOTE: waiting for instance recovery of group 2 Fri Apr 20 03:07:56 2012 NOTE: waiting for instance recovery of group 2 Fri Apr 20 03:07:56 2012 NOTE: waiting for instance recovery of group 2 Fri Apr 20 03:07:56 2012 NOTE: waiting for instance recovery of group 2 Fri Apr 20 03:07:56 2012 NOTE: waiting for instance recovery of group 2 Fri Apr 20 03:07:56 2012 kfdp_query(ORAFILES): 10 Fri Apr 20 03:07:56 2012 NOTE: PST enabling heartbeating (grp 2) NOTE: PST enabling heartbeating (grp 3) NOTE: PST enabling heartbeating (grp 1) kfdp_queryBg(): 10 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=2 ckpt=198.8787 group=1 NOTE: advancing ckpt for thread=2 ckpt=198.8787 NOTE: smon did instance recovery for domain 1 kfdp_query(FLASH): 11 kfdp_queryBg(): 11 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=1 ckpt=169.2600 group=2 NOTE: advancing ckpt for thread=1 ckpt=169.2600 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 12 kfdp_queryBg(): 12 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=1 ckpt=47.5989 group=3 NOTE: advancing ckpt for thread=1 ckpt=47.5989 NOTE: smon did instance recovery for domain 3 WARNING: System created alias index 605 WARNING: for File 623.781066677 not found WARNING: in alias directory of group FLASH (ignoring) NOTE: ASM did COD recovery for group 2/0x3a2f00c2 (FLASH) Fri Apr 20 03:19:53 2012 Reconfiguration started (old inc 90, new inc 92) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Fri Apr 20 03:19:53 2012 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 * domain 2 valid = 1 according to instance 0 * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Apr 20 03:19:54 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5460 GCS shadows traversed, 2765 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue May 08 15:50:49 2012 Trace dumping is performing id=[cdmp_20120508155048] Wed May 16 19:47:46 2012 Received detach msg from node 0 for dom 1 Received detach msg from node 0 for dom 2 Received detach msg from node 0 for dom 3 Wed May 16 19:47:55 2012 Reconfiguration started (old inc 92, new inc 94) List of nodes: 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed May 16 19:47:55 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5458 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Wed May 16 19:56:35 2012 Reconfiguration started (old inc 94, new inc 96) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Wed May 16 19:56:35 2012 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 * domain 2 valid = 1 according to instance 0 * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed May 16 19:56:35 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5458 GCS shadows traversed, 2779 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed May 16 20:24:24 2012 Trace dumping is performing id=[cdmp_20120516202423] Wed May 16 20:32:19 2012 Trace dumping is performing id=[cdmp_20120516203219] Wed May 16 20:32:20 2012 Reconfiguration started (old inc 96, new inc 98) List of nodes: 1 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed May 16 20:32:20 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5460 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Wed May 16 20:32:20 2012 kfdp_query(ORAFILES): 13 Reconfiguration complete Wed May 16 20:32:20 2012 kfdp_queryBg(): 13 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=2 ckpt=201.953 group=1 NOTE: advancing ckpt for thread=2 ckpt=201.953 NOTE: smon did instance recovery for domain 1 kfdp_query(FLASH): 14 kfdp_queryBg(): 14 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=1 ckpt=186.2515 group=2 NOTE: advancing ckpt for thread=1 ckpt=186.2515 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 15 kfdp_queryBg(): 15 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=1 ckpt=49.9253 group=3 NOTE: advancing ckpt for thread=1 ckpt=49.9253 NOTE: smon did instance recovery for domain 3 Wed May 16 20:48:37 2012 Reconfiguration started (old inc 98, new inc 100) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Wed May 16 20:48:37 2012 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 * domain 2 valid = 1 according to instance 0 * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed May 16 20:48:37 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5458 GCS shadows traversed, 2796 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Thu May 17 19:53:06 2012 Shutting down instance: further logons disabled Shutting down instance (immediate) License high water mark = 34 SQL> ALTER DISKGROUP ALL DISMOUNT NOTE: cache dismounting group 1/0xC60F00C3 (ORAFILES) Thu May 17 19:53:06 2012 kjbdomdet send to node 0 detach from dom 1, sending detach message to node 0 Thu May 17 19:53:06 2012 NOTE: detached from domain 1 NOTE: cache dismounted group 1/0xC60F00C3 (ORAFILES) kfdp_dismount(): 16 Thu May 17 19:53:06 2012 kfdp_dismountBg(): 16 NOTE: De-assigning number (1,0) from disk (/dev/mpath/mpath1p1) NOTE: De-assigning number (1,1) from disk (/dev/mpath/mpath2p1) NOTE: De-assigning number (1,2) from disk (/dev/mpath/mpath5p1) SUCCESS: diskgroup ORAFILES was dismounted NOTE: cache dismounting group 2/0x3A2F00C2 (FLASH) kjbdomdet send to node 0 detach from dom 2, sending detach message to node 0 NOTE: detached from domain 2 NOTE: cache dismounted group 2/0x3A2F00C2 (FLASH) kfdp_dismount(): 17 kfdp_dismountBg(): 17 NOTE: De-assigning number (2,0) from disk (/dev/mpath/mpath3p1) SUCCESS: diskgroup FLASH was dismounted NOTE: cache dismounting group 3/0xD3FF00C4 (PRD01) kjbdomdet send to node 0 detach from dom 3, sending detach message to node 0 NOTE: detached from domain 3 NOTE: cache dismounted group 3/0xD3FF00C4 (PRD01) kfdp_dismount(): 18 kfdp_dismountBg(): 18 NOTE: De-assigning number (3,0) from disk (/dev/mpath/mpath4p1) SUCCESS: diskgroup PRD01 was dismounted SUCCESS: ALTER DISKGROUP ALL DISMOUNT Thu May 17 19:53:09 2012 Stopping background process VKTM: Thu May 17 19:53:10 2012 freeing rdom 3 freeing rdom 2 freeing rdom 1 freeing rdom 0 Thu May 17 19:53:12 2012 Instance shutdown complete Thu May 17 20:47:50 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Thu May 17 20:47:51 2012 PMON started with pid=2, OS id=14241 Thu May 17 20:47:51 2012 VKTM started with pid=3, OS id=14243 at elevated priority Thu May 17 20:47:51 2012 DIAG started with pid=4, OS id=14248 VKTM running at (20)ms precision Thu May 17 20:47:51 2012 PING started with pid=5, OS id=14250 Thu May 17 20:47:51 2012 PSP0 started with pid=6, OS id=14252 Thu May 17 20:47:51 2012 DIA0 started with pid=7, OS id=14254 Thu May 17 20:47:51 2012 LMON started with pid=8, OS id=14256 Thu May 17 20:47:51 2012 LMD0 started with pid=9, OS id=14258 Thu May 17 20:47:51 2012 LMS0 started with pid=10, OS id=14260 at elevated priority Thu May 17 20:47:52 2012 MMAN started with pid=11, OS id=14264 Thu May 17 20:47:52 2012 DBW0 started with pid=12, OS id=14266 Thu May 17 20:47:52 2012 LGWR started with pid=13, OS id=14268 Thu May 17 20:47:52 2012 CKPT started with pid=14, OS id=14270 Thu May 17 20:47:52 2012 SMON started with pid=15, OS id=14272 Thu May 17 20:47:52 2012 RBAL started with pid=16, OS id=14274 Thu May 17 20:47:52 2012 GMON started with pid=17, OS id=14276 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 104) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Thu May 17 20:47:52 2012 LCK0 started with pid=18, OS id=14288 ORACLE_BASE from environment = /u01/app/oracle Thu May 17 20:47:54 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc608e099 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc608e099 NOTE: cache registered group FLASH number=2 incarn=0x3a28e098 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a28e098 NOTE: cache registered group PRD01 number=3 incarn=0xd3f8e09a NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f8e09a WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (not first) group 1/0xC608E099 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.38607007 ABA NOTE: seq=97 blk=2710 NOTE: cache mounting group 1/0xC608E099 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc608e099 kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A28E098 (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.68912453 ABA NOTE: seq=85 blk=9493 NOTE: cache mounting group 2/0x3A28E098 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a28e098 kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F8E09A (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3331978 ABA NOTE: seq=19 blk=7449 NOTE: cache mounting group 3/0xD3F8E09A (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f8e09a kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Thu May 17 20:48:04 2012 Starting background process ASMB Thu May 17 20:48:04 2012 ASMB started with pid=20, OS id=14499 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu May 17 20:48:52 2012 Starting background process ASMB Thu May 17 20:48:52 2012 ASMB started with pid=25, OS id=15993 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Sep 29 15:35:54 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Thu Sep 29 15:35:56 2011 PMON started with pid=2, OS id=10358 Thu Sep 29 15:35:56 2011 VKTM started with pid=3, OS id=10360 at elevated priority Thu Sep 29 15:35:56 2011 DIAG started with pid=4, OS id=10368 VKTM running at (20)ms precision Thu Sep 29 15:35:56 2011 PING started with pid=5, OS id=10370 Thu Sep 29 15:35:56 2011 PSP0 started with pid=6, OS id=10372 Thu Sep 29 15:35:56 2011 DIA0 started with pid=7, OS id=10380 Thu Sep 29 15:35:56 2011 LMON started with pid=8, OS id=10382 Thu Sep 29 15:35:56 2011 LMD0 started with pid=9, OS id=10384 Thu Sep 29 15:35:56 2011 LMS0 started with pid=10, OS id=10386 at elevated priority Thu Sep 29 15:35:56 2011 MMAN started with pid=11, OS id=10390 Thu Sep 29 15:35:57 2011 DBW0 started with pid=12, OS id=10392 Thu Sep 29 15:35:57 2011 LGWR started with pid=13, OS id=10394 Thu Sep 29 15:35:57 2011 CKPT started with pid=14, OS id=10396 Thu Sep 29 15:35:57 2011 SMON started with pid=15, OS id=10398 Thu Sep 29 15:35:57 2011 RBAL started with pid=16, OS id=10400 Thu Sep 29 15:35:57 2011 GMON started with pid=17, OS id=10402 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 108) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Thu Sep 29 15:35:57 2011 LCK0 started with pid=18, OS id=10407 ORACLE_BASE from environment = /u01/app/oracle Thu Sep 29 15:35:59 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc6086ce5 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc6086ce5 NOTE: cache registered group FLASH number=2 incarn=0x3a286ce4 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a286ce4 NOTE: cache registered group PRD01 number=3 incarn=0xd3f86ce6 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f86ce6 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (not first) group 1/0xC6086CE5 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.38638470 ABA NOTE: seq=98 blk=6568 NOTE: cache mounting group 1/0xC6086CE5 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc6086ce5 kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A286CE4 (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.74451969 ABA NOTE: seq=91 blk=10250 NOTE: cache mounting group 2/0x3A286CE4 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a286ce4 kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F86CE6 (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3399458 ABA NOTE: seq=20 blk=8247 NOTE: cache mounting group 3/0xD3F86CE6 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f86ce6 kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Thu Sep 29 15:36:10 2011 Starting background process ASMB Thu Sep 29 15:36:10 2011 ASMB started with pid=20, OS id=10821 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Sep 29 15:36:50 2011 Starting background process ASMB Thu Sep 29 15:36:50 2011 ASMB started with pid=24, OS id=12082 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed Aug 15 07:28:15 2012 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_10368.trc Wed Aug 15 07:34:07 2012 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_10368.trc Wed Aug 15 08:15:19 2012 Trace dumping is performing id=[cdmp_20120815081519] Wed Aug 15 08:16:57 2012 NOTE: ASM client dwh12:dwh1 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_12296.trc Wed Aug 15 08:16:57 2012 Trace dumping is performing id=[cdmp_20120815081657] Wed Aug 15 08:17:01 2012 Starting background process ASMB Wed Aug 15 08:17:01 2012 ASMB started with pid=22, OS id=32151 Wed Aug 15 08:18:27 2012 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_11182.trc Wed Aug 15 08:18:27 2012 Trace dumping is performing id=[cdmp_20120815081827] Wed Aug 15 08:19:42 2012 Shutting down instance (abort) License high water mark = 11 USER (ospid: 3212): terminating the instance Instance terminated by USER, pid = 3212 Wed Aug 15 08:19:43 2012 Instance shutdown complete Wed Aug 15 08:21:51 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Wed Aug 15 08:21:52 2012 PMON started with pid=2, OS id=8143 Wed Aug 15 08:21:52 2012 VKTM started with pid=3, OS id=8145 at elevated priority VKTM running at (20)ms precision Wed Aug 15 08:21:52 2012 DIAG started with pid=4, OS id=8149 Wed Aug 15 08:21:52 2012 PING started with pid=5, OS id=8151 Wed Aug 15 08:21:52 2012 PSP0 started with pid=6, OS id=8153 Wed Aug 15 08:21:52 2012 DIA0 started with pid=7, OS id=8155 Wed Aug 15 08:21:52 2012 LMON started with pid=8, OS id=8157 Wed Aug 15 08:21:52 2012 LMD0 started with pid=9, OS id=8159 Wed Aug 15 08:21:52 2012 LMS0 started with pid=10, OS id=8161 at elevated priority Wed Aug 15 08:21:52 2012 MMAN started with pid=11, OS id=8165 Wed Aug 15 08:21:52 2012 DBW0 started with pid=12, OS id=8167 Wed Aug 15 08:21:52 2012 LGWR started with pid=13, OS id=8169 Wed Aug 15 08:21:52 2012 CKPT started with pid=14, OS id=8171 Wed Aug 15 08:21:52 2012 SMON started with pid=15, OS id=8173 Wed Aug 15 08:21:52 2012 RBAL started with pid=16, OS id=8175 Wed Aug 15 08:21:52 2012 GMON started with pid=17, OS id=8177 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 112) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Aug 15 08:21:53 2012 LCK0 started with pid=18, OS id=8261 ORACLE_BASE from environment = /u01/app/oracle Wed Aug 15 08:21:53 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc604b925 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc604b925 NOTE: cache registered group FLASH number=2 incarn=0x3a24b924 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a24b924 NOTE: cache registered group PRD01 number=3 incarn=0xd3f4b926 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f4b926 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (not first) group 1/0xC604B925 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.38655295 ABA NOTE: seq=99 blk=10030 NOTE: cache mounting group 1/0xC604B925 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc604b925 kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A24B924 (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.83804877 ABA NOTE: seq=101 blk=7276 NOTE: cache mounting group 2/0x3A24B924 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a24b924 kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F4B926 (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3542587 ABA NOTE: seq=21 blk=9735 NOTE: cache mounting group 3/0xD3F4B926 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f4b926 kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 Wed Aug 15 08:24:11 2012 Reconfiguration started (old inc 112, new inc 114) List of nodes: 1 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Aug 15 08:24:11 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 17 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Wed Aug 15 08:24:11 2012 SUCCESS: diskgroup ORAFILES was mounted NOTE: waiting for instance recovery of group 2 Wed Aug 15 08:24:11 2012 kfdp_query(FLASH): 8 Wed Aug 15 08:24:11 2012 kfdp_query(ORAFILES): 8 Wed Aug 15 08:24:11 2012 NOTE: PST enabling heartbeating (grp 2) NOTE: PST enabling heartbeating (grp 3) NOTE: PST enabling heartbeating (grp 1) kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 NOTE: waiting for instance recovery of group 2 kfdp_queryBg(): 9 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=2 ckpt=202.8510 group=1 NOTE: advancing ckpt for thread=2 ckpt=202.8510 NOTE: smon did instance recovery for domain 1 kfdp_query(FLASH): 10 kfdp_queryBg(): 10 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=1 ckpt=227.3574 group=2 NOTE: advancing ckpt for thread=1 ckpt=227.3574 NOTE: waiting for instance recovery of group 3 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 11 kfdp_query(PRD01): 11 kfdp_queryBg(): 11 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 NOTE: waiting for instance recovery of group 3 kfdp_queryBg(): 12 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=1 ckpt=51.6441 group=3 NOTE: advancing ckpt for thread=1 ckpt=51.6441 NOTE: smon did instance recovery for domain 3 kfdp_query(FLASH): 15 kfdp_queryBg(): 15 NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a24b924 kfdp_query(PRD01): 16 kfdp_queryBg(): 16 NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f4b926 kfdp_query(FLASH): 17 kfdp_queryBg(): 17 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 18 kfdp_queryBg(): 18 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT NOTE: ASM did COD recovery for group 2/0x3a24b924 (FLASH) Wed Aug 15 08:24:17 2012 Starting background process ASMB Wed Aug 15 08:24:17 2012 ASMB started with pid=20, OS id=10713 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed Aug 15 08:33:22 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Wed Aug 15 08:33:23 2012 PMON started with pid=2, OS id=10163 Wed Aug 15 08:33:23 2012 VKTM started with pid=3, OS id=10165 at elevated priority VKTM running at (20)ms precision Wed Aug 15 08:33:23 2012 DIAG started with pid=4, OS id=10170 Wed Aug 15 08:33:23 2012 PING started with pid=5, OS id=10172 Wed Aug 15 08:33:23 2012 PSP0 started with pid=6, OS id=10174 Wed Aug 15 08:33:23 2012 DIA0 started with pid=7, OS id=10176 Wed Aug 15 08:33:23 2012 LMON started with pid=8, OS id=10178 Wed Aug 15 08:33:23 2012 LMD0 started with pid=9, OS id=10180 Wed Aug 15 08:33:23 2012 LMS0 started with pid=10, OS id=10183 at elevated priority Wed Aug 15 08:33:23 2012 MMAN started with pid=11, OS id=10187 Wed Aug 15 08:33:23 2012 DBW0 started with pid=12, OS id=10189 Wed Aug 15 08:33:23 2012 LGWR started with pid=13, OS id=10191 Wed Aug 15 08:33:24 2012 CKPT started with pid=14, OS id=10194 Wed Aug 15 08:33:24 2012 SMON started with pid=15, OS id=10196 Wed Aug 15 08:33:24 2012 RBAL started with pid=16, OS id=10198 Wed Aug 15 08:33:24 2012 GMON started with pid=17, OS id=10200 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 2) ASM instance List of nodes: 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Wed Aug 15 08:33:24 2012 LCK0 started with pid=19, OS id=10222 Reconfiguration started (old inc 2, new inc 4) List of nodes: 0 1 ORACLE_BASE from environment = /u01/app/oracle Global Resource Directory frozen Communication channels reestablished * domain 0 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Aug 15 08:33:26 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group FLASH number=1 incarn=0x97989864 NOTE: cache began mount (first) of group FLASH number=1 incarn=0x97989864 NOTE: cache registered group ORAFILES number=2 incarn=0x97a89865 NOTE: cache began mount (first) of group ORAFILES number=2 incarn=0x97a89865 NOTE: cache registered group PRD01 number=3 incarn=0x97a89866 NOTE: cache began mount (first) of group PRD01 number=3 incarn=0x97a89866 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (2,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (2,1) to disk (/dev/mpath/mpath2p1) ERROR: no PST quorum in group 2: required 1, found 0 NOTE: cache dismounting group 2/0x97A89865 (ORAFILES) NOTE: dbwr not being msg'd to dismount NOTE: lgwr not being msg'd to dismount NOTE: cache dismounted group 2/0x97A89865 (ORAFILES) NOTE: cache ending mount (fail) of group ORAFILES number=2 incarn=0x97a89865 kfdp_dismount(): 3 kfdp_dismountBg(): 3 NOTE: De-assigning number (2,1) from disk (/dev/mpath/mpath2p1) NOTE: De-assigning number (2,2) from disk (/dev/mpath/mpath5p1) ERROR: diskgroup ORAFILES was not mounted NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) NOTE: start heartbeating (grp 1) kfdp_query(FLASH): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 1: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (first) group 1/0x97989864 (FLASH) * allocate domain 1, invalid = TRUE kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: starting recovery of thread=2 ckpt=101.7321 group=1 NOTE: advancing ckpt for thread=2 ckpt=101.7321 NOTE: cache recovered group 1 to fcn 0.83805827 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.83798800 ABA NOTE: seq=228 blk=3575 NOTE: cache mounting group 1/0x97989864 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=1 incarn=0x97989864 NOTE: start heartbeating (grp 3) kfdp_query(PRD01): 8 kfdp_queryBg(): 8 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (first) group 3/0x97A89866 (PRD01) * allocate domain 3, invalid = TRUE kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: starting recovery of thread=2 ckpt=21.9772 group=3 NOTE: advancing ckpt for thread=2 ckpt=21.9772 NOTE: cache recovered group 3 to fcn 0.3543569 NOTE: LGWR attempting to mount thread 1 for diskgroup 3 NOTE: LGWR mounted thread 1 for disk group 3 NOTE: opening chunk 1 at fcn 0.3543473 ABA NOTE: seq=52 blk=6442 NOTE: cache mounting group 3/0x97A89866 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0x97a89866 kfdp_query(FLASH): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 10 kfdp_queryBg(): 10 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted ORA-15032: not all alterations performed ORA-15063: ASM discovered an insufficient number of disks for diskgroup "ORAFILES" ERROR: ALTER DISKGROUP ALL MOUNT Wed Aug 15 08:33:37 2012 * allocate domain 2, invalid = TRUE Wed Aug 15 08:33:41 2012 Starting background process ASMB Wed Aug 15 08:33:41 2012 ASMB started with pid=20, OS id=10723 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed Aug 15 08:34:07 2012 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_11069.trc Wed Aug 15 08:34:07 2012 Trace dumping is performing id=[cdmp_20120815083407] Wed Aug 15 08:35:54 2012 Starting background process ASMB Wed Aug 15 08:35:54 2012 ASMB started with pid=21, OS id=12878 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed Aug 15 08:43:28 2012 SQL> alter diskgroup orafiles mount WARNING: Deprecated privilege SYSDBA for command 'ALTER DISKGROUP MOUNT' NOTE: cache registered group ORAFILES number=2 incarn=0x97a89869 NOTE: cache began mount (not first) of group ORAFILES number=2 incarn=0x97a89869 NOTE: Assigning number (2,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (2,1) to disk (/dev/mpath/mpath2p1) Wed Aug 15 08:43:28 2012 ERROR: no PST quorum in group 2: required 1, found 0 NOTE: cache dismounting group 2/0x97A89869 (ORAFILES) NOTE: dbwr not being msg'd to dismount NOTE: lgwr not being msg'd to dismount NOTE: cache dismounted group 2/0x97A89869 (ORAFILES) NOTE: cache ending mount (fail) of group ORAFILES number=2 incarn=0x97a89869 kfdp_dismount(): 12 kfdp_dismountBg(): 12 NOTE: De-assigning number (2,1) from disk (/dev/mpath/mpath2p1) NOTE: De-assigning number (2,2) from disk (/dev/mpath/mpath5p1) ERROR: diskgroup ORAFILES was not mounted ORA-15032: not all alterations performed ORA-15063: ASM discovered an insufficient number of disks for diskgroup "ORAFILES" ERROR: alter diskgroup orafiles mount Wed Aug 15 08:47:45 2012 SQL> alter diskgroup ORAFILES mount WARNING: Deprecated privilege SYSDBA for command 'ALTER DISKGROUP MOUNT' NOTE: cache registered group ORAFILES number=2 incarn=0x97a8986f NOTE: cache began mount (not first) of group ORAFILES number=2 incarn=0x97a8986f NOTE: Assigning number (2,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (2,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath1p1) kfdp_query(ORAFILES): 14 Wed Aug 15 08:47:45 2012 kfdp_queryBg(): 14 NOTE: cache opening disk 0 of grp 2: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 2: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 2: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (not first) group 2/0x97A8986F (ORAFILES) Wed Aug 15 08:47:45 2012 kjbdomatt send to node 0 Wed Aug 15 08:47:46 2012 NOTE: attached to recovery domain 2 Wed Aug 15 08:47:46 2012 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.38655623 ABA NOTE: seq=203 blk=8511 NOTE: cache mounting group 2/0x97A8986F (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=2 incarn=0x97a8986f Wed Aug 15 08:47:46 2012 kfdp_query(ORAFILES): 15 kfdp_queryBg(): 15 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: alter diskgroup ORAFILES mount Wed Aug 15 08:48:26 2012 Starting background process ASMB Wed Aug 15 08:48:26 2012 ASMB started with pid=21, OS id=27085 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Aug 16 22:08:12 2012 Trace dumping is performing id=[cdmp_20120816220812] Trace dumping is performing id=[cdmp_20120816220817] Thu Aug 16 22:08:18 2012 Reconfiguration started (old inc 4, new inc 6) List of nodes: 1 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Thu Aug 16 22:08:19 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5405 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Thu Aug 16 22:08:19 2012 kfdp_query(FLASH): 16 Thu Aug 16 22:08:19 2012 NOTE: PST enabling heartbeating (grp 2) kfdp_queryBg(): 16 Thu Aug 16 22:08:19 2012 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_13161.trc NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=103.382 group=1 Trace dumping is performing id=[cdmp_20120816220819] NOTE: advancing ckpt for thread=2 ckpt=103.382 NOTE: smon did instance recovery for domain 1 kfdp_query(ORAFILES): 17 kfdp_queryBg(): 17 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=100.10185 group=2 NOTE: advancing ckpt for thread=1 ckpt=100.10185 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 18 kfdp_queryBg(): 18 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=22.10040 group=3 NOTE: advancing ckpt for thread=2 ckpt=22.10040 NOTE: smon did instance recovery for domain 3 Thu Aug 16 22:08:22 2012 Starting background process ASMB Thu Aug 16 22:08:22 2012 ASMB started with pid=23, OS id=19303 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Aug 16 22:09:20 2012 Reconfiguration started (old inc 6, new inc 8) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Thu Aug 16 22:09:20 2012 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 * domain 3 valid = 1 according to instance 0 * domain 2 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Thu Aug 16 22:09:20 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5405 GCS shadows traversed, 2736 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Thu Aug 16 22:12:43 2012 NOTE: ASM client dwh12:dwh1 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_27489.trc Thu Aug 16 22:12:43 2012 Trace dumping is performing id=[cdmp_20120816221243] Thu Aug 16 22:12:48 2012 NOTE: ASM client PRD012:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_ora_19699.trc Trace dumping is performing id=[cdmp_20120816221248] Thu Aug 16 22:12:53 2012 Shutting down instance (abort) License high water mark = 9 USER (ospid: 24196): terminating the instance Instance terminated by USER, pid = 24196 Thu Aug 16 22:12:54 2012 Instance shutdown complete Thu Aug 16 22:13:32 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "ORCL:*" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Thu Aug 16 22:13:32 2012 PMON started with pid=2, OS id=26879 Thu Aug 16 22:13:32 2012 VKTM started with pid=3, OS id=26881 at elevated priority VKTM running at (20)ms precision Thu Aug 16 22:13:32 2012 DIAG started with pid=4, OS id=26885 Thu Aug 16 22:13:32 2012 PING started with pid=5, OS id=26887 Thu Aug 16 22:13:32 2012 PSP0 started with pid=6, OS id=26889 Thu Aug 16 22:13:32 2012 DIA0 started with pid=7, OS id=26891 Thu Aug 16 22:13:32 2012 LMON started with pid=8, OS id=26893 Thu Aug 16 22:13:32 2012 LMD0 started with pid=9, OS id=26895 Thu Aug 16 22:13:32 2012 LMS0 started with pid=10, OS id=26898 at elevated priority Thu Aug 16 22:13:32 2012 MMAN started with pid=11, OS id=26902 Thu Aug 16 22:13:32 2012 DBW0 started with pid=12, OS id=26904 Thu Aug 16 22:13:33 2012 LGWR started with pid=13, OS id=26906 Thu Aug 16 22:13:33 2012 CKPT started with pid=14, OS id=26908 Thu Aug 16 22:13:33 2012 SMON started with pid=15, OS id=26910 Thu Aug 16 22:13:33 2012 RBAL started with pid=16, OS id=26912 Thu Aug 16 22:13:33 2012 GMON started with pid=17, OS id=26914 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 12) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Thu Aug 16 22:13:33 2012 LCK0 started with pid=18, OS id=26973 ORACLE_BASE from environment = /u01/app/oracle Thu Aug 16 22:13:34 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group FLASH number=1 incarn=0x97977ca7 NOTE: cache began mount (not first) of group FLASH number=1 incarn=0x97977ca7 NOTE: cache registered group ORAFILES number=2 incarn=0x97a77ca8 NOTE: cache began mount (not first) of group ORAFILES number=2 incarn=0x97a77ca8 NOTE: cache registered group PRD01 number=3 incarn=0x97a77ca9 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0x97a77ca9 NOTE:Loaded lib: /opt/oracle/extapi/64/asm/orcl/1/libasm.so NOTE: Assigning number (1,0) to disk (ORCL:FLASH0) NOTE: Assigning number (2,0) to disk (ORCL:ORAFILES0) NOTE: Assigning number (2,1) to disk (ORCL:ORAFILES1) NOTE: Assigning number (2,2) to disk (ORCL:ORAFILES2) NOTE: Assigning number (3,0) to disk (ORCL:PRD010) kfdp_query(FLASH): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: FLASH_0000 label:FLASH0 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x97977CA7 (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.83951863 ABA NOTE: seq=229 blk=5363 NOTE: cache mounting group 1/0x97977CA7 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=1 incarn=0x97977ca7 kfdp_query(ORAFILES): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: ORAFILES_0000 label:ORAFILES0 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 2: ORAFILES_0001 label:ORAFILES1 NOTE: cache opening disk 2 of grp 2: ORAFILES_0002 label:ORAFILES2 NOTE: cache mounting (not first) group 2/0x97A77CA8 (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.38656017 ABA NOTE: seq=204 blk=8584 NOTE: cache mounting group 2/0x97A77CA8 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=2 incarn=0x97a77ca8 kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 label:PRD010 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0x97A77CA9 (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 1 for diskgroup 3 NOTE: LGWR mounted thread 1 for disk group 3 NOTE: opening chunk 1 at fcn 0.3549085 ABA NOTE: seq=53 blk=6508 NOTE: cache mounting group 3/0x97A77CA9 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0x97a77ca9 kfdp_query(FLASH): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup FLASH was mounted kfdp_query(ORAFILES): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Thu Aug 16 22:13:41 2012 Starting background process ASMB Thu Aug 16 22:13:41 2012 ASMB started with pid=20, OS id=27131 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Aug 16 22:14:28 2012 Starting background process ASMB Thu Aug 16 22:14:28 2012 ASMB started with pid=25, OS id=28522 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed Sep 19 19:53:53 2012 Received detach msg from node 0 for dom 1 Wed Sep 19 19:53:54 2012 NOTE: PST enabling heartbeating (grp 1) Received detach msg from node 0 for dom 2 NOTE: PST enabling heartbeating (grp 2) Received detach msg from node 0 for dom 3 NOTE: PST enabling heartbeating (grp 3) Wed Sep 19 19:53:58 2012 Reconfiguration started (old inc 12, new inc 14) List of nodes: 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Sep 19 19:53:58 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5405 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Wed Sep 19 20:01:38 2012 Reconfiguration started (old inc 14, new inc 16) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Wed Sep 19 20:01:38 2012 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 * domain 3 valid = 1 according to instance 0 * domain 2 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Sep 19 20:01:38 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5405 GCS shadows traversed, 2730 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Sep 19 20:21:24 2012 Shutting down instance: further logons disabled Shutting down instance (immediate) License high water mark = 12 SQL> ALTER DISKGROUP ALL DISMOUNT NOTE: cache dismounting group 1/0x97977CA7 (FLASH) Wed Sep 19 20:21:24 2012 kjbdomdet send to node 0 detach from dom 1, sending detach message to node 0 Wed Sep 19 20:21:24 2012 NOTE: detached from domain 1 NOTE: cache dismounted group 1/0x97977CA7 (FLASH) kfdp_dismount(): 10 Wed Sep 19 20:21:25 2012 kfdp_dismountBg(): 10 NOTE: De-assigning number (1,0) from disk (ORCL:FLASH0) SUCCESS: diskgroup FLASH was dismounted NOTE: cache dismounting group 2/0x97A77CA8 (ORAFILES) kjbdomdet send to node 0 detach from dom 2, sending detach message to node 0 NOTE: detached from domain 2 NOTE: cache dismounted group 2/0x97A77CA8 (ORAFILES) kfdp_dismount(): 11 kfdp_dismountBg(): 11 NOTE: De-assigning number (2,0) from disk (ORCL:ORAFILES0) NOTE: De-assigning number (2,1) from disk (ORCL:ORAFILES1) NOTE: De-assigning number (2,2) from disk (ORCL:ORAFILES2) SUCCESS: diskgroup ORAFILES was dismounted NOTE: cache dismounting group 3/0x97A77CA9 (PRD01) kjbdomdet send to node 0 detach from dom 3, sending detach message to node 0 NOTE: detached from domain 3 NOTE: cache dismounted group 3/0x97A77CA9 (PRD01) kfdp_dismount(): 12 kfdp_dismountBg(): 12 NOTE: De-assigning number (3,0) from disk (ORCL:PRD010) SUCCESS: diskgroup PRD01 was dismounted SUCCESS: ALTER DISKGROUP ALL DISMOUNT Wed Sep 19 20:21:27 2012 Stopping background process VKTM: Wed Sep 19 20:21:29 2012 freeing rdom 2 freeing rdom 3 freeing rdom 1 freeing rdom 0 Wed Sep 19 20:21:31 2012 Instance shutdown complete Wed Sep 19 20:30:33 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "ORCL:*" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Wed Sep 19 20:30:35 2012 PMON started with pid=2, OS id=10923 Wed Sep 19 20:30:35 2012 VKTM started with pid=3, OS id=10925 at elevated priority Wed Sep 19 20:30:35 2012 DIAG started with pid=4, OS id=10930 VKTM running at (20)ms precision Wed Sep 19 20:30:35 2012 PING started with pid=5, OS id=10932 Wed Sep 19 20:30:35 2012 PSP0 started with pid=6, OS id=10934 Wed Sep 19 20:30:35 2012 DIA0 started with pid=7, OS id=10936 Wed Sep 19 20:30:35 2012 LMON started with pid=8, OS id=10938 Wed Sep 19 20:30:35 2012 LMD0 started with pid=9, OS id=10940 Wed Sep 19 20:30:35 2012 LMS0 started with pid=10, OS id=10942 at elevated priority Wed Sep 19 20:30:35 2012 MMAN started with pid=11, OS id=10946 Wed Sep 19 20:30:35 2012 DBW0 started with pid=12, OS id=10948 Wed Sep 19 20:30:35 2012 LGWR started with pid=13, OS id=10950 Wed Sep 19 20:30:35 2012 CKPT started with pid=14, OS id=10952 Wed Sep 19 20:30:35 2012 SMON started with pid=15, OS id=10954 Wed Sep 19 20:30:35 2012 RBAL started with pid=16, OS id=10956 Wed Sep 19 20:30:35 2012 GMON started with pid=17, OS id=10958 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 20) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Sep 19 20:30:36 2012 LCK0 started with pid=18, OS id=10970 ORACLE_BASE from environment = /u01/app/oracle Wed Sep 19 20:30:37 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group FLASH number=1 incarn=0x97988c0e NOTE: cache began mount (not first) of group FLASH number=1 incarn=0x97988c0e NOTE: cache registered group ORAFILES number=2 incarn=0x97a88c0f NOTE: cache began mount (not first) of group ORAFILES number=2 incarn=0x97a88c0f NOTE: cache registered group PRD01 number=3 incarn=0x97a88c10 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0x97a88c10 NOTE:Loaded lib: /opt/oracle/extapi/64/asm/orcl/1/libasm.so NOTE: Assigning number (1,0) to disk (ORCL:FLASH0) NOTE: Assigning number (2,0) to disk (ORCL:ORAFILES0) NOTE: Assigning number (2,1) to disk (ORCL:ORAFILES1) NOTE: Assigning number (2,2) to disk (ORCL:ORAFILES2) NOTE: Assigning number (3,0) to disk (ORCL:PRD010) kfdp_query(FLASH): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: FLASH_0000 label:FLASH0 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x97988C0E (FLASH) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.88410913 ABA NOTE: seq=235 blk=2864 NOTE: cache mounting group 1/0x97988C0E (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=1 incarn=0x97988c0e kfdp_query(ORAFILES): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: ORAFILES_0000 label:ORAFILES0 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 2: ORAFILES_0001 label:ORAFILES1 NOTE: cache opening disk 2 of grp 2: ORAFILES_0002 label:ORAFILES2 NOTE: cache mounting (not first) group 2/0x97A88C0F (ORAFILES) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.38660957 ABA NOTE: seq=205 blk=10376 NOTE: cache mounting group 2/0x97A88C0F (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=2 incarn=0x97a88c0f kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 label:PRD010 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0x97A88C10 (PRD01) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 1 for diskgroup 3 NOTE: LGWR mounted thread 1 for disk group 3 NOTE: opening chunk 1 at fcn 0.3646347 ABA NOTE: seq=54 blk=7326 NOTE: cache mounting group 3/0x97A88C10 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0x97a88c10 kfdp_query(FLASH): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup FLASH was mounted kfdp_query(ORAFILES): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Wed Sep 19 20:30:50 2012 Starting background process ASMB Wed Sep 19 20:30:50 2012 ASMB started with pid=20, OS id=11334 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed Sep 19 20:31:43 2012 Starting background process ASMB Wed Sep 19 20:31:43 2012 ASMB started with pid=24, OS id=12887 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed Sep 19 20:36:45 2012 * allocate domain 4, invalid = TRUE freeing rdom 4 Received detach msg from node 0 for dom 4 * allocate domain 4, invalid = TRUE Wed Sep 19 20:37:08 2012 SQL> alter diskgroup redoa mount WARNING: Deprecated privilege SYSDBA for command 'ALTER DISKGROUP MOUNT' NOTE: cache registered group REDOA number=4 incarn=0x79f88c12 NOTE: cache began mount (not first) of group REDOA number=4 incarn=0x79f88c12 NOTE: Assigning number (4,0) to disk (ORCL:REDOA0) kfdp_query(REDOA): 11 Wed Sep 19 20:37:08 2012 kfdp_queryBg(): 11 NOTE: cache opening disk 0 of grp 4: REDOA0 label:REDOA0 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 4/0x79F88C12 (REDOA) Wed Sep 19 20:37:08 2012 kjbdomatt send to node 0 Wed Sep 19 20:37:08 2012 NOTE: attached to recovery domain 4 Wed Sep 19 20:37:08 2012 NOTE: LGWR attempting to mount thread 2 for diskgroup 4 NOTE: ACD expansion required for disk group 4 NOTE: LGWR attempting to mount thread 2 for diskgroup 4 NOTE: LGWR mounted thread 2 for disk group 4 NOTE: opening chunk 2 at fcn 0.0 ABA NOTE: seq=2 blk=0 NOTE: cache mounting group 4/0x79F88C12 (REDOA) succeeded NOTE: cache ending mount (success) of group REDOA number=4 incarn=0x79f88c12 Wed Sep 19 20:37:10 2012 kfdp_query(REDOA): 12 kfdp_queryBg(): 12 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 4 SUCCESS: diskgroup REDOA was mounted SUCCESS: alter diskgroup redoa mount Wed Sep 19 20:37:19 2012 * allocate domain 5, invalid = TRUE freeing rdom 5 Received detach msg from node 0 for dom 5 * allocate domain 5, invalid = TRUE Wed Sep 19 20:37:27 2012 SQL> alter diskgroup redob mount WARNING: Deprecated privilege SYSDBA for command 'ALTER DISKGROUP MOUNT' NOTE: cache registered group REDOB number=5 incarn=0x48588c14 NOTE: cache began mount (not first) of group REDOB number=5 incarn=0x48588c14 NOTE: Assigning number (5,0) to disk (ORCL:REDOB0) kfdp_query(REDOB): 14 Wed Sep 19 20:37:27 2012 kfdp_queryBg(): 14 NOTE: cache opening disk 0 of grp 5: REDOB0 label:REDOB0 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 5/0x48588C14 (REDOB) kjbdomatt send to node 0 Wed Sep 19 20:37:28 2012 NOTE: attached to recovery domain 5 Wed Sep 19 20:37:28 2012 NOTE: LGWR attempting to mount thread 2 for diskgroup 5 NOTE: ACD expansion required for disk group 5 NOTE: LGWR attempting to mount thread 2 for diskgroup 5 NOTE: LGWR mounted thread 2 for disk group 5 NOTE: opening chunk 2 at fcn 0.0 ABA NOTE: seq=2 blk=0 NOTE: cache mounting group 5/0x48588C14 (REDOB) succeeded NOTE: cache ending mount (success) of group REDOB number=5 incarn=0x48588c14 Wed Sep 19 20:37:31 2012 kfdp_query(REDOB): 15 kfdp_queryBg(): 15 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 5 SUCCESS: diskgroup REDOB was mounted SUCCESS: alter diskgroup redob mount Wed Sep 19 20:41:45 2012 Received detach msg from node 0 for dom 1 Wed Sep 19 20:41:45 2012 NOTE: PST enabling heartbeating (grp 1) Received detach msg from node 0 for dom 2 NOTE: PST enabling heartbeating (grp 2) Received detach msg from node 0 for dom 3 NOTE: PST enabling heartbeating (grp 3) Received detach msg from node 0 for dom 4 NOTE: PST enabling heartbeating (grp 4) Received detach msg from node 0 for dom 5 NOTE: PST enabling heartbeating (grp 5) Wed Sep 19 20:41:49 2012 Reconfiguration started (old inc 20, new inc 22) List of nodes: 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Sep 19 20:41:49 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 1762 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Wed Sep 19 20:42:06 2012 Shutting down instance: further logons disabled Shutting down instance (immediate) License high water mark = 7 SQL> ALTER DISKGROUP ALL DISMOUNT NOTE: cache dismounting group 1/0x97988C0E (FLASH) Wed Sep 19 20:42:06 2012 freeing rdom 1 Wed Sep 19 20:42:06 2012 NOTE: detached from domain 1 NOTE: cache dismounted group 1/0x97988C0E (FLASH) kfdp_dismount(): 16 Wed Sep 19 20:42:06 2012 kfdp_dismountBg(): 16 NOTE: De-assigning number (1,0) from disk (ORCL:FLASH0) SUCCESS: diskgroup FLASH was dismounted NOTE: cache dismounting group 2/0x97A88C0F (ORAFILES) freeing rdom 2 NOTE: detached from domain 2 NOTE: cache dismounted group 2/0x97A88C0F (ORAFILES) kfdp_dismount(): 17 kfdp_dismountBg(): 17 NOTE: De-assigning number (2,0) from disk (ORCL:ORAFILES0) NOTE: De-assigning number (2,1) from disk (ORCL:ORAFILES1) NOTE: De-assigning number (2,2) from disk (ORCL:ORAFILES2) SUCCESS: diskgroup ORAFILES was dismounted NOTE: cache dismounting group 3/0x97A88C10 (PRD01) freeing rdom 3 NOTE: detached from domain 3 NOTE: cache dismounted group 3/0x97A88C10 (PRD01) kfdp_dismount(): 18 kfdp_dismountBg(): 18 NOTE: De-assigning number (3,0) from disk (ORCL:PRD010) SUCCESS: diskgroup PRD01 was dismounted NOTE: cache dismounting group 4/0x79F88C12 (REDOA) freeing rdom 4 NOTE: detached from domain 4 NOTE: cache dismounted group 4/0x79F88C12 (REDOA) kfdp_dismount(): 19 kfdp_dismountBg(): 19 NOTE: De-assigning number (4,0) from disk (ORCL:REDOA0) SUCCESS: diskgroup REDOA was dismounted NOTE: cache dismounting group 5/0x48588C14 (REDOB) freeing rdom 5 NOTE: detached from domain 5 NOTE: cache dismounted group 5/0x48588C14 (REDOB) kfdp_dismount(): 20 kfdp_dismountBg(): 20 NOTE: De-assigning number (5,0) from disk (ORCL:REDOB0) SUCCESS: diskgroup REDOB was dismounted SUCCESS: ALTER DISKGROUP ALL DISMOUNT Wed Sep 19 20:42:09 2012 Stopping background process VKTM: Wed Sep 19 20:42:11 2012 freeing rdom 0 Wed Sep 19 20:42:12 2012 Instance shutdown complete Wed Sep 19 20:50:42 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 2 asm_diskstring = "ORCL:*" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" asm_diskgroups = "ORAFILES" asm_diskgroups = "REDOA" asm_diskgroups = "REDOB" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.187 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Wed Sep 19 20:50:43 2012 PMON started with pid=2, OS id=28156 Wed Sep 19 20:50:43 2012 VKTM started with pid=3, OS id=28158 at elevated priority VKTM running at (20)ms precision Wed Sep 19 20:50:43 2012 DIAG started with pid=4, OS id=28162 Wed Sep 19 20:50:43 2012 PING started with pid=5, OS id=28164 Wed Sep 19 20:50:43 2012 PSP0 started with pid=6, OS id=28166 Wed Sep 19 20:50:43 2012 DIA0 started with pid=7, OS id=28168 Wed Sep 19 20:50:43 2012 LMON started with pid=8, OS id=28170 Wed Sep 19 20:50:43 2012 LMD0 started with pid=9, OS id=28172 Wed Sep 19 20:50:43 2012 LMS0 started with pid=10, OS id=28175 at elevated priority Wed Sep 19 20:50:43 2012 MMAN started with pid=11, OS id=28179 Wed Sep 19 20:50:43 2012 DBW0 started with pid=12, OS id=28181 Wed Sep 19 20:50:43 2012 LGWR started with pid=13, OS id=28183 Wed Sep 19 20:50:43 2012 CKPT started with pid=14, OS id=28185 Wed Sep 19 20:50:43 2012 SMON started with pid=15, OS id=28188 Wed Sep 19 20:50:43 2012 RBAL started with pid=16, OS id=28190 Wed Sep 19 20:50:43 2012 GMON started with pid=17, OS id=28192 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 2) ASM instance List of nodes: 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Wed Sep 19 20:50:43 2012 LCK0 started with pid=18, OS id=28197 ORACLE_BASE from environment = /u01/app/oracle Wed Sep 19 20:50:44 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group FLASH number=1 incarn=0x496a63e1 NOTE: cache began mount (first) of group FLASH number=1 incarn=0x496a63e1 NOTE: cache registered group ORAFILES number=2 incarn=0x496a63e2 NOTE: cache began mount (first) of group ORAFILES number=2 incarn=0x496a63e2 NOTE: cache registered group PRD01 number=3 incarn=0x496a63e3 NOTE: cache began mount (first) of group PRD01 number=3 incarn=0x496a63e3 NOTE: cache registered group REDOA number=4 incarn=0x496a63e4 NOTE: cache began mount (first) of group REDOA number=4 incarn=0x496a63e4 NOTE: cache registered group REDOB number=5 incarn=0x497a63e5 NOTE: cache began mount (first) of group REDOB number=5 incarn=0x497a63e5 NOTE:Loaded lib: /opt/oracle/extapi/64/asm/orcl/1/libasm.so NOTE: Assigning number (1,0) to disk (ORCL:FLASH0) NOTE: Assigning number (2,0) to disk (ORCL:ORAFILES0) NOTE: Assigning number (2,1) to disk (ORCL:ORAFILES1) NOTE: Assigning number (2,2) to disk (ORCL:ORAFILES2) NOTE: Assigning number (3,0) to disk (ORCL:PRD010) NOTE: Assigning number (4,0) to disk (ORCL:REDOA0) NOTE: Assigning number (5,0) to disk (ORCL:REDOB0) NOTE: start heartbeating (grp 1) kfdp_query(FLASH): 7 kfdp_queryBg(): 7 NOTE: cache opening disk 0 of grp 1: FLASH_0000 label:FLASH0 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (first) group 1/0x496A63E1 (FLASH) * allocate domain 1, invalid = TRUE NOTE: attached to recovery domain 1 NOTE: cache recovered group 1 to fcn 0.88424362 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.88410913 ABA NOTE: seq=236 blk=2864 NOTE: cache mounting group 1/0x496A63E1 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=1 incarn=0x496a63e1 NOTE: start heartbeating (grp 2) kfdp_query(ORAFILES): 9 kfdp_queryBg(): 9 NOTE: cache opening disk 0 of grp 2: ORAFILES_0000 label:ORAFILES0 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 2: ORAFILES_0001 label:ORAFILES1 NOTE: cache opening disk 2 of grp 2: ORAFILES_0002 label:ORAFILES2 NOTE: cache mounting (first) group 2/0x496A63E2 (ORAFILES) * allocate domain 2, invalid = TRUE NOTE: attached to recovery domain 2 NOTE: cache recovered group 2 to fcn 0.38661442 NOTE: LGWR attempting to mount thread 1 for diskgroup 2 NOTE: LGWR mounted thread 1 for disk group 2 NOTE: opening chunk 1 at fcn 0.38661442 ABA NOTE: seq=104 blk=1726 NOTE: cache mounting group 2/0x496A63E2 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=2 incarn=0x496a63e2 NOTE: start heartbeating (grp 3) kfdp_query(PRD01): 11 kfdp_queryBg(): 11 NOTE: cache opening disk 0 of grp 3: PRD01_0000 label:PRD010 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (first) group 3/0x496A63E3 (PRD01) * allocate domain 3, invalid = TRUE NOTE: attached to recovery domain 3 NOTE: cache recovered group 3 to fcn 0.3646394 NOTE: LGWR attempting to mount thread 1 for diskgroup 3 NOTE: LGWR mounted thread 1 for disk group 3 NOTE: opening chunk 1 at fcn 0.3646362 ABA NOTE: seq=55 blk=7332 NOTE: cache mounting group 3/0x496A63E3 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0x496a63e3 NOTE: start heartbeating (grp 4) kfdp_query(REDOA): 13 kfdp_queryBg(): 13 NOTE: cache opening disk 0 of grp 4: REDOA0 label:REDOA0 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (first) group 4/0x496A63E4 (REDOA) * allocate domain 4, invalid = TRUE NOTE: attached to recovery domain 4 NOTE: cache recovered group 4 to fcn 0.562 NOTE: LGWR attempting to mount thread 1 for diskgroup 4 NOTE: LGWR mounted thread 1 for disk group 4 NOTE: opening chunk 1 at fcn 0.562 ABA NOTE: seq=3 blk=12 NOTE: cache mounting group 4/0x496A63E4 (REDOA) succeeded NOTE: cache ending mount (success) of group REDOA number=4 incarn=0x496a63e4 NOTE: start heartbeating (grp 5) kfdp_query(REDOB): 15 kfdp_queryBg(): 15 NOTE: cache opening disk 0 of grp 5: REDOB0 label:REDOB0 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (first) group 5/0x497A63E5 (REDOB) * allocate domain 5, invalid = TRUE NOTE: attached to recovery domain 5 NOTE: cache recovered group 5 to fcn 0.562 NOTE: LGWR attempting to mount thread 1 for diskgroup 5 NOTE: LGWR mounted thread 1 for disk group 5 NOTE: opening chunk 1 at fcn 0.562 ABA NOTE: seq=3 blk=13 NOTE: cache mounting group 5/0x497A63E5 (REDOB) succeeded NOTE: cache ending mount (success) of group REDOB number=5 incarn=0x497a63e5 kfdp_query(FLASH): 16 kfdp_queryBg(): 16 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup FLASH was mounted Wed Sep 19 20:50:53 2012 kfdp_query(ORAFILES): 17 Wed Sep 19 20:50:53 2012 kfdp_queryBg(): 17 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(PRD01): 18 kfdp_queryBg(): 18 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted kfdp_query(REDOA): 19 kfdp_queryBg(): 19 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 4 SUCCESS: diskgroup REDOA was mounted kfdp_query(REDOB): 20 kfdp_queryBg(): 20 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 5 SUCCESS: diskgroup REDOB was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Wed Sep 19 20:51:08 2012 Reconfiguration started (old inc 2, new inc 4) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Wed Sep 19 20:51:08 2012 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 * domain 2 valid = 1 according to instance 0 * domain 3 valid = 1 according to instance 0 * domain 4 valid = 1 according to instance 0 * domain 5 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Sep 19 20:51:08 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 45 GCS shadows traversed, 45 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Sep 19 20:51:32 2012 Starting background process ASMB Wed Sep 19 20:51:32 2012 ASMB started with pid=21, OS id=28823 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed Sep 19 20:52:09 2012 Starting background process ASMB Wed Sep 19 20:52:09 2012 ASMB started with pid=26, OS id=29595 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Nov 24 13:34:01 2012 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_28162.trc Sat Nov 24 13:34:35 2012 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_28162.trc Sat Nov 24 14:06:36 2012 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_28162.trc Sat Nov 24 14:06:38 2012 Reconfiguration started (old inc 4, new inc 6) List of nodes: 1 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE * dead instance detected - domain 4 invalid = TRUE * dead instance detected - domain 5 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Nov 24 14:06:38 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5458 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sat Nov 24 14:06:38 2012 NOTE: waiting for instance recovery of group 1 Sat Nov 24 14:06:38 2012 NOTE: waiting for instance recovery of group 1 Sat Nov 24 14:06:38 2012 kfdp_query(FLASH): 21 Sat Nov 24 14:06:39 2012 kfdp_queryBg(): 21 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=140.1765 group=1 NOTE: advancing ckpt for thread=2 ckpt=140.1765 NOTE: smon did instance recovery for domain 1 kfdp_query(ORAFILES): 22 kfdp_queryBg(): 22 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=2 ckpt=207.722 group=2 NOTE: advancing ckpt for thread=2 ckpt=207.722 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 23 Sat Nov 24 14:06:40 2012 NOTE: waiting for instance recovery of group 4 kfdp_queryBg(): 23 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=26.7941 group=3 NOTE: advancing ckpt for thread=2 ckpt=26.7941 NOTE: smon did instance recovery for domain 3 kfdp_query(REDOA): 24 kfdp_queryBg(): 24 NOTE: SMON starting instance recovery for group 4 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 Sat Nov 24 14:06:41 2012 NOTE: waiting for instance recovery of group 4 Sat Nov 24 14:06:41 2012 NOTE: waiting for instance recovery of group 4 Sat Nov 24 14:06:41 2012 NOTE: waiting for instance recovery of group 4 NOTE: starting recovery of thread=2 ckpt=3.4854 group=4 NOTE: advancing ckpt for thread=2 ckpt=3.4854 NOTE: smon did instance recovery for domain 4 kfdp_query(REDOB): 25 NOTE: waiting for instance recovery of group 5 NOTE: waiting for instance recovery of group 5 NOTE: waiting for instance recovery of group 5 kfdp_queryBg(): 25 NOTE: SMON starting instance recovery for group 5 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=3.4878 group=5 NOTE: advancing ckpt for thread=2 ckpt=3.4878 NOTE: smon did instance recovery for domain 5 Sat Nov 24 14:06:48 2012 Reconfiguration started (old inc 6, new inc 8) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Sat Nov 24 14:06:49 2012 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 * domain 2 valid = 1 according to instance 0 * domain 3 valid = 1 according to instance 0 * domain 4 valid = 1 according to instance 0 * domain 5 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Nov 24 14:06:49 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5449 GCS shadows traversed, 2793 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sun Jan 27 02:31:28 2013 Trace dumping is performing id=[cdmp_20130127023128] Mon Mar 04 16:02:54 2013 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_28162.trc Mon Mar 04 16:15:51 2013 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_28162.trc Sun Apr 21 04:33:12 2013 Trace dumping is performing id=[cdmp_20130421043311] Sun Apr 21 04:41:02 2013 Trace dumping is performing id=[cdmp_20130421044102] Sun Apr 21 04:41:18 2013 Trace dumping is performing id=[cdmp_20130421044118] Sun Apr 21 04:41:20 2013 Reconfiguration started (old inc 8, new inc 10) List of nodes: 1 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE * dead instance detected - domain 4 invalid = TRUE * dead instance detected - domain 5 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sun Apr 21 04:41:20 2013 LMS 0: 1 GCS shadows cancelled, 1 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5458 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sun Apr 21 04:41:21 2013 NOTE: waiting for instance recovery of group 1 Sun Apr 21 04:41:21 2013 kfdp_query(FLASH): 26 Sun Apr 21 04:41:21 2013 NOTE: waiting for instance recovery of group 1 Sun Apr 21 04:41:21 2013 NOTE: waiting for instance recovery of group 1 Sun Apr 21 04:41:21 2013 NOTE: waiting for instance recovery of group 1 Sun Apr 21 04:41:21 2013 NOTE: waiting for instance recovery of group 1 Sun Apr 21 04:41:21 2013 NOTE: waiting for instance recovery of group 1 Sun Apr 21 04:41:22 2013 kfdp_queryBg(): 26 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=196.3666 group=1 NOTE: advancing ckpt for thread=2 ckpt=196.3666 NOTE: smon did instance recovery for domain 1 kfdp_query(ORAFILES): 27 kfdp_queryBg(): 27 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=2 ckpt=208.5093 group=2 NOTE: advancing ckpt for thread=2 ckpt=208.5093 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 28 kfdp_queryBg(): 28 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=31.7103 group=3 NOTE: advancing ckpt for thread=2 ckpt=31.7103 NOTE: smon did instance recovery for domain 3 kfdp_query(REDOA): 29 kfdp_queryBg(): 29 NOTE: SMON starting instance recovery for group 4 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=5.2966 group=4 NOTE: advancing ckpt for thread=2 ckpt=5.2966 NOTE: smon did instance recovery for domain 4 kfdp_query(REDOB): 30 kfdp_queryBg(): 30 NOTE: SMON starting instance recovery for group 5 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=5.2990 group=5 NOTE: advancing ckpt for thread=2 ckpt=5.2990 NOTE: smon did instance recovery for domain 5 Sun Apr 21 04:57:01 2013 Reconfiguration started (old inc 10, new inc 12) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Sun Apr 21 04:57:02 2013 * domain 0 valid = 1 according to instance 0 * domain 1 valid = 1 according to instance 0 * domain 2 valid = 1 according to instance 0 * domain 3 valid = 1 according to instance 0 * domain 4 valid = 1 according to instance 0 * domain 5 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sun Apr 21 04:57:02 2013 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5458 GCS shadows traversed, 2755 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete