Fri Apr 15 17:01:00 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Shared memory segment for instance monitoring created Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side pfile /u01/app/oracle/product/11.1.0/db_1/dbs/init+ASM1.ora System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Fri Apr 15 17:01:01 2011 PMON started with pid=2, OS id=31900 Fri Apr 15 17:01:01 2011 VKTM started with pid=3, OS id=31902 at elevated priority VKTM running at (20)ms precision Fri Apr 15 17:01:01 2011 DIAG started with pid=4, OS id=31906 Fri Apr 15 17:01:01 2011 PING started with pid=5, OS id=31908 Fri Apr 15 17:01:01 2011 PSP0 started with pid=6, OS id=31910 Fri Apr 15 17:01:01 2011 DIA0 started with pid=7, OS id=31912 Fri Apr 15 17:01:01 2011 LMON started with pid=8, OS id=31914 Fri Apr 15 17:01:01 2011 LMD0 started with pid=9, OS id=31916 Fri Apr 15 17:01:01 2011 LMS0 started with pid=10, OS id=31918 at elevated priority Fri Apr 15 17:01:01 2011 MMAN started with pid=11, OS id=31922 Fri Apr 15 17:01:01 2011 DBW0 started with pid=12, OS id=31924 Fri Apr 15 17:01:01 2011 LGWR started with pid=13, OS id=31926 Fri Apr 15 17:01:01 2011 CKPT started with pid=14, OS id=31928 Fri Apr 15 17:01:01 2011 SMON started with pid=15, OS id=31930 Fri Apr 15 17:01:01 2011 RBAL started with pid=16, OS id=31932 Fri Apr 15 17:01:01 2011 GMON started with pid=17, OS id=31934 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 2) ASM instance List of nodes: 0 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Fri Apr 15 17:01:01 2011 LCK0 started with pid=18, OS id=31945 ORACLE_BASE from environment = /u01/app/oracle Spfile /dev/mpath/mpath0p3 is in old pre-11 format and compatible >= 11.0.0; converting to new H.A.R.D. compliant format. Fri Apr 15 17:01:02 2011 SQL> ALTER DISKGROUP ALL MOUNT ALTER SYSTEM SET asm_diskgroups='' SCOPE=BOTH SID='+ASM1'; ALTER SYSTEM SET asm_diskgroups='' SCOPE=BOTH SID='+ASM2'; Fri Apr 15 17:01:05 2011 SQL> alter diskgroup all mount Fri Apr 15 17:01:34 2011 Reconfiguration started (old inc 2, new inc 4) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Apr 15 17:01:34 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Fri Apr 15 17:01:38 2011 WARNING::ASMLIB library not found. See trace file for details. Fri Apr 15 17:02:57 2011 ALTER SYSTEM SET asm_diskstring='/dev/mpath/mpath*' SCOPE=BOTH SID='+ASM1'; ALTER SYSTEM SET asm_diskstring='/dev/mpath/mpath*' SCOPE=BOTH SID='+ASM2'; Fri Apr 15 17:03:23 2011 SQL> CREATE DISKGROUP ORAFILES External REDUNDANCY DISK '/dev/mpath/mpath1p1' SIZE 307196M NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: initializing header on grp 1 disk ORAFILES_0000 NOTE: initiating PST update: grp = 1 kfdp_update(): 1 Fri Apr 15 17:03:23 2011 kfdp_updateBg(): 1 NOTE: group ORAFILES: initial PST location: disk 0000 (PST copy 0) NOTE: PST update grp = 1 completed successfully NOTE: cache registered group ORAFILES number=1 incarn=0x51f64d86 NOTE: cache began mount (first) of group ORAFILES number=1 incarn=0x51f64d86 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 Fri Apr 15 17:03:23 2011 * allocate domain 1, invalid = TRUE kjbdomatt send to node 1 Fri Apr 15 17:03:23 2011 NOTE: attached to recovery domain 1 NOTE: cache creating group 1/0x51F64D86 (ORAFILES) NOTE: cache mounting group 1/0x51F64D86 (ORAFILES) succeeded NOTE: allocating F1X0 on grp 1 disk ORAFILES_0000 NOTE: diskgroup must now be re-mounted prior to first use NOTE: cache dismounting group 1/0x51F64D86 (ORAFILES) NOTE: lgwr not being msg'd to dismount kjbdomdet send to node 1 detach from dom 1, sending detach message to node 1 freeing rdom 1 NOTE: detached from domain 1 NOTE: cache dismounted group 1/0x51F64D86 (ORAFILES) NOTE: cache ending mount (fail) of group ORAFILES number=1 incarn=0x51f64d86 kfdp_dismount(): 2 kfdp_dismountBg(): 2 kfdp_dismount(): 3 kfdp_dismountBg(): 3 NOTE: De-assigning number (1,0) from disk (/dev/mpath/mpath1p1) SUCCESS: diskgroup ORAFILES was created NOTE: cache registered group ORAFILES number=1 incarn=0x60b64d88 NOTE: cache began mount (first) of group ORAFILES number=1 incarn=0x60b64d88 NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: start heartbeating (grp 1) kfdp_query(): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (first) group 1/0x60B64D88 (ORAFILES) * allocate domain 1, invalid = TRUE kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: cache recovered group 1 to fcn 0.0 Fri Apr 15 17:03:30 2011 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.0 ABA NOTE: seq=2 blk=0 NOTE: cache mounting group 1/0x60B64D88 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x60b64d88 Fri Apr 15 17:03:30 2011 kfdp_query(): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: CREATE DISKGROUP ORAFILES External REDUNDANCY DISK '/dev/mpath/mpath1p1' SIZE 307196M ALTER SYSTEM SET asm_diskgroups='ORAFILES' SCOPE=BOTH SID='+ASM1'; ALTER SYSTEM SET asm_diskgroups='ORAFILES' SCOPE=BOTH SID='+ASM2'; NOTE: enlarging ACD for group 1/0x60b64d88 (ORAFILES) SUCCESS: ACD enlarged for group 1/0x60b64d88 (ORAFILES) Fri Apr 15 18:55:46 2011 Shutting down instance: further logons disabled Shutting down instance (immediate) License high water mark = 3 SQL> ALTER DISKGROUP ALL DISMOUNT NOTE: cache dismounting group 1/0x60B64D88 (ORAFILES) Fri Apr 15 18:55:46 2011 kjbdomdet send to node 1 detach from dom 1, sending detach message to node 1 Fri Apr 15 18:55:46 2011 NOTE: detached from domain 1 NOTE: cache dismounted group 1/0x60B64D88 (ORAFILES) kfdp_dismount(): 8 Fri Apr 15 18:55:46 2011 kfdp_dismountBg(): 8 NOTE: De-assigning number (1,0) from disk (/dev/mpath/mpath1p1) SUCCESS: diskgroup ORAFILES was dismounted SUCCESS: ALTER DISKGROUP ALL DISMOUNT Fri Apr 15 18:55:47 2011 Stopping background process VKTM: Fri Apr 15 18:55:49 2011 freeing rdom 1 freeing rdom 0 Fri Apr 15 18:55:50 2011 Instance shutdown complete Fri Apr 15 20:48:35 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Fri Apr 15 20:48:36 2011 PMON started with pid=2, OS id=29427 Fri Apr 15 20:48:36 2011 VKTM started with pid=3, OS id=29429 at elevated priority VKTM running at (20)ms precision Fri Apr 15 20:48:36 2011 DIAG started with pid=4, OS id=29433 Fri Apr 15 20:48:36 2011 PING started with pid=5, OS id=29435 Fri Apr 15 20:48:36 2011 PSP0 started with pid=6, OS id=29437 Fri Apr 15 20:48:36 2011 DIA0 started with pid=7, OS id=29439 Fri Apr 15 20:48:36 2011 LMON started with pid=8, OS id=29441 Fri Apr 15 20:48:36 2011 LMD0 started with pid=9, OS id=29443 Fri Apr 15 20:48:36 2011 LMS0 started with pid=10, OS id=29445 at elevated priority Fri Apr 15 20:48:36 2011 MMAN started with pid=11, OS id=29449 Fri Apr 15 20:48:36 2011 DBW0 started with pid=12, OS id=29451 Fri Apr 15 20:48:36 2011 LGWR started with pid=13, OS id=29453 Fri Apr 15 20:48:36 2011 CKPT started with pid=14, OS id=29455 Fri Apr 15 20:48:36 2011 SMON started with pid=15, OS id=29457 Fri Apr 15 20:48:36 2011 RBAL started with pid=16, OS id=29459 Fri Apr 15 20:48:36 2011 GMON started with pid=17, OS id=29461 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 8) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Fri Apr 15 20:48:36 2011 LCK0 started with pid=18, OS id=29530 ORACLE_BASE from environment = /u01/app/oracle Fri Apr 15 20:48:37 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x60bb22b1 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x60bb22b1 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x60BB22B1 (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.562 ABA NOTE: seq=3 blk=12 NOTE: cache mounting group 1/0x60BB22B1 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x60bb22b1 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Fri Apr 15 20:49:06 2011 Shutting down instance: further logons disabled Shutting down instance (immediate) License high water mark = 1 SQL> ALTER DISKGROUP ALL DISMOUNT NOTE: cache dismounting group 1/0x60BB22B1 (ORAFILES) Fri Apr 15 20:49:06 2011 kjbdomdet send to node 1 detach from dom 1, sending detach message to node 1 Fri Apr 15 20:49:07 2011 NOTE: detached from domain 1 NOTE: cache dismounted group 1/0x60BB22B1 (ORAFILES) kfdp_dismount(): 4 Fri Apr 15 20:49:07 2011 kfdp_dismountBg(): 4 NOTE: De-assigning number (1,0) from disk (/dev/mpath/mpath1p1) SUCCESS: diskgroup ORAFILES was dismounted SUCCESS: ALTER DISKGROUP ALL DISMOUNT Fri Apr 15 20:49:08 2011 Stopping background process VKTM: Fri Apr 15 20:49:09 2011 freeing rdom 1 freeing rdom 0 Fri Apr 15 20:49:11 2011 Instance shutdown complete Fri Apr 15 20:56:51 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Fri Apr 15 20:56:51 2011 PMON started with pid=2, OS id=5419 Fri Apr 15 20:56:51 2011 VKTM started with pid=3, OS id=5421 at elevated priority Fri Apr 15 20:56:51 2011 DIAG started with pid=4, OS id=5425 Fri Apr 15 20:56:52 2011 PING started with pid=5, OS id=5429 Fri Apr 15 20:56:52 2011 PSP0 started with pid=6, OS id=5435 Fri Apr 15 20:56:52 2011 DIA0 started with pid=7, OS id=5437 Fri Apr 15 20:56:52 2011 Fri Apr 15 20:56:52 2011 LMON started with pid=8, OS id=5439 LMD0 started with pid=9, OS id=5441 VKTM running at (20)ms precision Fri Apr 15 20:56:52 2011 LMS0 started with pid=10, OS id=5443 at elevated priority Fri Apr 15 20:56:52 2011 MMAN started with pid=11, OS id=5447 Fri Apr 15 20:56:52 2011 DBW0 started with pid=12, OS id=5449 Fri Apr 15 20:56:52 2011 LGWR started with pid=13, OS id=5451 Fri Apr 15 20:56:52 2011 CKPT started with pid=14, OS id=5453 Fri Apr 15 20:56:52 2011 SMON started with pid=15, OS id=5455 Fri Apr 15 20:56:52 2011 RBAL started with pid=16, OS id=5457 Fri Apr 15 20:56:52 2011 GMON started with pid=17, OS id=5459 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 12) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Fri Apr 15 20:56:52 2011 LCK0 started with pid=18, OS id=5516 ORACLE_BASE from environment = /u01/app/oracle Fri Apr 15 20:56:53 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x60bbe45c NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x60bbe45c WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x60BBE45C (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.562 ABA NOTE: seq=4 blk=12 NOTE: cache mounting group 1/0x60BBE45C (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x60bbe45c kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Fri Apr 15 20:57:55 2011 Received detach msg from node 1 for dom 1 Fri Apr 15 20:57:55 2011 NOTE: PST enabling heartbeating (grp 1) Fri Apr 15 20:57:57 2011 Reconfiguration started (old inc 12, new inc 14) List of nodes: 0 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Apr 15 20:57:57 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 9 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Fri Apr 15 21:05:03 2011 Reconfiguration started (old inc 14, new inc 16) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Apr 15 21:05:03 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 9 GCS shadows traversed, 9 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Fri Apr 15 21:06:24 2011 Shutting down instance: further logons disabled Shutting down instance (immediate) License high water mark = 1 SQL> ALTER DISKGROUP ALL DISMOUNT NOTE: cache dismounting group 1/0x60BBE45C (ORAFILES) Fri Apr 15 21:06:24 2011 kjbdomdet send to node 1 detach from dom 1, sending detach message to node 1 Fri Apr 15 21:06:24 2011 NOTE: detached from domain 1 NOTE: cache dismounted group 1/0x60BBE45C (ORAFILES) kfdp_dismount(): 4 Fri Apr 15 21:06:24 2011 kfdp_dismountBg(): 4 NOTE: De-assigning number (1,0) from disk (/dev/mpath/mpath1p1) SUCCESS: diskgroup ORAFILES was dismounted SUCCESS: ALTER DISKGROUP ALL DISMOUNT Fri Apr 15 21:06:25 2011 Stopping background process VKTM: Fri Apr 15 21:06:27 2011 freeing rdom 1 freeing rdom 0 Fri Apr 15 21:06:28 2011 Instance shutdown complete Fri Apr 15 21:20:11 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Fri Apr 15 21:20:12 2011 PMON started with pid=2, OS id=25180 Fri Apr 15 21:20:12 2011 VKTM started with pid=3, OS id=25182 at elevated priority VKTM running at (20)ms precision Fri Apr 15 21:20:12 2011 DIAG started with pid=4, OS id=25186 Fri Apr 15 21:20:12 2011 PING started with pid=5, OS id=25188 Fri Apr 15 21:20:12 2011 PSP0 started with pid=6, OS id=25190 Fri Apr 15 21:20:12 2011 DIA0 started with pid=7, OS id=25192 Fri Apr 15 21:20:12 2011 LMON started with pid=8, OS id=25194 Fri Apr 15 21:20:12 2011 LMD0 started with pid=9, OS id=25196 Fri Apr 15 21:20:12 2011 LMS0 started with pid=10, OS id=25198 at elevated priority Fri Apr 15 21:20:12 2011 MMAN started with pid=11, OS id=25202 Fri Apr 15 21:20:12 2011 DBW0 started with pid=12, OS id=25204 Fri Apr 15 21:20:12 2011 LGWR started with pid=13, OS id=25206 Fri Apr 15 21:20:12 2011 CKPT started with pid=14, OS id=25208 Fri Apr 15 21:20:12 2011 SMON started with pid=15, OS id=25210 Fri Apr 15 21:20:12 2011 RBAL started with pid=16, OS id=25212 Fri Apr 15 21:20:12 2011 GMON started with pid=17, OS id=25214 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 20) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Fri Apr 15 21:20:13 2011 LCK0 started with pid=18, OS id=25225 ORACLE_BASE from environment = /u01/app/oracle Fri Apr 15 21:20:13 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x60be0759 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x60be0759 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x60BE0759 (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.562 ABA NOTE: seq=5 blk=12 NOTE: cache mounting group 1/0x60BE0759 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x60be0759 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Fri Apr 15 21:20:21 2011 Shutting down instance (abort) License high water mark = 1 USER (ospid: 25386): terminating the instance Instance terminated by USER, pid = 25386 Fri Apr 15 21:20:22 2011 Instance shutdown complete Fri Apr 15 21:22:35 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Fri Apr 15 21:22:36 2011 PMON started with pid=2, OS id=26146 Fri Apr 15 21:22:36 2011 VKTM started with pid=3, OS id=26148 at elevated priority VKTM running at (20)ms precision Fri Apr 15 21:22:36 2011 DIAG started with pid=4, OS id=26152 Fri Apr 15 21:22:36 2011 PING started with pid=5, OS id=26154 Fri Apr 15 21:22:36 2011 PSP0 started with pid=6, OS id=26156 Fri Apr 15 21:22:36 2011 DIA0 started with pid=7, OS id=26158 Fri Apr 15 21:22:36 2011 LMON started with pid=8, OS id=26160 Fri Apr 15 21:22:36 2011 LMD0 started with pid=9, OS id=26162 Fri Apr 15 21:22:36 2011 LMS0 started with pid=10, OS id=26164 at elevated priority Fri Apr 15 21:22:36 2011 MMAN started with pid=11, OS id=26168 Fri Apr 15 21:22:36 2011 DBW0 started with pid=12, OS id=26170 Fri Apr 15 21:22:36 2011 LGWR started with pid=13, OS id=26172 Fri Apr 15 21:22:36 2011 CKPT started with pid=14, OS id=26174 Fri Apr 15 21:22:36 2011 SMON started with pid=15, OS id=26176 Fri Apr 15 21:22:36 2011 RBAL started with pid=16, OS id=26178 Fri Apr 15 21:22:36 2011 GMON started with pid=17, OS id=26180 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 24) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Fri Apr 15 21:22:37 2011 LCK0 started with pid=18, OS id=26191 ORACLE_BASE from environment = /u01/app/oracle Fri Apr 15 21:22:37 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x60be3f8d NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x60be3f8d WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x60BE3F8D (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.562 ABA NOTE: seq=6 blk=13 NOTE: cache mounting group 1/0x60BE3F8D (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x60be3f8d kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Tue Apr 26 10:48:58 2011 Shutting down instance (abort) License high water mark = 1 USER (ospid: 29568): terminating the instance Instance terminated by USER, pid = 29568 Tue Apr 26 10:48:59 2011 Instance shutdown complete Tue Apr 26 11:00:22 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Tue Apr 26 11:00:23 2011 PMON started with pid=2, OS id=10784 Tue Apr 26 11:00:23 2011 VKTM started with pid=3, OS id=10786 at elevated priority VKTM running at (20)ms precision Tue Apr 26 11:00:23 2011 DIAG started with pid=4, OS id=10790 Tue Apr 26 11:00:23 2011 PING started with pid=5, OS id=10792 Tue Apr 26 11:00:23 2011 PSP0 started with pid=6, OS id=10794 Tue Apr 26 11:00:23 2011 DIA0 started with pid=7, OS id=10796 Tue Apr 26 11:00:23 2011 LMON started with pid=8, OS id=10798 Tue Apr 26 11:00:23 2011 LMD0 started with pid=9, OS id=10800 Tue Apr 26 11:00:23 2011 LMS0 started with pid=10, OS id=10802 at elevated priority Tue Apr 26 11:00:23 2011 MMAN started with pid=11, OS id=10806 Tue Apr 26 11:00:24 2011 DBW0 started with pid=12, OS id=10808 Tue Apr 26 11:00:24 2011 LGWR started with pid=13, OS id=10810 Tue Apr 26 11:00:24 2011 CKPT started with pid=14, OS id=10812 Tue Apr 26 11:00:24 2011 SMON started with pid=15, OS id=10814 Tue Apr 26 11:00:24 2011 RBAL started with pid=16, OS id=10816 Tue Apr 26 11:00:24 2011 GMON started with pid=17, OS id=10818 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 28) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Apr 26 11:00:24 2011 LCK0 started with pid=18, OS id=10832 ORACLE_BASE from environment = /u01/app/oracle Tue Apr 26 11:00:25 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x60b8cf3a NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x60b8cf3a WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x60B8CF3A (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.562 ABA NOTE: seq=7 blk=14 NOTE: cache mounting group 1/0x60B8CF3A (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x60b8cf3a kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Tue Apr 26 11:08:59 2011 Reconfiguration started (old inc 28, new inc 30) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Apr 26 11:08:59 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 9 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Tue Apr 26 11:08:59 2011 kfdp_query(): 4 Tue Apr 26 11:08:59 2011 NOTE: PST enabling heartbeating (grp 1) kfdp_queryBg(): 4 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=3.0 group=1 NOTE: advancing ckpt for thread=2 ckpt=3.0 NOTE: smon did instance recovery for domain 1 Tue Apr 26 11:20:06 2011 Reconfiguration started (old inc 30, new inc 32) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Apr 26 11:20:06 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 9 GCS shadows traversed, 9 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Apr 26 11:40:26 2011 Reconfiguration started (old inc 32, new inc 34) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Apr 26 11:40:26 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 9 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Tue Apr 26 11:40:27 2011 kfdp_query(): 5 Tue Apr 26 11:40:27 2011 kfdp_queryBg(): 5 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=4.1 group=1 NOTE: advancing ckpt for thread=2 ckpt=4.1 NOTE: smon did instance recovery for domain 1 Tue Apr 26 21:18:55 2011 Shutting down instance (abort) License high water mark = 1 USER (ospid: 433): terminating the instance Instance terminated by USER, pid = 433 Tue Apr 26 21:18:56 2011 Instance shutdown complete Tue Apr 26 21:42:45 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Tue Apr 26 21:42:47 2011 PMON started with pid=2, OS id=8037 Tue Apr 26 21:42:47 2011 VKTM started with pid=3, OS id=8039 at elevated priority VKTM running at (20)ms precision Tue Apr 26 21:42:47 2011 DIAG started with pid=4, OS id=8043 Tue Apr 26 21:42:47 2011 PING started with pid=5, OS id=8045 Tue Apr 26 21:42:47 2011 PSP0 started with pid=6, OS id=8047 Tue Apr 26 21:42:47 2011 DIA0 started with pid=7, OS id=8049 Tue Apr 26 21:42:47 2011 LMON started with pid=8, OS id=8051 Tue Apr 26 21:42:47 2011 LMD0 started with pid=9, OS id=8053 Tue Apr 26 21:42:47 2011 LMS0 started with pid=10, OS id=8055 at elevated priority Tue Apr 26 21:42:47 2011 MMAN started with pid=11, OS id=8059 Tue Apr 26 21:42:47 2011 DBW0 started with pid=12, OS id=8061 Tue Apr 26 21:42:47 2011 LGWR started with pid=13, OS id=8063 Tue Apr 26 21:42:47 2011 CKPT started with pid=14, OS id=8065 Tue Apr 26 21:42:47 2011 SMON started with pid=15, OS id=8067 Tue Apr 26 21:42:47 2011 RBAL started with pid=16, OS id=8069 Tue Apr 26 21:42:47 2011 GMON started with pid=17, OS id=8071 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 2) ASM instance List of nodes: 0 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Tue Apr 26 21:42:48 2011 LCK0 started with pid=18, OS id=8082 ORACLE_BASE from environment = /u01/app/oracle Tue Apr 26 21:42:49 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x9ed88922 NOTE: cache began mount (first) of group ORAFILES number=1 incarn=0x9ed88922 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: start heartbeating (grp 1) kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (first) group 1/0x9ED88922 (ORAFILES) * allocate domain 1, invalid = TRUE NOTE: attached to recovery domain 1 NOTE: starting recovery of thread=1 ckpt=7.14 group=1 NOTE: advancing ckpt for thread=1 ckpt=7.14 NOTE: cache recovered group 1 to fcn 0.562 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.562 ABA NOTE: seq=8 blk=15 NOTE: cache mounting group 1/0x9ED88922 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x9ed88922 kfdp_query(): 4 kfdp_queryBg(): 4 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Tue Apr 26 21:49:38 2011 Reconfiguration started (old inc 2, new inc 4) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Apr 26 21:49:38 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 9 GCS shadows traversed, 9 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Apr 26 21:52:27 2011 Reconfiguration started (old inc 4, new inc 6) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Apr 26 21:52:27 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 9 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Tue Apr 26 21:52:27 2011 kfdp_query(): 5 Tue Apr 26 21:52:27 2011 kfdp_queryBg(): 5 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=5.2 group=1 NOTE: advancing ckpt for thread=2 ckpt=5.2 NOTE: smon did instance recovery for domain 1 Tue Apr 26 21:55:06 2011 Reconfiguration started (old inc 6, new inc 8) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Apr 26 21:55:06 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 9 GCS shadows traversed, 9 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Apr 26 22:10:22 2011 Reconfiguration started (old inc 8, new inc 10) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Apr 26 22:10:22 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 9 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Tue Apr 26 22:10:22 2011 kfdp_query(): 6 Tue Apr 26 22:10:22 2011 kfdp_queryBg(): 6 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=6.3 group=1 NOTE: advancing ckpt for thread=2 ckpt=6.3 NOTE: smon did instance recovery for domain 1 Tue Apr 26 22:28:24 2011 Reconfiguration started (old inc 10, new inc 12) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Apr 26 22:28:24 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 9 GCS shadows traversed, 9 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Apr 26 22:38:09 2011 Reconfiguration started (old inc 12, new inc 14) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Apr 26 22:38:09 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 9 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Tue Apr 26 22:38:09 2011 kfdp_query(): 7 Tue Apr 26 22:38:09 2011 kfdp_queryBg(): 7 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=7.4 group=1 NOTE: advancing ckpt for thread=2 ckpt=7.4 NOTE: smon did instance recovery for domain 1 Tue Apr 26 22:43:43 2011 Reconfiguration started (old inc 14, new inc 16) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Apr 26 22:43:43 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 9 GCS shadows traversed, 9 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Apr 26 22:47:53 2011 Shutting down instance (abort) License high water mark = 1 USER (ospid: 30292): terminating the instance Instance terminated by USER, pid = 30292 Tue Apr 26 22:47:54 2011 Instance shutdown complete Tue Apr 26 22:49:24 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Tue Apr 26 22:49:24 2011 PMON started with pid=2, OS id=2593 Tue Apr 26 22:49:25 2011 VKTM started with pid=3, OS id=2595 at elevated priority VKTM running at (20)ms precision Tue Apr 26 22:49:25 2011 DIAG started with pid=4, OS id=2605 Tue Apr 26 22:49:25 2011 PING started with pid=5, OS id=2607 Tue Apr 26 22:49:25 2011 PSP0 started with pid=6, OS id=2609 Tue Apr 26 22:49:25 2011 DIA0 started with pid=7, OS id=2611 Tue Apr 26 22:49:25 2011 LMON started with pid=8, OS id=2613 Tue Apr 26 22:49:25 2011 LMD0 started with pid=9, OS id=2615 Tue Apr 26 22:49:25 2011 LMS0 started with pid=10, OS id=2617 at elevated priority Tue Apr 26 22:49:25 2011 MMAN started with pid=11, OS id=2621 Tue Apr 26 22:49:25 2011 DBW0 started with pid=12, OS id=2623 Tue Apr 26 22:49:25 2011 LGWR started with pid=13, OS id=2625 Tue Apr 26 22:49:25 2011 CKPT started with pid=14, OS id=2627 Tue Apr 26 22:49:25 2011 SMON started with pid=15, OS id=2629 Tue Apr 26 22:49:25 2011 RBAL started with pid=16, OS id=2631 Tue Apr 26 22:49:25 2011 GMON started with pid=17, OS id=2633 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 20) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Apr 26 22:49:25 2011 LCK0 started with pid=18, OS id=2669 ORACLE_BASE from environment = /u01/app/oracle Tue Apr 26 22:49:26 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x9edea28a NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x9edea28a WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x9EDEA28A (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.566 ABA NOTE: seq=9 blk=17 NOTE: cache mounting group 1/0x9EDEA28A (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x9edea28a kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Tue Apr 26 22:49:49 2011 Shutting down instance (abort) License high water mark = 1 USER (ospid: 3393): terminating the instance Instance terminated by USER, pid = 3393 Tue Apr 26 22:49:50 2011 Instance shutdown complete Tue Apr 26 22:52:47 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Tue Apr 26 22:52:47 2011 PMON started with pid=2, OS id=11072 Tue Apr 26 22:52:47 2011 VKTM started with pid=3, OS id=11074 at elevated priority VKTM running at (20)ms precision Tue Apr 26 22:52:47 2011 DIAG started with pid=4, OS id=11078 Tue Apr 26 22:52:47 2011 PING started with pid=5, OS id=11080 Tue Apr 26 22:52:47 2011 PSP0 started with pid=6, OS id=11082 Tue Apr 26 22:52:47 2011 DIA0 started with pid=7, OS id=11084 Tue Apr 26 22:52:47 2011 LMON started with pid=8, OS id=11086 Tue Apr 26 22:52:47 2011 LMD0 started with pid=9, OS id=11088 Tue Apr 26 22:52:47 2011 LMS0 started with pid=10, OS id=11090 at elevated priority Tue Apr 26 22:52:47 2011 MMAN started with pid=11, OS id=11094 Tue Apr 26 22:52:47 2011 DBW0 started with pid=12, OS id=11096 Tue Apr 26 22:52:47 2011 LGWR started with pid=13, OS id=11098 Tue Apr 26 22:52:48 2011 CKPT started with pid=14, OS id=11100 Tue Apr 26 22:52:48 2011 SMON started with pid=15, OS id=11102 Tue Apr 26 22:52:48 2011 RBAL started with pid=16, OS id=11104 Tue Apr 26 22:52:48 2011 GMON started with pid=17, OS id=11106 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 24) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Apr 26 22:52:48 2011 LCK0 started with pid=18, OS id=11160 ORACLE_BASE from environment = /u01/app/oracle Tue Apr 26 22:52:49 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x9edef1c1 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x9edef1c1 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x9EDEF1C1 (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.566 ABA NOTE: seq=10 blk=18 NOTE: cache mounting group 1/0x9EDEF1C1 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x9edef1c1 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Mon May 02 11:27:04 2011 Shutting down instance (abort) License high water mark = 1 USER (ospid: 16961): terminating the instance Instance terminated by USER, pid = 16961 Mon May 02 11:27:05 2011 Instance shutdown complete Mon May 02 11:32:48 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Mon May 02 11:32:50 2011 PMON started with pid=2, OS id=7749 Mon May 02 11:32:50 2011 VKTM started with pid=3, OS id=7751 at elevated priority VKTM running at (20)ms precision Mon May 02 11:32:50 2011 DIAG started with pid=4, OS id=7755 Mon May 02 11:32:50 2011 PING started with pid=5, OS id=7757 Mon May 02 11:32:50 2011 PSP0 started with pid=6, OS id=7759 Mon May 02 11:32:50 2011 DIA0 started with pid=7, OS id=7761 Mon May 02 11:32:50 2011 LMON started with pid=8, OS id=7763 Mon May 02 11:32:50 2011 LMD0 started with pid=9, OS id=7765 Mon May 02 11:32:50 2011 LMS0 started with pid=10, OS id=7767 at elevated priority Mon May 02 11:32:50 2011 MMAN started with pid=11, OS id=7771 Mon May 02 11:32:50 2011 DBW0 started with pid=12, OS id=7773 Mon May 02 11:32:50 2011 LGWR started with pid=13, OS id=7781 Mon May 02 11:32:50 2011 CKPT started with pid=14, OS id=7783 Mon May 02 11:32:50 2011 SMON started with pid=15, OS id=7785 Mon May 02 11:32:50 2011 RBAL started with pid=16, OS id=7787 Mon May 02 11:32:50 2011 GMON started with pid=17, OS id=7789 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 2) ASM instance List of nodes: 0 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Mon May 02 11:32:51 2011 LCK0 started with pid=18, OS id=7794 ORACLE_BASE from environment = /u01/app/oracle Mon May 02 11:32:52 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xe2f86d63 NOTE: cache began mount (first) of group ORAFILES number=1 incarn=0xe2f86d63 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: start heartbeating (grp 1) kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (first) group 1/0xE2F86D63 (ORAFILES) * allocate domain 1, invalid = TRUE NOTE: attached to recovery domain 1 NOTE: starting recovery of thread=2 ckpt=8.5 group=1 NOTE: advancing ckpt for thread=2 ckpt=8.5 NOTE: cache recovered group 1 to fcn 0.566 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.566 ABA NOTE: seq=11 blk=19 NOTE: cache mounting group 1/0xE2F86D63 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xe2f86d63 kfdp_query(): 4 kfdp_queryBg(): 4 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Mon May 02 11:42:57 2011 Reconfiguration started (old inc 2, new inc 4) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Mon May 02 11:42:57 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 9 GCS shadows traversed, 9 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Mon May 02 11:43:41 2011 SQL> alter diskgroup orafiles add disk '/dev/mpath/mpath2p2' ORA-15032: not all alterations performed ORA-15031: disk specification '/dev/mpath/mpath2p2' matches no disks ERROR: alter diskgroup orafiles add disk '/dev/mpath/mpath2p2' Mon May 02 11:43:55 2011 SQL> alter diskgroup orafiles add disk '/dev/mpath/mpath2p1' NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: requesting all-instance membership refresh for group=1 NOTE: initializing header on grp 1 disk ORAFILES_0001 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: requesting all-instance disk validation for group=1 Mon May 02 11:43:57 2011 NOTE: disk validation pending for group 1/0xe2f86d63 (ORAFILES) SUCCESS: validated disks for 1/0xe2f86d63 (ORAFILES) NOTE: initiating PST update: grp = 1 kfdp_update(): 5 Mon May 02 11:44:00 2011 kfdp_updateBg(): 5 NOTE: PST update grp = 1 completed successfully NOTE: membership refresh pending for group 1/0xe2f86d63 (ORAFILES) kfdp_query(): 6 kfdp_queryBg(): 6 kfdp_query(): 7 kfdp_queryBg(): 7 SUCCESS: refreshed membership for 1/0xe2f86d63 (ORAFILES) Mon May 02 11:44:06 2011 SUCCESS: alter diskgroup orafiles add disk '/dev/mpath/mpath2p1' NOTE: starting rebalance of group 1/0xe2f86d63 (ORAFILES) at power 1 Starting background process ARB0 Mon May 02 11:44:06 2011 ARB0 started with pid=20, OS id=15827 NOTE: assigning ARB0 to group 1/0xe2f86d63 (ORAFILES) Mon May 02 11:44:12 2011 NOTE: stopping process ARB0 SUCCESS: rebalance completed for group 1/0xe2f86d63 (ORAFILES) Mon May 02 11:44:15 2011 NOTE: requesting all-instance membership refresh for group=1 NOTE: initiating PST update: grp = 1 kfdp_update(): 8 Mon May 02 11:44:18 2011 kfdp_updateBg(): 8 NOTE: PST update grp = 1 completed successfully NOTE: initiating PST update: grp = 1 kfdp_update(): 9 kfdp_updateBg(): 9 NOTE: PST update grp = 1 completed successfully NOTE: membership refresh pending for group 1/0xe2f86d63 (ORAFILES) kfdp_query(): 10 kfdp_queryBg(): 10 SUCCESS: refreshed membership for 1/0xe2f86d63 (ORAFILES) Thu May 12 21:31:32 2011 NOTE: ASM client dwh1:dwh1 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_13041.trc Thu May 12 21:31:33 2011 Trace dumping is performing id=[cdmp_20110512213133] Thu May 12 21:31:53 2011 Starting background process ASMB Thu May 12 21:31:53 2011 ASMB started with pid=21, OS id=3113 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_3105.trc: ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +ORAFILES/dwh1/spfiledwh1.ora ORA-17503: ksfdopn:2 Failed to open file +ORAFILES/dwh1/spfiledwh1.ora ORA-15173: entry 'spfiledwh1.ora' does not exist in directory 'dwh1' NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu May 12 21:33:15 2011 Starting background process ASMB Thu May 12 21:33:15 2011 ASMB started with pid=21, OS id=3968 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu May 12 21:38:22 2011 Starting background process ASMB Thu May 12 21:38:22 2011 ASMB started with pid=21, OS id=7012 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu May 12 21:39:13 2011 Starting background process ASMB Thu May 12 21:39:13 2011 ASMB started with pid=21, OS id=7563 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu May 12 21:39:56 2011 Starting background process ASMB Thu May 12 21:39:56 2011 ASMB started with pid=21, OS id=7986 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu May 12 21:40:18 2011 Starting background process ASMB Thu May 12 21:40:18 2011 ASMB started with pid=21, OS id=8340 Thu May 12 21:40:31 2011 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu May 12 21:57:12 2011 NOTE: ASM client dwh1:dwh1 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_8463.trc Thu May 12 21:57:12 2011 Trace dumping is performing id=[cdmp_20110512215712] Thu May 12 22:01:40 2011 Starting background process ASMB Thu May 12 22:01:40 2011 ASMB started with pid=21, OS id=20366 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu May 12 22:10:24 2011 Starting background process ASMB Thu May 12 22:10:24 2011 ASMB started with pid=21, OS id=5641 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Fri May 13 07:09:44 2011 Starting background process ASMB Fri May 13 07:09:44 2011 ASMB started with pid=20, OS id=31917 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Fri May 13 16:42:10 2011 Starting background process ASMB Fri May 13 16:42:10 2011 ASMB started with pid=20, OS id=14262 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Fri May 13 17:08:50 2011 Reconfiguration started (old inc 4, new inc 6) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri May 13 17:08:50 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5460 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Fri May 13 17:08:50 2011 kfdp_query(): 11 Fri May 13 17:08:51 2011 kfdp_queryBg(): 11 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=2 ckpt=9.8 group=1 NOTE: advancing ckpt for thread=2 ckpt=9.8 NOTE: smon did instance recovery for domain 1 Fri May 13 17:19:19 2011 Shutting down instance (abort) License high water mark = 11 USER (ospid: 17713): terminating the instance Instance terminated by USER, pid = 17713 Fri May 13 17:19:20 2011 Instance shutdown complete Fri May 13 17:23:44 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Fri May 13 17:23:46 2011 PMON started with pid=2, OS id=7089 Fri May 13 17:23:46 2011 VKTM started with pid=3, OS id=7091 at elevated priority VKTM running at (20)ms precision Fri May 13 17:23:46 2011 DIAG started with pid=4, OS id=7095 Fri May 13 17:23:46 2011 PING started with pid=5, OS id=7097 Fri May 13 17:23:46 2011 PSP0 started with pid=6, OS id=7103 Fri May 13 17:23:46 2011 DIA0 started with pid=7, OS id=7107 Fri May 13 17:23:46 2011 LMON started with pid=8, OS id=7109 Fri May 13 17:23:46 2011 LMD0 started with pid=9, OS id=7111 Fri May 13 17:23:46 2011 LMS0 started with pid=10, OS id=7113 at elevated priority Fri May 13 17:23:46 2011 MMAN started with pid=11, OS id=7117 Fri May 13 17:23:46 2011 DBW0 started with pid=12, OS id=7119 Fri May 13 17:23:46 2011 LGWR started with pid=13, OS id=7121 Fri May 13 17:23:46 2011 CKPT started with pid=14, OS id=7123 Fri May 13 17:23:46 2011 SMON started with pid=15, OS id=7125 Fri May 13 17:23:46 2011 RBAL started with pid=16, OS id=7127 Fri May 13 17:23:46 2011 GMON started with pid=17, OS id=7129 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 4) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Fri May 13 17:23:47 2011 LCK0 started with pid=18, OS id=7134 ORACLE_BASE from environment = /u01/app/oracle Fri May 13 17:23:48 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x2ea84e7d NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x2ea84e7d WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x2EA84E7D (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.573 ABA NOTE: seq=10 blk=9 NOTE: cache mounting group 1/0x2EA84E7D (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x2ea84e7d kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Fri May 13 17:42:19 2011 Starting background process ASMB Fri May 13 17:42:19 2011 ASMB started with pid=20, OS id=17512 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Fri May 13 18:58:19 2011 NOTE: ASM client dwh1:dwh1 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_17593.trc Fri May 13 18:58:19 2011 Trace dumping is performing id=[cdmp_20110513185819] Fri May 13 18:59:05 2011 Starting background process ASMB Fri May 13 18:59:05 2011 ASMB started with pid=20, OS id=8583 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Fri May 13 19:00:51 2011 Starting background process ASMB Fri May 13 19:00:51 2011 ASMB started with pid=20, OS id=9840 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Fri May 13 19:03:40 2011 Starting background process ASMB Fri May 13 19:03:40 2011 ASMB started with pid=20, OS id=23993 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon May 16 22:04:46 2011 Reconfiguration started (old inc 4, new inc 6) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Mon May 16 22:04:46 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5405 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Mon May 16 22:04:46 2011 kfdp_query(): 4 Mon May 16 22:04:47 2011 NOTE: PST enabling heartbeating (grp 1) kfdp_queryBg(): 4 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=16.5607 group=1 NOTE: advancing ckpt for thread=1 ckpt=16.5607 NOTE: smon did instance recovery for domain 1 Mon May 16 22:08:46 2011 Reconfiguration started (old inc 6, new inc 8) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Mon May 16 22:08:46 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5405 GCS shadows traversed, 2914 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue May 17 22:02:04 2011 Reconfiguration started (old inc 8, new inc 10) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue May 17 22:02:04 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5460 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Tue May 17 22:02:04 2011 kfdp_query(): 5 Reconfiguration complete Tue May 17 22:02:04 2011 kfdp_queryBg(): 5 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=17.8337 group=1 NOTE: advancing ckpt for thread=1 ckpt=17.8337 NOTE: smon did instance recovery for domain 1 Fri May 20 19:10:49 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dia0_7107.trc: ORA-27508: IPC error sending a message ORA-27300: OS system dependent operation:sendmsg failed with status: 22 ORA-27301: OS failure message: Invalid argument ORA-27302: failure occurred at: sskgxpsnd1 Fri May 20 19:16:36 2011 IPC Send timeout detected.Sender: ospid 7097 Receiver: inst 1 binc 429411995 ospid 7097 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ping_7097.trc: ORA-27508: IPC error sending a message Fri May 20 22:02:25 2011 Reconfiguration started (old inc 10, new inc 12) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri May 20 22:02:25 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5460 GCS shadows traversed, 2804 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Mon May 23 22:35:20 2011 Reconfiguration started (old inc 12, new inc 14) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Mon May 23 22:35:20 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5460 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Mon May 23 22:35:20 2011 kfdp_query(): 6 Mon May 23 22:35:20 2011 kfdp_queryBg(): 6 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=19.166 group=1 NOTE: advancing ckpt for thread=1 ckpt=19.166 NOTE: smon did instance recovery for domain 1 Wed Jun 08 09:18:22 2011 ospid 7111: network interface with IP address 10.0.0.185 no longer running (check cable) Wed Jun 08 09:24:22 2011 ospid 7111: network interface with IP address 10.0.0.185 is now running Wed Jun 08 09:43:54 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dia0_7107.trc: ORA-27508: IPC error sending a message ORA-27300: OS system dependent operation:sendmsg failed with status: 22 ORA-27301: OS failure message: Invalid argument ORA-27302: failure occurred at: sskgxpsnd1 Wed Jun 08 09:49:33 2011 IPC Send timeout detected.Sender: ospid 7097 Receiver: inst 1 binc 429411995 ospid 7097 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ping_7097.trc: ORA-27508: IPC error sending a message Wed Jun 08 11:53:19 2011 Reconfiguration started (old inc 14, new inc 16) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Jun 08 11:53:19 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5460 GCS shadows traversed, 2779 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Jun 08 22:13:39 2011 Reconfiguration started (old inc 16, new inc 18) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Jun 08 22:13:39 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5460 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Wed Jun 08 22:13:39 2011 kfdp_query(): 7 Wed Jun 08 22:13:39 2011 kfdp_queryBg(): 7 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=20.484 group=1 NOTE: advancing ckpt for thread=1 ckpt=20.484 NOTE: smon did instance recovery for domain 1 Thu Jun 09 09:59:59 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dia0_7107.trc: ORA-27508: IPC error sending a message ORA-27300: OS system dependent operation:sendmsg failed with status: 22 ORA-27301: OS failure message: Invalid argument ORA-27302: failure occurred at: sskgxpsnd1 Thu Jun 09 10:00:03 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_21650.trc (incident=8801): ORA-00603: ORACLE server session terminated by fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed with status: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 10.0.0.185 not found. Check output from ifconfig command Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_8801/+ASM1_ora_21650_i8801.trc Thu Jun 09 10:00:05 2011 Trace dumping is performing id=[cdmp_20110609100005] ORA-603 : opiodr aborting process unknown ospid (21650_140464494659296) Thu Jun 09 10:00:05 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_21668.trc (incident=8802): ORA-00603: ORACLE server session terminated by fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed with status: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 10.0.0.185 not found. Check output from ifconfig command Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_8802/+ASM1_ora_21668_i8802.trc Trace dumping is performing id=[cdmp_20110609100006] ORA-603 : opiodr aborting process unknown ospid (21668_140330100676320) Thu Jun 09 10:00:07 2011 ospid 7111: network interface with IP address 10.0.0.185 no longer operational requested interface 10.0.0.185 not found. Check output from ifconfig command Thu Jun 09 10:00:10 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_21898.trc (incident=8803): ORA-00603: ORACLE server session terminated by fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed with status: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 10.0.0.185 not found. Check output from ifconfig command Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_8803/+ASM1_ora_21898_i8803.trc Thu Jun 09 10:00:10 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_21896.trc (incident=8804): ORA-00603: ORACLE server session terminated by fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed with status: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 10.0.0.185 not found. Check output from ifconfig command Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_8804/+ASM1_ora_21896_i8804.trc ORA-603 : opiodr aborting process unknown ospid (21898_140533404829408) ORA-603 : opiodr aborting process unknown ospid (21896_140408920979168) Thu Jun 09 10:00:11 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_21912.trc (incident=8805): ORA-00603: ORACLE server session terminated by fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed with status: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 10.0.0.185 not found. Check output from ifconfig command Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_8805/+ASM1_ora_21912_i8805.trc Thu Jun 09 10:00:11 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_21910.trc (incident=8806): ORA-00603: ORACLE server session terminated by fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed with status: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 10.0.0.185 not found. Check output from ifconfig command ORA-603 : opiodr aborting process unknown ospid (21910_140418672858848) Trace dumping is performing id=[cdmp_20110609100011] Thu Jun 09 10:00:11 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_21938.trc (incident=8807): ORA-00603: ORACLE server session terminated by fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed with status: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 10.0.0.185 not found. Check output from ifconfig command ORA-603 : opiodr aborting process unknown ospid (21938_140053587314400) ORA-603 : opiodr aborting process unknown ospid (21912_139747009836768) Trace dumping is performing id=[cdmp_20110609100012] Thu Jun 09 10:00:12 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_21942.trc (incident=8808): ORA-00603: ORACLE server session terminated by fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed with status: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 10.0.0.185 not found. Check output from ifconfig command ORA-603 : opiodr aborting process unknown ospid (21942_140045171680992) Thu Jun 09 10:00:12 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_21944.trc (incident=8809): ORA-00603: ORACLE server session terminated by fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed with status: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 10.0.0.185 not found. Check output from ifconfig command ORA-603 : opiodr aborting process unknown ospid (21944_140105985988320) Thu Jun 09 10:00:12 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_21954.trc (incident=8810): ORA-00603: ORACLE server session terminated by fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed with status: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 10.0.0.185 not found. Check output from ifconfig command ORA-603 : opiodr aborting process unknown ospid (21954_140069121795808) Thu Jun 09 10:00:12 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_21952.trc (incident=8811): ORA-00603: ORACLE server session terminated by fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed with status: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 10.0.0.185 not found. Check output from ifconfig command ORA-603 : opiodr aborting process unknown ospid (21952_139759512409824) Thu Jun 09 10:00:21 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_22053.trc (incident=8812): ORA-00603: ORACLE server session terminated by fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed with status: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 10.0.0.185 not found. Check output from ifconfig command ORA-603 : opiodr aborting process unknown ospid (22053_140454317000416) Thu Jun 09 10:00:22 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_22055.trc (incident=8813): ORA-00603: ORACLE server session terminated by fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed with status: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 10.0.0.185 not found. Check output from ifconfig command ORA-603 : opiodr aborting process unknown ospid (22055_139641372350176) Thu Jun 09 10:05:50 2011 IPC Send timeout detected.Sender: ospid 7097 Receiver: inst 1 binc 429411995 ospid 7097 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ping_7097.trc: ORA-27508: IPC error sending a message Thu Jun 09 10:18:01 2011 Reconfiguration started (old inc 18, new inc 20) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Thu Jun 09 10:18:01 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5460 GCS shadows traversed, 2791 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Thu Jun 09 16:03:09 2011 Received detach msg from node 1 for dom 1 Thu Jun 09 16:03:12 2011 Reconfiguration started (old inc 20, new inc 22) List of nodes: 0 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Thu Jun 09 16:03:12 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5405 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Thu Jun 09 16:07:09 2011 Shutting down instance (abort) License high water mark = 12 USER (ospid: 5088): terminating the instance Instance terminated by USER, pid = 5088 Thu Jun 09 16:07:10 2011 Instance shutdown complete Thu Jun 09 16:30:10 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Thu Jun 09 16:30:11 2011 PMON started with pid=2, OS id=7816 Thu Jun 09 16:30:11 2011 VKTM started with pid=3, OS id=7818 at elevated priority VKTM running at (20)ms precision Thu Jun 09 16:30:11 2011 DIAG started with pid=4, OS id=7822 Thu Jun 09 16:30:11 2011 PING started with pid=5, OS id=7824 Thu Jun 09 16:30:11 2011 PSP0 started with pid=6, OS id=7826 Thu Jun 09 16:30:11 2011 DIA0 started with pid=7, OS id=7828 Thu Jun 09 16:30:11 2011 LMON started with pid=8, OS id=7830 Thu Jun 09 16:30:11 2011 LMD0 started with pid=9, OS id=7832 Thu Jun 09 16:30:11 2011 LMS0 started with pid=10, OS id=7834 at elevated priority Thu Jun 09 16:30:11 2011 MMAN started with pid=11, OS id=7838 Thu Jun 09 16:30:11 2011 DBW0 started with pid=12, OS id=7840 Thu Jun 09 16:30:11 2011 LGWR started with pid=13, OS id=7842 Thu Jun 09 16:30:11 2011 CKPT started with pid=14, OS id=7844 Thu Jun 09 16:30:11 2011 SMON started with pid=15, OS id=7846 Thu Jun 09 16:30:11 2011 RBAL started with pid=16, OS id=7848 Thu Jun 09 16:30:11 2011 GMON started with pid=17, OS id=7850 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 4) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Thu Jun 09 16:30:12 2011 LCK0 started with pid=18, OS id=7861 ORACLE_BASE from environment = /u01/app/oracle Thu Jun 09 16:30:13 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58e85808 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58e85808 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58E85808 (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.8974608 ABA NOTE: seq=37 blk=4811 NOTE: cache mounting group 1/0x58E85808 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58e85808 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Thu Jun 09 16:30:25 2011 Starting background process ASMB Thu Jun 09 16:30:25 2011 ASMB started with pid=20, OS id=8084 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Jun 09 16:33:53 2011 Starting background process ASMB Thu Jun 09 16:33:53 2011 ASMB started with pid=20, OS id=10908 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Jun 09 22:23:58 2011 Reconfiguration started (old inc 4, new inc 6) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Thu Jun 09 22:23:58 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 4109 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Thu Jun 09 22:23:58 2011 kfdp_query(): 4 Thu Jun 09 22:23:58 2011 NOTE: PST enabling heartbeating (grp 1) kfdp_queryBg(): 4 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=22.1673 group=1 NOTE: advancing ckpt for thread=1 ckpt=22.1673 NOTE: smon did instance recovery for domain 1 Fri Jun 10 08:06:19 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dia0_7828.trc: ORA-27508: IPC error sending a message ORA-27300: OS system dependent operation:sendmsg failed with status: 22 ORA-27301: OS failure message: Invalid argument ORA-27302: failure occurred at: sskgxpsnd1 Sat Jun 11 03:48:47 2011 Reconfiguration started (old inc 6, new inc 8) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Jun 11 03:48:47 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5460 GCS shadows traversed, 3007 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Jun 11 21:02:24 2011 Reconfiguration started (old inc 8, new inc 10) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Jun 11 21:02:24 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5459 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sat Jun 11 21:02:24 2011 kfdp_query(): 5 Sat Jun 11 21:02:24 2011 kfdp_queryBg(): 5 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=24.9028 group=1 NOTE: advancing ckpt for thread=1 ckpt=24.9028 NOTE: smon did instance recovery for domain 1 Sun Jun 12 02:32:01 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dia0_7828.trc: ORA-27508: IPC error sending a message ORA-27300: OS system dependent operation:sendmsg failed with status: 22 ORA-27301: OS failure message: Invalid argument ORA-27302: failure occurred at: sskgxpsnd1 Sun Jun 12 03:15:02 2011 Reconfiguration started (old inc 10, new inc 12) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sun Jun 12 03:15:02 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5460 GCS shadows traversed, 3224 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sun Jun 12 12:24:18 2011 Reconfiguration started (old inc 12, new inc 14) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sun Jun 12 12:24:18 2011 LMS 0: 1 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Sun Jun 12 12:24:18 2011 kfdp_query(): 6 LMS 0: 5459 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sun Jun 12 12:24:18 2011 NOTE: waiting for instance recovery of group 1 Sun Jun 12 12:24:18 2011 kfdp_queryBg(): 6 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=26.4309 group=1 NOTE: advancing ckpt for thread=1 ckpt=26.4309 NOTE: smon did instance recovery for domain 1 Mon Jun 13 00:11:52 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dia0_7828.trc: ORA-27508: IPC error sending a message ORA-27300: OS system dependent operation:sendmsg failed with status: 22 ORA-27301: OS failure message: Invalid argument ORA-27302: failure occurred at: sskgxpsnd1 Mon Jun 13 05:29:19 2011 Reconfiguration started (old inc 14, new inc 16) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Mon Jun 13 05:29:19 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5460 GCS shadows traversed, 2761 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Jun 14 22:23:56 2011 Reconfiguration started (old inc 16, new inc 18) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Jun 14 22:23:56 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5460 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Tue Jun 14 22:23:56 2011 kfdp_query(): 7 Tue Jun 14 22:23:56 2011 kfdp_queryBg(): 7 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=27.8767 group=1 NOTE: advancing ckpt for thread=1 ckpt=27.8767 NOTE: smon did instance recovery for domain 1 Tue Jun 14 23:07:45 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dia0_7828.trc: ORA-27508: IPC error sending a message ORA-27300: OS system dependent operation:sendmsg failed with status: 22 ORA-27301: OS failure message: Invalid argument ORA-27302: failure occurred at: sskgxpsnd1 Tue Jun 14 23:07:56 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_5519.trc (incident=9614): ORA-00603: ORACLE server session terminated by fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed with status: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 10.0.0.185 not found. Check output from ifconfig command Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_9614/+ASM1_ora_5519_i9614.trc ORA-603 : opiodr aborting process unknown ospid (5519_139703182288608) Tue Jun 14 23:07:57 2011 Trace dumping is performing id=[cdmp_20110614230757] Tue Jun 14 23:07:57 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_5526.trc (incident=9615): ORA-00603: ORACLE server session terminated by fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed with status: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 10.0.0.185 not found. Check output from ifconfig command Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_9615/+ASM1_ora_5526_i9615.trc ORA-603 : opiodr aborting process unknown ospid (5526_140310452836064) Trace dumping is performing id=[cdmp_20110614230757] Tue Jun 14 23:11:05 2011 Reconfiguration started (old inc 18, new inc 20) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Jun 14 23:11:06 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5459 GCS shadows traversed, 2843 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Jun 14 23:14:16 2011 IPC Send timeout detected.Sender: ospid 7824 Receiver: inst 1 binc 429414437 ospid 7824 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ping_7824.trc: ORA-27508: IPC error sending a message Fri Jun 17 20:58:21 2011 Shutting down instance (abort) License high water mark = 11 USER (ospid: 20457): terminating the instance Instance terminated by USER, pid = 20457 Fri Jun 17 20:58:23 2011 Instance shutdown complete Fri Jun 17 21:15:37 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Fri Jun 17 21:15:39 2011 PMON started with pid=2, OS id=17939 Fri Jun 17 21:15:39 2011 VKTM started with pid=3, OS id=17941 at elevated priority VKTM running at (20)ms precision Fri Jun 17 21:15:39 2011 DIAG started with pid=4, OS id=17945 Fri Jun 17 21:15:39 2011 PING started with pid=5, OS id=17947 Fri Jun 17 21:15:39 2011 PSP0 started with pid=6, OS id=17949 Fri Jun 17 21:15:39 2011 DIA0 started with pid=7, OS id=17951 Fri Jun 17 21:15:39 2011 LMON started with pid=8, OS id=17953 Fri Jun 17 21:15:39 2011 LMD0 started with pid=9, OS id=17955 Fri Jun 17 21:15:39 2011 LMS0 started with pid=10, OS id=17957 at elevated priority Fri Jun 17 21:15:39 2011 MMAN started with pid=11, OS id=17961 Fri Jun 17 21:15:39 2011 DBW0 started with pid=12, OS id=17963 Fri Jun 17 21:15:39 2011 LGWR started with pid=13, OS id=17965 Fri Jun 17 21:15:39 2011 CKPT started with pid=14, OS id=17967 Fri Jun 17 21:15:39 2011 SMON started with pid=15, OS id=17969 Fri Jun 17 21:15:39 2011 RBAL started with pid=16, OS id=17971 Fri Jun 17 21:15:39 2011 GMON started with pid=17, OS id=17973 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 24) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Fri Jun 17 21:15:40 2011 LCK0 started with pid=18, OS id=17989 ORACLE_BASE from environment = /u01/app/oracle Fri Jun 17 21:15:41 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58e0feb2 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58e0feb2 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58E0FEB2 (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.12728024 ABA NOTE: seq=50 blk=1472 NOTE: cache mounting group 1/0x58E0FEB2 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58e0feb2 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Fri Jun 17 21:15:48 2011 Starting background process ASMB Fri Jun 17 21:15:48 2011 ASMB started with pid=20, OS id=18116 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Jun 21 22:33:47 2011 Reconfiguration started (old inc 24, new inc 26) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Jun 21 22:33:47 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5457 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Tue Jun 21 22:33:47 2011 kfdp_query(): 4 Reconfiguration complete Tue Jun 21 22:33:48 2011 NOTE: PST enabling heartbeating (grp 1) kfdp_queryBg(): 4 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=34.338 group=1 NOTE: advancing ckpt for thread=1 ckpt=34.338 NOTE: smon did instance recovery for domain 1 Tue Jun 21 23:05:43 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dia0_17951.trc: ORA-27508: IPC error sending a message ORA-27300: OS system dependent operation:sendmsg failed with status: 22 ORA-27301: OS failure message: Invalid argument ORA-27302: failure occurred at: sskgxpsnd1 Tue Jun 21 23:11:24 2011 IPC Send timeout detected.Sender: ospid 17947 Receiver: inst 1 binc 500235973 ospid 17947 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ping_17947.trc: ORA-27508: IPC error sending a message Tue Jun 21 23:20:59 2011 Reconfiguration started (old inc 26, new inc 28) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Jun 21 23:21:00 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5460 GCS shadows traversed, 2856 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Jun 22 15:25:16 2011 Reconfiguration started (old inc 28, new inc 30) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Jun 22 15:25:16 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Wed Jun 22 15:25:16 2011 kfdp_query(): 5 LMS 0: 5451 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Wed Jun 22 15:25:16 2011 kfdp_queryBg(): 5 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=35.3774 group=1 NOTE: advancing ckpt for thread=1 ckpt=35.3774 NOTE: smon did instance recovery for domain 1 Wed Jun 22 16:04:53 2011 Reconfiguration started (old inc 30, new inc 32) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Jun 22 16:04:54 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5458 GCS shadows traversed, 2856 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Jun 22 20:31:17 2011 Shutting down instance (abort) License high water mark = 11 USER (ospid: 5789): terminating the instance Instance terminated by USER, pid = 5789 Wed Jun 22 20:31:18 2011 Instance shutdown complete Wed Jun 22 20:36:12 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Wed Jun 22 20:36:14 2011 PMON started with pid=2, OS id=7703 Wed Jun 22 20:36:14 2011 VKTM started with pid=3, OS id=7705 at elevated priority Wed Jun 22 20:36:14 2011 DIAG started with pid=4, OS id=7709 VKTM running at (20)ms precision Wed Jun 22 20:36:14 2011 PING started with pid=5, OS id=7711 Wed Jun 22 20:36:14 2011 PSP0 started with pid=6, OS id=7713 Wed Jun 22 20:36:14 2011 DIA0 started with pid=7, OS id=7715 Wed Jun 22 20:36:14 2011 LMON started with pid=8, OS id=7717 Wed Jun 22 20:36:14 2011 LMD0 started with pid=9, OS id=7719 Wed Jun 22 20:36:14 2011 LMS0 started with pid=10, OS id=7721 at elevated priority Wed Jun 22 20:36:14 2011 MMAN started with pid=11, OS id=7725 Wed Jun 22 20:36:15 2011 DBW0 started with pid=12, OS id=7733 Wed Jun 22 20:36:15 2011 LGWR started with pid=13, OS id=7735 Wed Jun 22 20:36:15 2011 CKPT started with pid=14, OS id=7737 Wed Jun 22 20:36:15 2011 SMON started with pid=15, OS id=7739 Wed Jun 22 20:36:15 2011 RBAL started with pid=16, OS id=7741 Wed Jun 22 20:36:15 2011 GMON started with pid=17, OS id=7743 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 36) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Jun 22 20:36:15 2011 LCK0 started with pid=18, OS id=7748 ORACLE_BASE from environment = /u01/app/oracle Wed Jun 22 20:36:16 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58e8580b NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58e8580b WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58E8580B (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.14679244 ABA NOTE: seq=55 blk=6429 NOTE: cache mounting group 1/0x58E8580B (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58e8580b kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Wed Jun 22 20:36:29 2011 Starting background process ASMB Wed Jun 22 20:36:29 2011 ASMB started with pid=20, OS id=7975 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Jun 23 15:55:41 2011 Reconfiguration started (old inc 36, new inc 38) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Thu Jun 23 15:55:41 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5436 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Thu Jun 23 15:55:41 2011 kfdp_query(): 4 Thu Jun 23 15:55:41 2011 NOTE: PST enabling heartbeating (grp 1) kfdp_queryBg(): 4 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=36.7503 group=1 NOTE: advancing ckpt for thread=1 ckpt=36.7503 NOTE: smon did instance recovery for domain 1 Thu Jun 23 16:35:58 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dia0_7715.trc: ORA-27508: IPC error sending a message ORA-27300: OS system dependent operation:sendmsg failed with status: 22 ORA-27301: OS failure message: Invalid argument ORA-27302: failure occurred at: sskgxpsnd1 Thu Jun 23 16:37:48 2011 Reconfiguration started (old inc 38, new inc 40) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Thu Jun 23 16:37:48 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5460 GCS shadows traversed, 2835 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Thu Jun 23 16:40:50 2011 Reconfiguration started (old inc 40, new inc 42) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Thu Jun 23 16:40:50 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Thu Jun 23 16:40:50 2011 kfdp_query(): 5 LMS 0: 5460 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Thu Jun 23 16:40:50 2011 kfdp_queryBg(): 5 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=37.7576 group=1 NOTE: advancing ckpt for thread=1 ckpt=37.7576 NOTE: smon did instance recovery for domain 1 Thu Jun 23 19:22:04 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dia0_7715.trc: ORA-27508: IPC error sending a message ORA-27300: OS system dependent operation:sendmsg failed with status: 22 ORA-27301: OS failure message: Invalid argument ORA-27302: failure occurred at: sskgxpsnd1 Thu Jun 23 19:23:13 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_28227.trc (incident=11216): ORA-00603: ORACLE server session terminated by fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed with status: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 10.0.0.185 not found. Check output from ifconfig command Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_11216/+ASM1_ora_28227_i11216.trc ORA-603 : opiodr aborting process unknown ospid (28227_139832638129888) Thu Jun 23 19:23:15 2011 Trace dumping is performing id=[cdmp_20110623192315] Thu Jun 23 19:27:56 2011 IPC Send timeout detected.Sender: ospid 7711 Receiver: inst 1 binc 429414458 ospid 7711 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ping_7711.trc: ORA-27508: IPC error sending a message Thu Jun 23 19:55:09 2011 IPC Send timeout detected.Sender: ospid 7711 Receiver: inst 1 binc 429414458 ospid 7711 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ping_7711.trc: ORA-27508: IPC error sending a message Thu Jun 23 21:41:11 2011 Reconfiguration started (old inc 42, new inc 44) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Thu Jun 23 21:41:11 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5460 GCS shadows traversed, 2819 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Thu Jun 23 21:41:15 2011 IPC Send timeout detected.Sender: ospid 7711 Receiver: inst 1 binc 429414458 ospid 7711 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ping_7711.trc: ORA-27508: IPC error sending a message Fri Jun 24 22:20:48 2011 Reconfiguration started (old inc 44, new inc 46) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Jun 24 22:20:48 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Fri Jun 24 22:20:48 2011 kfdp_query(): 6 LMS 0: 5460 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Fri Jun 24 22:20:48 2011 kfdp_queryBg(): 6 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=38.10532 group=1 NOTE: advancing ckpt for thread=1 ckpt=38.10532 NOTE: smon did instance recovery for domain 1 Fri Jun 24 22:25:21 2011 Reconfiguration started (old inc 46, new inc 48) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Jun 24 22:25:22 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5459 GCS shadows traversed, 2841 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Jun 25 06:26:35 2011 Reconfiguration started (old inc 48, new inc 50) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Jun 25 06:26:35 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5457 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Sat Jun 25 06:26:35 2011 kfdp_query(): 7 Reconfiguration complete Sat Jun 25 06:26:35 2011 kfdp_queryBg(): 7 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=40.4370 group=1 NOTE: advancing ckpt for thread=1 ckpt=40.4370 NOTE: smon did instance recovery for domain 1 Sat Jun 25 06:31:04 2011 Reconfiguration started (old inc 50, new inc 52) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Jun 25 06:31:04 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5457 GCS shadows traversed, 2778 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Mon Jun 27 11:04:52 2011 Reconfiguration started (old inc 52, new inc 54) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Mon Jun 27 11:04:52 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5460 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Mon Jun 27 11:04:52 2011 kfdp_query(): 8 Reconfiguration complete Mon Jun 27 11:04:52 2011 kfdp_queryBg(): 8 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=42.2814 group=1 NOTE: advancing ckpt for thread=1 ckpt=42.2814 NOTE: smon did instance recovery for domain 1 Mon Jun 27 12:28:23 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dia0_7715.trc: ORA-27508: IPC error sending a message ORA-27300: OS system dependent operation:sendmsg failed with status: 22 ORA-27301: OS failure message: Invalid argument ORA-27302: failure occurred at: sskgxpsnd1 Mon Jun 27 12:29:53 2011 Reconfiguration started (old inc 54, new inc 56) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Mon Jun 27 12:29:53 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5459 GCS shadows traversed, 2748 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Mon Jun 27 15:46:08 2011 Reconfiguration started (old inc 56, new inc 58) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Mon Jun 27 15:46:08 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Mon Jun 27 15:46:08 2011 kfdp_query(): 9 LMS 0: 5460 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Mon Jun 27 15:46:08 2011 kfdp_queryBg(): 9 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=43.2921 group=1 NOTE: advancing ckpt for thread=1 ckpt=43.2921 NOTE: smon did instance recovery for domain 1 Mon Jun 27 17:36:23 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dia0_7715.trc: ORA-27508: IPC error sending a message ORA-27300: OS system dependent operation:sendmsg failed with status: 22 ORA-27301: OS failure message: Invalid argument ORA-27302: failure occurred at: sskgxpsnd1 Mon Jun 27 17:37:52 2011 Reconfiguration started (old inc 58, new inc 60) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Mon Jun 27 17:37:52 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5459 GCS shadows traversed, 2845 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Mon Jun 27 17:43:03 2011 IPC Send timeout detected.Sender: ospid 7711 Receiver: inst 1 binc 429414458 ospid 7711 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ping_7711.trc: ORA-27508: IPC error sending a message Mon Jun 27 18:06:06 2011 Shutting down instance (abort) License high water mark = 14 USER (ospid: 7219): terminating the instance Instance terminated by USER, pid = 7219 Mon Jun 27 18:06:07 2011 Instance shutdown complete Mon Jun 27 19:06:26 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth3 10.0.0.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.0.0.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Mon Jun 27 19:06:28 2011 PMON started with pid=2, OS id=4266 Mon Jun 27 19:06:28 2011 VKTM started with pid=3, OS id=4268 at elevated priority VKTM running at (20)ms precision Mon Jun 27 19:06:28 2011 DIAG started with pid=4, OS id=4272 Mon Jun 27 19:06:28 2011 PING started with pid=5, OS id=4274 Mon Jun 27 19:06:28 2011 PSP0 started with pid=6, OS id=4276 Mon Jun 27 19:06:28 2011 DIA0 started with pid=7, OS id=4278 Mon Jun 27 19:06:28 2011 LMON started with pid=8, OS id=4280 Mon Jun 27 19:06:28 2011 LMD0 started with pid=9, OS id=4282 Mon Jun 27 19:06:28 2011 LMS0 started with pid=10, OS id=4284 at elevated priority Mon Jun 27 19:06:28 2011 MMAN started with pid=11, OS id=4288 Mon Jun 27 19:06:28 2011 DBW0 started with pid=12, OS id=4290 Mon Jun 27 19:06:28 2011 LGWR started with pid=13, OS id=4296 Mon Jun 27 19:06:28 2011 CKPT started with pid=14, OS id=4300 Mon Jun 27 19:06:28 2011 SMON started with pid=15, OS id=4302 Mon Jun 27 19:06:28 2011 RBAL started with pid=16, OS id=4304 Mon Jun 27 19:06:28 2011 GMON started with pid=17, OS id=4306 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 64) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Mon Jun 27 19:06:29 2011 LCK0 started with pid=18, OS id=4316 ORACLE_BASE from environment = /u01/app/oracle Mon Jun 27 19:06:30 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0x58e9dbd0 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0x58e9dbd0 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0x58E9DBD0 (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.16467001 ABA NOTE: seq=61 blk=1924 NOTE: cache mounting group 1/0x58E9DBD0 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0x58e9dbd0 kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Mon Jun 27 19:06:40 2011 Starting background process ASMB Mon Jun 27 19:06:40 2011 ASMB started with pid=20, OS id=4485 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Jun 28 15:21:27 2011 Reconfiguration started (old inc 64, new inc 66) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Jun 28 15:21:27 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5460 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Tue Jun 28 15:21:27 2011 kfdp_query(): 4 Tue Jun 28 15:21:27 2011 NOTE: PST enabling heartbeating (grp 1) kfdp_queryBg(): 4 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=44.6194 group=1 NOTE: advancing ckpt for thread=1 ckpt=44.6194 NOTE: smon did instance recovery for domain 1 Tue Jun 28 15:26:03 2011 Reconfiguration started (old inc 66, new inc 68) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Jun 28 15:26:03 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5459 GCS shadows traversed, 2838 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Jun 28 22:29:14 2011 Reconfiguration started (old inc 68, new inc 70) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Jun 28 22:29:14 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Tue Jun 28 22:29:14 2011 kfdp_query(): 5 LMS 0: 5460 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Tue Jun 28 22:29:14 2011 kfdp_queryBg(): 5 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=45.6283 group=1 NOTE: advancing ckpt for thread=1 ckpt=45.6283 NOTE: smon did instance recovery for domain 1 Tue Jun 28 22:49:33 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dia0_4278.trc: ORA-27508: IPC error sending a message ORA-27300: OS system dependent operation:sendmsg failed with status: 22 ORA-27301: OS failure message: Invalid argument ORA-27302: failure occurred at: sskgxpsnd1 Tue Jun 28 22:49:39 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26230.trc (incident=12017): ORA-00603: ORACLE server session terminated by fatal error ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:if_not_found failed with status: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 ORA-27303: additional information: requested interface 10.0.0.185 not found. Check output from ifconfig command Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_12017/+ASM1_ora_26230_i12017.trc ORA-603 : opiodr aborting process unknown ospid (26230_140518370457312) Tue Jun 28 22:49:39 2011 Trace dumping is performing id=[cdmp_20110628224939] Tue Jun 28 22:50:53 2011 Reconfiguration started (old inc 70, new inc 72) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Tue Jun 28 22:50:53 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5459 GCS shadows traversed, 2857 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Jun 28 22:56:12 2011 IPC Send timeout detected.Sender: ospid 4274 Receiver: inst 1 binc 429513708 ospid 4274 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ping_4274.trc: ORA-27508: IPC error sending a message Wed Jun 29 14:17:51 2011 Reconfiguration started (old inc 72, new inc 74) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Jun 29 14:17:51 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Wed Jun 29 14:17:51 2011 kfdp_query(): 6 LMS 0: 5460 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Wed Jun 29 14:17:51 2011 kfdp_queryBg(): 6 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=47.103 group=1 NOTE: advancing ckpt for thread=1 ckpt=47.103 NOTE: smon did instance recovery for domain 1 Wed Jun 29 14:22:23 2011 Reconfiguration started (old inc 74, new inc 76) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Jun 29 14:22:23 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5460 GCS shadows traversed, 2857 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Jun 29 17:57:02 2011 Reconfiguration started (old inc 76, new inc 78) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Jun 29 17:57:02 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Wed Jun 29 17:57:02 2011 kfdp_query(): 7 LMS 0: 5460 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Wed Jun 29 17:57:03 2011 kfdp_queryBg(): 7 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=48.189 group=1 NOTE: advancing ckpt for thread=1 ckpt=48.189 NOTE: smon did instance recovery for domain 1 Wed Jun 29 21:41:23 2011 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_dia0_4278.trc: ORA-27508: IPC error sending a message ORA-27300: OS system dependent operation:sendmsg failed with status: 22 ORA-27301: OS failure message: Invalid argument ORA-27302: failure occurred at: sskgxpsnd1 Wed Jun 29 21:43:07 2011 Reconfiguration started (old inc 78, new inc 80) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Jun 29 21:43:07 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5460 GCS shadows traversed, 2857 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Jun 29 21:48:08 2011 IPC Send timeout detected.Sender: ospid 4274 Receiver: inst 1 binc 429513708 ospid 4274 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ping_4274.trc: ORA-27508: IPC error sending a message Thu Jun 30 22:28:53 2011 Reconfiguration started (old inc 80, new inc 82) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Thu Jun 30 22:28:53 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Thu Jun 30 22:28:53 2011 kfdp_query(): 8 LMS 0: 5452 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Thu Jun 30 22:28:54 2011 kfdp_queryBg(): 8 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=49.5326 group=1 NOTE: advancing ckpt for thread=1 ckpt=49.5326 NOTE: smon did instance recovery for domain 1 Thu Jun 30 22:33:29 2011 Reconfiguration started (old inc 82, new inc 84) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Thu Jun 30 22:33:29 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5455 GCS shadows traversed, 2856 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Jul 02 06:27:08 2011 Reconfiguration started (old inc 84, new inc 86) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Jul 02 06:27:08 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5443 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sat Jul 02 06:27:09 2011 kfdp_query(): 9 Sat Jul 02 06:27:09 2011 kfdp_queryBg(): 9 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=51.5606 group=1 NOTE: advancing ckpt for thread=1 ckpt=51.5606 NOTE: smon did instance recovery for domain 1 Tue Jul 05 18:03:22 2011 Shutting down instance (abort) License high water mark = 11 USER (ospid: 28734): terminating the instance Instance terminated by USER, pid = 28734 Tue Jul 05 18:03:23 2011 Instance shutdown complete Tue Jul 05 18:08:28 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Tue Jul 05 18:08:30 2011 PMON started with pid=2, OS id=8044 Tue Jul 05 18:08:30 2011 VKTM started with pid=3, OS id=8046 at elevated priority Tue Jul 05 18:08:30 2011 DIAG started with pid=4, OS id=8050 VKTM running at (20)ms precision Tue Jul 05 18:08:30 2011 PING started with pid=5, OS id=8052 Tue Jul 05 18:08:30 2011 PSP0 started with pid=6, OS id=8054 Tue Jul 05 18:08:30 2011 DIA0 started with pid=7, OS id=8061 Tue Jul 05 18:08:30 2011 LMON started with pid=8, OS id=8064 Tue Jul 05 18:08:30 2011 LMD0 started with pid=9, OS id=8066 Tue Jul 05 18:08:30 2011 LMS0 started with pid=10, OS id=8068 at elevated priority Tue Jul 05 18:08:30 2011 MMAN started with pid=11, OS id=8072 Tue Jul 05 18:08:30 2011 DBW0 started with pid=12, OS id=8074 Tue Jul 05 18:08:30 2011 LGWR started with pid=13, OS id=8076 Tue Jul 05 18:08:30 2011 CKPT started with pid=14, OS id=8078 Tue Jul 05 18:08:30 2011 SMON started with pid=15, OS id=8080 Tue Jul 05 18:08:30 2011 RBAL started with pid=16, OS id=8082 Tue Jul 05 18:08:30 2011 GMON started with pid=17, OS id=8084 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 4) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Tue Jul 05 18:08:31 2011 LCK0 started with pid=18, OS id=8089 ORACLE_BASE from environment = /u01/app/oracle Tue Jul 05 18:08:32 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc6085a3c NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc6085a3c WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) kfdp_query(): 2 kfdp_queryBg(): 2 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0xC6085A3C (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.20046928 ABA NOTE: seq=93 blk=6532 NOTE: cache mounting group 1/0xC6085A3C (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc6085a3c kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Tue Jul 05 18:08:43 2011 Starting background process ASMB Tue Jul 05 18:08:43 2011 ASMB started with pid=20, OS id=8312 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Jul 05 18:11:17 2011 Starting background process ASMB Tue Jul 05 18:11:17 2011 ASMB started with pid=20, OS id=10798 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Jul 05 18:12:36 2011 Starting background process ASMB Tue Jul 05 18:12:36 2011 ASMB started with pid=20, OS id=11953 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Aug 16 09:02:28 2011 SQL> CREATE DISKGROUP FLASH External REDUNDANCY DISK '/dev/mpath/mpath3p1' WARNING: Deprecated privilege SYSDBA for command 'CREATE DISKGROUP' NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: initializing header on grp 2 disk FLASH_0000 NOTE: initiating PST update: grp = 2 kfdp_update(): 4 Tue Aug 16 09:02:29 2011 kfdp_updateBg(): 4 NOTE: group FLASH: initial PST location: disk 0000 (PST copy 0) NOTE: PST update grp = 2 completed successfully NOTE: cache registered group FLASH number=2 incarn=0x1c685a42 NOTE: cache began mount (first) of group FLASH number=2 incarn=0x1c685a42 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 Tue Aug 16 09:02:29 2011 * allocate domain 2, invalid = TRUE kjbdomatt send to node 1 Tue Aug 16 09:02:29 2011 NOTE: attached to recovery domain 2 NOTE: cache creating group 2/0x1C685A42 (FLASH) NOTE: cache mounting group 2/0x1C685A42 (FLASH) succeeded NOTE: allocating F1X0 on grp 2 disk FLASH_0000 NOTE: diskgroup must now be re-mounted prior to first use NOTE: cache dismounting group 2/0x1C685A42 (FLASH) NOTE: lgwr not being msg'd to dismount kjbdomdet send to node 1 detach from dom 2, sending detach message to node 1 freeing rdom 2 NOTE: detached from domain 2 NOTE: cache dismounted group 2/0x1C685A42 (FLASH) NOTE: cache ending mount (fail) of group FLASH number=2 incarn=0x1c685a42 kfdp_dismount(): 5 kfdp_dismountBg(): 5 kfdp_dismount(): 6 kfdp_dismountBg(): 6 NOTE: De-assigning number (2,0) from disk (/dev/mpath/mpath3p1) SUCCESS: diskgroup FLASH was created NOTE: cache registered group FLASH number=2 incarn=0x3a285a44 NOTE: cache began mount (first) of group FLASH number=2 incarn=0x3a285a44 NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: start heartbeating (grp 2) kfdp_query(): 9 kfdp_queryBg(): 9 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (first) group 2/0x3A285A44 (FLASH) * allocate domain 2, invalid = TRUE kjbdomatt send to node 1 NOTE: attached to recovery domain 2 NOTE: cache recovered group 2 to fcn 0.0 Tue Aug 16 09:02:38 2011 NOTE: LGWR attempting to mount thread 1 for diskgroup 2 NOTE: LGWR mounted thread 1 for disk group 2 NOTE: opening chunk 1 at fcn 0.0 ABA NOTE: seq=2 blk=0 NOTE: cache mounting group 2/0x3A285A44 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a285a44 Tue Aug 16 09:02:38 2011 kfdp_query(): 10 kfdp_queryBg(): 10 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 Tue Aug 16 09:02:38 2011 SUCCESS: diskgroup FLASH was mounted SUCCESS: CREATE DISKGROUP FLASH External REDUNDANCY DISK '/dev/mpath/mpath3p1' ALTER SYSTEM SET asm_diskgroups='ORAFILES','FLASH' SCOPE=BOTH SID='*'; NOTE: enlarging ACD for group 2/0x3a285a44 (FLASH) SUCCESS: ACD enlarged for group 2/0x3a285a44 (FLASH) Mon Aug 29 15:10:30 2011 Starting background process ASMB Mon Aug 29 15:10:30 2011 ASMB started with pid=20, OS id=17623 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Aug 29 15:21:52 2011 Starting background process ASMB Mon Aug 29 15:21:52 2011 ASMB started with pid=20, OS id=25487 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Aug 29 16:07:29 2011 Trace dumping is performing id=[cdmp_20110829160729] Sun Sep 18 18:27:36 2011 Shutting down instance: further logons disabled Shutting down instance (immediate) License high water mark = 14 SQL> ALTER DISKGROUP ALL DISMOUNT NOTE: cache dismounting group 1/0xC6085A3C (ORAFILES) Sun Sep 18 18:27:36 2011 kjbdomdet send to node 1 detach from dom 1, sending detach message to node 1 Sun Sep 18 18:27:37 2011 NOTE: detached from domain 1 NOTE: cache dismounted group 1/0xC6085A3C (ORAFILES) kfdp_dismount(): 11 Sun Sep 18 18:27:37 2011 kfdp_dismountBg(): 11 NOTE: De-assigning number (1,0) from disk (/dev/mpath/mpath1p1) NOTE: De-assigning number (1,1) from disk (/dev/mpath/mpath2p1) SUCCESS: diskgroup ORAFILES was dismounted NOTE: cache dismounting group 2/0x3A285A44 (FLASH) kjbdomdet send to node 1 detach from dom 2, sending detach message to node 1 NOTE: detached from domain 2 NOTE: cache dismounted group 2/0x3A285A44 (FLASH) kfdp_dismount(): 12 kfdp_dismountBg(): 12 NOTE: De-assigning number (2,0) from disk (/dev/mpath/mpath3p1) SUCCESS: diskgroup FLASH was dismounted SUCCESS: ALTER DISKGROUP ALL DISMOUNT Sun Sep 18 18:27:38 2011 Stopping background process VKTM: Sun Sep 18 18:27:40 2011 freeing rdom 2 freeing rdom 1 freeing rdom 0 Sun Sep 18 18:27:42 2011 Instance shutdown complete Sun Sep 18 18:49:29 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sun Sep 18 18:49:30 2011 PMON started with pid=2, OS id=9151 Sun Sep 18 18:49:31 2011 VKTM started with pid=3, OS id=9153 at elevated priority Sun Sep 18 18:49:31 2011 DIAG started with pid=4, OS id=9157 VKTM running at (20)ms precision Sun Sep 18 18:49:31 2011 PING started with pid=5, OS id=9159 Sun Sep 18 18:49:31 2011 PSP0 started with pid=6, OS id=9161 Sun Sep 18 18:49:31 2011 DIA0 started with pid=7, OS id=9163 Sun Sep 18 18:49:31 2011 LMON started with pid=8, OS id=9165 Sun Sep 18 18:49:31 2011 LMD0 started with pid=9, OS id=9167 Sun Sep 18 18:49:31 2011 LMS0 started with pid=10, OS id=9169 at elevated priority Sun Sep 18 18:49:31 2011 MMAN started with pid=11, OS id=9173 Sun Sep 18 18:49:31 2011 DBW0 started with pid=12, OS id=9175 Sun Sep 18 18:49:31 2011 LGWR started with pid=13, OS id=9177 Sun Sep 18 18:49:31 2011 CKPT started with pid=14, OS id=9181 Sun Sep 18 18:49:31 2011 SMON started with pid=15, OS id=9183 Sun Sep 18 18:49:31 2011 RBAL started with pid=16, OS id=9185 Sun Sep 18 18:49:31 2011 GMON started with pid=17, OS id=9187 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 8) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * allocate domain 2, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 * domain 2 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sun Sep 18 18:49:32 2011 LCK0 started with pid=18, OS id=9198 ORACLE_BASE from environment = /u01/app/oracle Sun Sep 18 18:49:33 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc6088648 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc6088648 NOTE: cache registered group FLASH number=2 incarn=0x3a288647 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a288647 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) kfdp_query(): 3 kfdp_queryBg(): 3 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0xC6088648 (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.37583076 ABA NOTE: seq=163 blk=349 NOTE: cache mounting group 1/0xC6088648 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc6088648 kfdp_query(): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A288647 (FLASH) kjbdomatt send to node 1 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 1 for diskgroup 2 NOTE: LGWR mounted thread 1 for disk group 2 NOTE: opening chunk 1 at fcn 0.13548939 ABA NOTE: seq=35 blk=10256 NOTE: cache mounting group 2/0x3A288647 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a288647 kfdp_query(): 5 kfdp_queryBg(): 5 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(): 6 kfdp_queryBg(): 6 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sun Sep 18 18:49:43 2011 Starting background process ASMB Sun Sep 18 18:49:43 2011 ASMB started with pid=20, OS id=13027 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 18 19:23:54 2011 Received detach msg from node 1 for dom 1 Sun Sep 18 19:23:55 2011 NOTE: PST enabling heartbeating (grp 1) Received detach msg from node 1 for dom 2 NOTE: PST enabling heartbeating (grp 2) Sun Sep 18 19:24:00 2011 Reconfiguration started (old inc 8, new inc 10) List of nodes: 0 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sun Sep 18 19:24:00 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 3842 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sun Sep 18 19:44:56 2011 Reconfiguration started (old inc 10, new inc 12) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sun Sep 18 19:44:56 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 3842 GCS shadows traversed, 2173 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Mon Sep 19 21:19:15 2011 SQL> create diskgroup prd01 external redundancy disk '/dev/mpath/mpath4p1' NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) NOTE: initializing header on grp 3 disk PRD01_0000 NOTE: initiating PST update: grp = 3 kfdp_update(): 7 Mon Sep 19 21:19:15 2011 kfdp_updateBg(): 7 NOTE: group PRD01: initial PST location: disk 0000 (PST copy 0) NOTE: PST update grp = 3 completed successfully NOTE: cache registered group PRD01 number=3 incarn=0xbe68864e NOTE: cache began mount (first) of group PRD01 number=3 incarn=0xbe68864e NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 Mon Sep 19 21:19:15 2011 * allocate domain 3, invalid = TRUE kjbdomatt send to node 1 Mon Sep 19 21:19:15 2011 NOTE: attached to recovery domain 3 NOTE: cache creating group 3/0xBE68864E (PRD01) NOTE: cache mounting group 3/0xBE68864E (PRD01) succeeded NOTE: allocating F1X0 on grp 3 disk PRD01_0000 NOTE: diskgroup must now be re-mounted prior to first use NOTE: cache dismounting group 3/0xBE68864E (PRD01) NOTE: lgwr not being msg'd to dismount kjbdomdet send to node 1 detach from dom 3, sending detach message to node 1 freeing rdom 3 NOTE: detached from domain 3 NOTE: cache dismounted group 3/0xBE68864E (PRD01) NOTE: cache ending mount (fail) of group PRD01 number=3 incarn=0xbe68864e kfdp_dismount(): 8 kfdp_dismountBg(): 8 kfdp_dismount(): 9 kfdp_dismountBg(): 9 NOTE: De-assigning number (3,0) from disk (/dev/mpath/mpath4p1) SUCCESS: diskgroup PRD01 was created NOTE: cache registered group PRD01 number=3 incarn=0xd3f88650 NOTE: cache began mount (first) of group PRD01 number=3 incarn=0xd3f88650 NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) NOTE: start heartbeating (grp 3) kfdp_query(): 12 kfdp_queryBg(): 12 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (first) group 3/0xD3F88650 (PRD01) * allocate domain 3, invalid = TRUE kjbdomatt send to node 1 NOTE: attached to recovery domain 3 NOTE: cache recovered group 3 to fcn 0.0 Mon Sep 19 21:19:23 2011 NOTE: LGWR attempting to mount thread 1 for diskgroup 3 NOTE: LGWR mounted thread 1 for disk group 3 NOTE: opening chunk 1 at fcn 0.0 ABA NOTE: seq=2 blk=0 NOTE: cache mounting group 3/0xD3F88650 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f88650 Mon Sep 19 21:19:23 2011 kfdp_query(): 13 kfdp_queryBg(): 13 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: create diskgroup prd01 external redundancy disk '/dev/mpath/mpath4p1' Mon Sep 19 21:19:50 2011 NOTE: enlarging ACD for group 3/0xd3f88650 (PRD01) SUCCESS: ACD enlarged for group 3/0xd3f88650 (PRD01) Mon Sep 19 22:09:02 2011 Starting background process ASMB Mon Sep 19 22:09:02 2011 ASMB started with pid=22, OS id=22913 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Sep 19 22:11:46 2011 Starting background process ASMB Mon Sep 19 22:11:46 2011 ASMB started with pid=22, OS id=29380 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Sep 19 22:27:28 2011 Starting background process ASMB Mon Sep 19 22:27:28 2011 ASMB started with pid=21, OS id=9928 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Sep 19 22:27:43 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_10077.trc Mon Sep 19 22:27:43 2011 Trace dumping is performing id=[cdmp_20110919222743] Mon Sep 19 22:31:31 2011 Starting background process ASMB Mon Sep 19 22:31:31 2011 ASMB started with pid=21, OS id=13482 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 12:09:03 2011 Starting background process ASMB Tue Sep 20 12:09:03 2011 ASMB started with pid=21, OS id=17097 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 12:17:44 2011 Starting background process ASMB Tue Sep 20 12:17:44 2011 ASMB started with pid=21, OS id=23471 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 13:39:10 2011 Starting background process ASMB Tue Sep 20 13:39:10 2011 ASMB started with pid=21, OS id=1260 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 13:47:58 2011 Starting background process ASMB Tue Sep 20 13:47:58 2011 ASMB started with pid=21, OS id=7858 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 13:52:48 2011 Starting background process ASMB Tue Sep 20 13:52:48 2011 ASMB started with pid=21, OS id=12596 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 14:09:09 2011 Starting background process ASMB Tue Sep 20 14:09:09 2011 ASMB started with pid=22, OS id=6069 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 14:29:12 2011 Starting background process ASMB Tue Sep 20 14:29:12 2011 ASMB started with pid=21, OS id=22288 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 14:29:40 2011 Starting background process ASMB Tue Sep 20 14:29:40 2011 ASMB started with pid=21, OS id=22847 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 14:35:51 2011 Starting background process ASMB Tue Sep 20 14:35:51 2011 ASMB started with pid=21, OS id=27801 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 14:41:54 2011 Starting background process ASMB Tue Sep 20 14:41:54 2011 ASMB started with pid=21, OS id=440 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 14:43:49 2011 Starting background process ASMB Tue Sep 20 14:43:49 2011 ASMB started with pid=21, OS id=2281 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 14:44:22 2011 Starting background process ASMB Tue Sep 20 14:44:22 2011 ASMB started with pid=21, OS id=2668 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 14:46:00 2011 Starting background process ASMB Tue Sep 20 14:46:00 2011 ASMB started with pid=21, OS id=3692 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 14:47:44 2011 Starting background process ASMB Tue Sep 20 14:47:44 2011 ASMB started with pid=21, OS id=5060 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 14:47:55 2011 Starting background process ASMB Tue Sep 20 14:47:55 2011 ASMB started with pid=21, OS id=5186 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 14:53:07 2011 Starting background process ASMB Tue Sep 20 14:53:07 2011 ASMB started with pid=21, OS id=9383 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 14:58:50 2011 Starting background process ASMB Tue Sep 20 14:58:50 2011 ASMB started with pid=22, OS id=14229 Tue Sep 20 14:58:50 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_13814.trc Tue Sep 20 14:58:50 2011 Trace dumping is performing id=[cdmp_20110920145850] NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 14:59:23 2011 Starting background process ASMB Tue Sep 20 14:59:23 2011 ASMB started with pid=21, OS id=14636 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 15:01:30 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_16271.trc Tue Sep 20 15:01:30 2011 Trace dumping is performing id=[cdmp_20110920150130] Tue Sep 20 15:01:31 2011 Starting background process ASMB Tue Sep 20 15:01:31 2011 ASMB started with pid=21, OS id=16720 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 15:03:24 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_18107.trc Tue Sep 20 15:03:24 2011 Trace dumping is performing id=[cdmp_20110920150324] Tue Sep 20 15:03:24 2011 Starting background process ASMB Tue Sep 20 15:03:24 2011 ASMB started with pid=21, OS id=23691 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 15:11:29 2011 Starting background process ASMB Tue Sep 20 15:11:29 2011 ASMB started with pid=22, OS id=4708 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 15:11:59 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_482.trc Tue Sep 20 15:11:59 2011 Trace dumping is performing id=[cdmp_20110920151159] Tue Sep 20 15:14:34 2011 Starting background process ASMB Tue Sep 20 15:14:34 2011 ASMB started with pid=21, OS id=7816 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 15:16:05 2011 Starting background process ASMB Tue Sep 20 15:16:05 2011 ASMB started with pid=21, OS id=8843 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 15:16:12 2011 Starting background process ASMB Tue Sep 20 15:16:12 2011 ASMB started with pid=21, OS id=9029 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 15:16:24 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_9192.trc Tue Sep 20 15:16:24 2011 Trace dumping is performing id=[cdmp_20110920151624] Tue Sep 20 15:18:42 2011 Starting background process ASMB Tue Sep 20 15:18:42 2011 ASMB started with pid=21, OS id=11306 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 15:18:50 2011 Starting background process ASMB Tue Sep 20 15:18:50 2011 ASMB started with pid=21, OS id=11368 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 15:19:02 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_11544.trc Tue Sep 20 15:19:02 2011 Trace dumping is performing id=[cdmp_20110920151902] Tue Sep 20 15:21:21 2011 Starting background process ASMB Tue Sep 20 15:21:21 2011 ASMB started with pid=21, OS id=13510 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 15:21:33 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_13699.trc Tue Sep 20 15:21:33 2011 Trace dumping is performing id=[cdmp_20110920152133] Tue Sep 20 15:23:15 2011 Starting background process ASMB Tue Sep 20 15:23:15 2011 ASMB started with pid=21, OS id=15204 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 15:23:32 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_15696.trc Tue Sep 20 15:23:32 2011 Trace dumping is performing id=[cdmp_20110920152332] Tue Sep 20 15:24:56 2011 Starting background process ASMB Tue Sep 20 15:24:57 2011 ASMB started with pid=21, OS id=16794 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 15:25:02 2011 Starting background process ASMB Tue Sep 20 15:25:02 2011 ASMB started with pid=21, OS id=16878 NOTE: ASMB process exiting due to lack of ASM file activity for 6 seconds Tue Sep 20 15:25:14 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_17024.trc Tue Sep 20 15:25:14 2011 Trace dumping is performing id=[cdmp_20110920152514] Tue Sep 20 15:29:52 2011 Starting background process ASMB Tue Sep 20 15:29:52 2011 ASMB started with pid=21, OS id=20876 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 15:30:09 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_21047.trc Tue Sep 20 15:30:09 2011 Trace dumping is performing id=[cdmp_20110920153009] Tue Sep 20 15:35:06 2011 Starting background process ASMB Tue Sep 20 15:35:06 2011 ASMB started with pid=21, OS id=25198 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 15:35:19 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_25390.trc Tue Sep 20 15:35:19 2011 Trace dumping is performing id=[cdmp_20110920153519] Tue Sep 20 15:52:12 2011 Starting background process ASMB Tue Sep 20 15:52:12 2011 ASMB started with pid=21, OS id=7086 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 15:52:24 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_7236.trc Tue Sep 20 15:52:24 2011 Trace dumping is performing id=[cdmp_20110920155224] Tue Sep 20 15:57:22 2011 Starting background process ASMB Tue Sep 20 15:57:22 2011 ASMB started with pid=21, OS id=11253 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 15:57:34 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_11399.trc Tue Sep 20 15:57:34 2011 Trace dumping is performing id=[cdmp_20110920155734] Tue Sep 20 16:03:04 2011 Starting background process ASMB Tue Sep 20 16:03:04 2011 ASMB started with pid=21, OS id=15865 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:03:17 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_16052.trc Tue Sep 20 16:03:17 2011 Trace dumping is performing id=[cdmp_20110920160317] Tue Sep 20 16:07:56 2011 Starting background process ASMB Tue Sep 20 16:07:56 2011 ASMB started with pid=21, OS id=639 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:08:05 2011 Starting background process ASMB Tue Sep 20 16:08:05 2011 ASMB started with pid=21, OS id=718 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:08:17 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_869.trc Tue Sep 20 16:08:17 2011 Trace dumping is performing id=[cdmp_20110920160817] Tue Sep 20 16:12:57 2011 Starting background process ASMB Tue Sep 20 16:12:57 2011 ASMB started with pid=21, OS id=4934 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:13:08 2011 Starting background process ASMB Tue Sep 20 16:13:08 2011 ASMB started with pid=21, OS id=5053 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:15:43 2011 Starting background process ASMB Tue Sep 20 16:15:43 2011 ASMB started with pid=21, OS id=7406 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:16:07 2011 Starting background process ASMB Tue Sep 20 16:16:07 2011 ASMB started with pid=22, OS id=8272 Tue Sep 20 16:16:07 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_7719.trc Tue Sep 20 16:16:07 2011 Trace dumping is performing id=[cdmp_20110920161607] NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:16:28 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_8435.trc Tue Sep 20 16:16:29 2011 Trace dumping is performing id=[cdmp_20110920161629] Tue Sep 20 16:18:17 2011 Starting background process ASMB Tue Sep 20 16:18:17 2011 ASMB started with pid=21, OS id=10586 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:18:33 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_10866.trc Tue Sep 20 16:18:33 2011 Trace dumping is performing id=[cdmp_20110920161833] Tue Sep 20 16:19:42 2011 Starting background process ASMB Tue Sep 20 16:19:42 2011 ASMB started with pid=21, OS id=11885 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:19:54 2011 Starting background process ASMB Tue Sep 20 16:19:54 2011 ASMB started with pid=21, OS id=12090 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:20:54 2011 Trace dumping is performing id=[cdmp_20110920162054] Tue Sep 20 16:22:49 2011 Trace dumping is performing id=[cdmp_20110920162249] Tue Sep 20 16:25:10 2011 Trace dumping is performing id=[cdmp_20110920162510] Tue Sep 20 16:25:14 2011 Starting background process ASMB Tue Sep 20 16:25:14 2011 ASMB started with pid=22, OS id=17085 Tue Sep 20 16:25:14 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12233.trc Trace dumping is performing id=[cdmp_20110920162515] NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:30:35 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. Tue Sep 20 16:30:36 2011 Starting background process ASMB NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_17239.trc Tue Sep 20 16:30:36 2011 ASMB started with pid=22, OS id=21661 Tue Sep 20 16:30:36 2011 Trace dumping is performing id=[cdmp_20110920163036] NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:34:45 2011 SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' Tue Sep 20 16:35:16 2011 Starting background process ASMB Tue Sep 20 16:35:16 2011 ASMB started with pid=22, OS id=25569 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:35:25 2011 Starting background process ASMB Tue Sep 20 16:35:25 2011 ASMB started with pid=22, OS id=25671 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:35:37 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_25818.trc Tue Sep 20 16:35:37 2011 Trace dumping is performing id=[cdmp_20110920163537] Tue Sep 20 16:40:35 2011 Starting background process ASMB Tue Sep 20 16:40:35 2011 ASMB started with pid=22, OS id=29941 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:40:47 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_30087.trc Tue Sep 20 16:40:47 2011 Trace dumping is performing id=[cdmp_20110920164047] Tue Sep 20 16:48:48 2011 SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' Tue Sep 20 16:49:20 2011 Starting background process ASMB Tue Sep 20 16:49:20 2011 ASMB started with pid=22, OS id=4289 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:50:28 2011 Trace dumping is performing id=[cdmp_20110920165028] Tue Sep 20 16:51:37 2011 Trace dumping is performing id=[cdmp_20110920165137] Tue Sep 20 16:54:52 2011 SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' Tue Sep 20 16:55:08 2011 Starting background process ASMB Tue Sep 20 16:55:08 2011 ASMB started with pid=22, OS id=9288 Tue Sep 20 16:55:15 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_9505.trc Tue Sep 20 16:55:15 2011 Trace dumping is performing id=[cdmp_20110920165515] NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:55:45 2011 SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' Tue Sep 20 16:57:01 2011 Starting background process ASMB Tue Sep 20 16:57:01 2011 ASMB started with pid=22, OS id=10770 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 16:57:17 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_11120.trc Tue Sep 20 16:57:17 2011 Trace dumping is performing id=[cdmp_20110920165717] Tue Sep 20 17:02:15 2011 Starting background process ASMB Tue Sep 20 17:02:15 2011 ASMB started with pid=22, OS id=15394 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 17:02:27 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_15548.trc Tue Sep 20 17:02:27 2011 Trace dumping is performing id=[cdmp_20110920170227] Tue Sep 20 17:45:39 2011 SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' Tue Sep 20 17:46:18 2011 Starting background process ASMB Tue Sep 20 17:46:18 2011 ASMB started with pid=22, OS id=30493 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 17:49:46 2011 Starting background process ASMB Tue Sep 20 17:49:46 2011 ASMB started with pid=23, OS id=1021 Tue Sep 20 17:49:46 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_30715.trc Tue Sep 20 17:49:47 2011 Trace dumping is performing id=[cdmp_20110920174947] NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 17:50:10 2011 Starting background process ASMB Tue Sep 20 17:50:10 2011 ASMB started with pid=23, OS id=1937 Tue Sep 20 17:50:10 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_1169.trc Tue Sep 20 17:50:10 2011 Trace dumping is performing id=[cdmp_20110920175010] NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 17:51:03 2011 Starting background process ASMB Tue Sep 20 17:51:03 2011 ASMB started with pid=28, OS id=2976 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 17:52:20 2011 Starting background process ASMB Tue Sep 20 17:52:20 2011 ASMB started with pid=22, OS id=4163 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 17:54:25 2011 Trace dumping is performing id=[cdmp_20110920175425] Tue Sep 20 17:57:44 2011 Starting background process ASMB Tue Sep 20 17:57:44 2011 ASMB started with pid=23, OS id=8717 Tue Sep 20 17:57:44 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_4382.trc Tue Sep 20 17:57:45 2011 Trace dumping is performing id=[cdmp_20110920175745] NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 18:02:06 2011 SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' Tue Sep 20 18:03:21 2011 Starting background process ASMB Tue Sep 20 18:03:21 2011 ASMB started with pid=22, OS id=17312 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 18:04:32 2011 Trace dumping is performing id=[cdmp_20110920180431] Tue Sep 20 18:18:25 2011 Trace dumping is performing id=[cdmp_20110920181824] Tue Sep 20 18:23:36 2011 Trace dumping is performing id=[cdmp_20110920182336] Tue Sep 20 18:24:19 2011 Starting background process ASMB Tue Sep 20 18:24:19 2011 ASMB started with pid=21, OS id=11740 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 18:27:15 2011 Trace dumping is performing id=[cdmp_20110920182715] Tue Sep 20 18:27:27 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_11909.trc Tue Sep 20 18:27:27 2011 Trace dumping is performing id=[cdmp_20110920182727] Tue Sep 20 18:27:28 2011 Starting background process ASMB Tue Sep 20 18:27:28 2011 ASMB started with pid=21, OS id=14536 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 18:27:52 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_14742.trc Tue Sep 20 18:27:52 2011 Trace dumping is performing id=[cdmp_20110920182752] Tue Sep 20 18:32:27 2011 Trace dumping is performing id=[cdmp_20110920183227] Tue Sep 20 18:36:38 2011 Starting background process ASMB Tue Sep 20 18:36:38 2011 ASMB started with pid=21, OS id=21976 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 18:37:02 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_22193.trc Tue Sep 20 18:37:02 2011 Trace dumping is performing id=[cdmp_20110920183702] Tue Sep 20 18:37:52 2011 Starting background process ASMB Tue Sep 20 18:37:52 2011 ASMB started with pid=21, OS id=23304 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 18:38:14 2011 Starting background process ASMB Tue Sep 20 18:38:14 2011 ASMB started with pid=22, OS id=24007 Tue Sep 20 18:38:14 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_23504.trc Tue Sep 20 18:38:14 2011 Trace dumping is performing id=[cdmp_20110920183814] NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 18:38:38 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_24494.trc Tue Sep 20 18:38:38 2011 Trace dumping is performing id=[cdmp_20110920183838] Tue Sep 20 18:54:29 2011 SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+prd01/prd01/spfileprd01.ora' Tue Sep 20 18:54:41 2011 Starting background process ASMB Tue Sep 20 18:54:41 2011 ASMB started with pid=22, OS id=4701 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 18:55:05 2011 Starting background process ASMB Tue Sep 20 18:55:05 2011 ASMB started with pid=23, OS id=5251 Tue Sep 20 18:55:05 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_4914.trc Tue Sep 20 18:55:05 2011 Trace dumping is performing id=[cdmp_20110920185505] NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 18:55:19 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_5397.trc Tue Sep 20 18:55:20 2011 Trace dumping is performing id=[cdmp_20110920185520] Tue Sep 20 19:07:48 2011 Starting background process ASMB Tue Sep 20 19:07:48 2011 ASMB started with pid=22, OS id=28275 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Sep 20 19:09:11 2011 Trace dumping is performing id=[cdmp_20110920190911] Tue Sep 20 19:14:19 2011 Trace dumping is performing id=[cdmp_20110920191419] Sat Sep 24 17:53:33 2011 SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo01a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo01a.dbf' Sat Sep 24 17:53:44 2011 SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo02a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo02a.dbf' Sat Sep 24 17:54:14 2011 SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo03a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo03a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo04a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo04a.dbf' Sat Sep 24 17:54:25 2011 SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo05a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo05a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo06a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo06a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo07a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo07a.dbf' Sat Sep 24 17:54:35 2011 SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo08a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo08a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo09a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo09a.dbf' Sat Sep 24 17:54:51 2011 SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo10a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo10a.dbf' Sat Sep 24 17:55:09 2011 SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo11a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo11a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo12a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo12a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo13a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo13a.dbf' Sat Sep 24 17:55:19 2011 SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo14a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo14a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo15a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo15a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo16a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo16a.dbf' Sat Sep 24 17:55:31 2011 SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo17a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo17a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo18a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo18a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo19a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo19a.dbf' Sat Sep 24 17:55:42 2011 SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo20a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo20a.dbf' Sat Sep 24 17:55:56 2011 SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo21a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo21a.dbf' Sat Sep 24 17:56:06 2011 SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo22a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo22a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo23a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo23a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo24a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo24a.dbf' Sat Sep 24 17:56:18 2011 SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo25a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo25a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo26a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo26a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo27a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo27a.dbf' Sat Sep 24 17:56:29 2011 SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo28a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo28a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo29a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo29a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo30a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo30a.dbf' Sat Sep 24 17:56:47 2011 SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo31a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo31a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo32a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo32a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo33a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo33a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo34a.dbf' Sat Sep 24 17:56:59 2011 SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo34a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo35a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo35a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo36a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo36a.dbf' Sat Sep 24 17:57:10 2011 SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo37a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo37a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo38a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo38a.dbf' Sat Sep 24 17:57:21 2011 SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo39a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo39a.dbf' SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo40a.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/DWH1/redo40a.dbf' Sun Sep 25 18:00:52 2011 Starting background process ASMB Sun Sep 25 18:00:52 2011 ASMB started with pid=22, OS id=23127 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 18:01:06 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_23266.trc Sun Sep 25 18:01:06 2011 Trace dumping is performing id=[cdmp_20110925180106] Sun Sep 25 18:32:27 2011 Shutting down instance: further logons disabled Shutting down instance (immediate) License high water mark = 13 SQL> ALTER DISKGROUP ALL DISMOUNT NOTE: cache dismounting group 1/0xC6088648 (ORAFILES) Sun Sep 25 18:32:29 2011 kjbdomdet send to node 1 detach from dom 1, sending detach message to node 1 Sun Sep 25 18:32:29 2011 NOTE: detached from domain 1 NOTE: cache dismounted group 1/0xC6088648 (ORAFILES) kfdp_dismount(): 14 Sun Sep 25 18:32:29 2011 kfdp_dismountBg(): 14 NOTE: De-assigning number (1,0) from disk (/dev/mpath/mpath1p1) NOTE: De-assigning number (1,1) from disk (/dev/mpath/mpath2p1) SUCCESS: diskgroup ORAFILES was dismounted NOTE: cache dismounting group 2/0x3A288647 (FLASH) kjbdomdet send to node 1 detach from dom 2, sending detach message to node 1 NOTE: detached from domain 2 NOTE: cache dismounted group 2/0x3A288647 (FLASH) kfdp_dismount(): 15 kfdp_dismountBg(): 15 NOTE: De-assigning number (2,0) from disk (/dev/mpath/mpath3p1) SUCCESS: diskgroup FLASH was dismounted NOTE: cache dismounting group 3/0xD3F88650 (PRD01) kjbdomdet send to node 1 detach from dom 3, sending detach message to node 1 NOTE: detached from domain 3 NOTE: cache dismounted group 3/0xD3F88650 (PRD01) kfdp_dismount(): 16 kfdp_dismountBg(): 16 NOTE: De-assigning number (3,0) from disk (/dev/mpath/mpath4p1) SUCCESS: diskgroup PRD01 was dismounted SUCCESS: ALTER DISKGROUP ALL DISMOUNT Sun Sep 25 18:32:31 2011 Stopping background process VKTM: Sun Sep 25 18:32:33 2011 freeing rdom 3 freeing rdom 2 freeing rdom 1 freeing rdom 0 Sun Sep 25 18:32:34 2011 Instance shutdown complete Sun Sep 25 18:39:40 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sun Sep 25 18:39:40 2011 PMON started with pid=2, OS id=8167 Sun Sep 25 18:39:40 2011 VKTM started with pid=3, OS id=8169 at elevated priority VKTM running at (20)ms precision Sun Sep 25 18:39:40 2011 DIAG started with pid=4, OS id=8173 Sun Sep 25 18:39:40 2011 PING started with pid=5, OS id=8175 Sun Sep 25 18:39:40 2011 PSP0 started with pid=6, OS id=8177 Sun Sep 25 18:39:40 2011 DIA0 started with pid=7, OS id=8179 Sun Sep 25 18:39:40 2011 LMON started with pid=8, OS id=8181 Sun Sep 25 18:39:40 2011 LMD0 started with pid=9, OS id=8183 Sun Sep 25 18:39:40 2011 LMS0 started with pid=10, OS id=8185 at elevated priority Sun Sep 25 18:39:40 2011 MMAN started with pid=11, OS id=8189 Sun Sep 25 18:39:40 2011 DBW0 started with pid=12, OS id=8191 Sun Sep 25 18:39:40 2011 LGWR started with pid=13, OS id=8193 Sun Sep 25 18:39:40 2011 CKPT started with pid=14, OS id=8195 Sun Sep 25 18:39:41 2011 SMON started with pid=15, OS id=8197 Sun Sep 25 18:39:41 2011 RBAL started with pid=16, OS id=8199 Sun Sep 25 18:39:41 2011 GMON started with pid=17, OS id=8201 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 16) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * allocate domain 2, invalid = TRUE * allocate domain 3, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 * domain 2 valid = 1 according to instance 1 * domain 3 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sun Sep 25 18:39:41 2011 LCK0 started with pid=18, OS id=8212 ORACLE_BASE from environment = /u01/app/oracle Sun Sep 25 18:39:42 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc6025592 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc6025592 NOTE: cache registered group FLASH number=2 incarn=0x3a225591 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a225591 NOTE: cache registered group PRD01 number=3 incarn=0xd3f25593 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f25593 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0xC6025592 (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.37637288 ABA NOTE: seq=164 blk=3272 NOTE: cache mounting group 1/0xC6025592 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc6025592 kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A225591 (FLASH) kjbdomatt send to node 1 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 1 for diskgroup 2 NOTE: LGWR mounted thread 1 for disk group 2 NOTE: opening chunk 1 at fcn 0.16562035 ABA NOTE: seq=44 blk=3158 NOTE: cache mounting group 2/0x3A225591 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a225591 kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F25593 (PRD01) kjbdomatt send to node 1 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 1 for diskgroup 3 NOTE: LGWR mounted thread 1 for disk group 3 NOTE: opening chunk 1 at fcn 0.10515 ABA NOTE: seq=3 blk=1586 NOTE: cache mounting group 3/0xD3F25593 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f25593 kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sun Sep 25 18:41:30 2011 Received detach msg from node 1 for dom 1 Sun Sep 25 18:41:30 2011 NOTE: PST enabling heartbeating (grp 1) Received detach msg from node 1 for dom 2 NOTE: PST enabling heartbeating (grp 2) Received detach msg from node 1 for dom 3 NOTE: PST enabling heartbeating (grp 3) Sun Sep 25 18:41:34 2011 Reconfiguration started (old inc 16, new inc 18) List of nodes: 0 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sun Sep 25 18:41:34 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 28 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sun Sep 25 18:44:33 2011 Reconfiguration started (old inc 18, new inc 20) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sun Sep 25 18:44:33 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 28 GCS shadows traversed, 28 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sun Sep 25 18:45:02 2011 Shutting down instance: further logons disabled Shutting down instance (immediate) License high water mark = 1 SQL> ALTER DISKGROUP ALL DISMOUNT NOTE: cache dismounting group 1/0xC6025592 (ORAFILES) Sun Sep 25 18:45:02 2011 kjbdomdet send to node 1 detach from dom 1, sending detach message to node 1 Sun Sep 25 18:45:02 2011 NOTE: detached from domain 1 NOTE: cache dismounted group 1/0xC6025592 (ORAFILES) kfdp_dismount(): 10 Sun Sep 25 18:45:02 2011 kfdp_dismountBg(): 10 NOTE: De-assigning number (1,0) from disk (/dev/mpath/mpath1p1) NOTE: De-assigning number (1,1) from disk (/dev/mpath/mpath2p1) SUCCESS: diskgroup ORAFILES was dismounted NOTE: cache dismounting group 2/0x3A225591 (FLASH) kjbdomdet send to node 1 detach from dom 2, sending detach message to node 1 NOTE: detached from domain 2 NOTE: cache dismounted group 2/0x3A225591 (FLASH) kfdp_dismount(): 11 kfdp_dismountBg(): 11 NOTE: De-assigning number (2,0) from disk (/dev/mpath/mpath3p1) SUCCESS: diskgroup FLASH was dismounted NOTE: cache dismounting group 3/0xD3F25593 (PRD01) kjbdomdet send to node 1 detach from dom 3, sending detach message to node 1 NOTE: detached from domain 3 NOTE: cache dismounted group 3/0xD3F25593 (PRD01) kfdp_dismount(): 12 kfdp_dismountBg(): 12 NOTE: De-assigning number (3,0) from disk (/dev/mpath/mpath4p1) SUCCESS: diskgroup PRD01 was dismounted SUCCESS: ALTER DISKGROUP ALL DISMOUNT Sun Sep 25 18:45:04 2011 Stopping background process VKTM: Sun Sep 25 18:45:05 2011 freeing rdom 3 freeing rdom 2 freeing rdom 1 freeing rdom 0 Sun Sep 25 18:45:07 2011 Instance shutdown complete Sun Sep 25 18:47:25 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sun Sep 25 18:47:25 2011 PMON started with pid=2, OS id=14852 Sun Sep 25 18:47:25 2011 VKTM started with pid=3, OS id=14854 at elevated priority VKTM running at (20)ms precision Sun Sep 25 18:47:26 2011 DIAG started with pid=4, OS id=14858 Sun Sep 25 18:47:26 2011 PING started with pid=5, OS id=14860 Sun Sep 25 18:47:26 2011 PSP0 started with pid=6, OS id=14862 Sun Sep 25 18:47:26 2011 DIA0 started with pid=7, OS id=14864 Sun Sep 25 18:47:26 2011 LMON started with pid=8, OS id=14866 Sun Sep 25 18:47:26 2011 LMD0 started with pid=9, OS id=14868 Sun Sep 25 18:47:26 2011 LMS0 started with pid=10, OS id=14870 at elevated priority Sun Sep 25 18:47:26 2011 MMAN started with pid=11, OS id=14874 Sun Sep 25 18:47:26 2011 DBW0 started with pid=12, OS id=14876 Sun Sep 25 18:47:26 2011 LGWR started with pid=13, OS id=14878 Sun Sep 25 18:47:26 2011 CKPT started with pid=14, OS id=14880 Sun Sep 25 18:47:26 2011 SMON started with pid=15, OS id=14882 Sun Sep 25 18:47:26 2011 RBAL started with pid=16, OS id=14884 Sun Sep 25 18:47:26 2011 GMON started with pid=17, OS id=14886 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 24) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * allocate domain 2, invalid = TRUE * allocate domain 3, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 * domain 2 valid = 1 according to instance 1 * domain 3 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sun Sep 25 18:47:26 2011 LCK0 started with pid=18, OS id=14898 ORACLE_BASE from environment = /u01/app/oracle Sun Sep 25 18:47:27 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc6030b43 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc6030b43 NOTE: cache registered group FLASH number=2 incarn=0x3a230b42 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a230b42 NOTE: cache registered group PRD01 number=3 incarn=0xd3f30b44 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f30b44 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0xC6030B43 (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.37637288 ABA NOTE: seq=165 blk=3272 NOTE: cache mounting group 1/0xC6030B43 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc6030b43 kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A230B42 (FLASH) kjbdomatt send to node 1 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 1 for diskgroup 2 NOTE: LGWR mounted thread 1 for disk group 2 NOTE: opening chunk 1 at fcn 0.16562035 ABA NOTE: seq=45 blk=3158 NOTE: cache mounting group 2/0x3A230B42 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a230b42 kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F30B44 (PRD01) kjbdomatt send to node 1 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 1 for diskgroup 3 NOTE: LGWR mounted thread 1 for disk group 3 NOTE: opening chunk 1 at fcn 0.10515 ABA NOTE: seq=4 blk=1586 NOTE: cache mounting group 3/0xD3F30B44 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f30b44 kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sun Sep 25 18:47:57 2011 Received detach msg from node 1 for dom 1 Sun Sep 25 18:47:57 2011 NOTE: PST enabling heartbeating (grp 1) Received detach msg from node 1 for dom 2 NOTE: PST enabling heartbeating (grp 2) Received detach msg from node 1 for dom 3 NOTE: PST enabling heartbeating (grp 3) Sun Sep 25 18:48:00 2011 Reconfiguration started (old inc 24, new inc 26) List of nodes: 0 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sun Sep 25 18:48:00 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 28 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sun Sep 25 18:49:17 2011 Reconfiguration started (old inc 26, new inc 28) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sun Sep 25 18:49:18 2011 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 28 GCS shadows traversed, 28 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sun Sep 25 18:57:22 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_23197.trc Sun Sep 25 18:57:22 2011 Trace dumping is performing id=[cdmp_20110925185722] Sun Sep 25 19:02:22 2011 Starting background process ASMB Sun Sep 25 19:02:22 2011 ASMB started with pid=21, OS id=28136 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 19:06:12 2011 Starting background process ASMB Sun Sep 25 19:06:12 2011 ASMB started with pid=21, OS id=18902 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 19:25:25 2011 Starting background process ASMB Sun Sep 25 19:25:25 2011 ASMB started with pid=20, OS id=2237 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 19:32:35 2011 Starting background process ASMB Sun Sep 25 19:32:35 2011 ASMB started with pid=20, OS id=8445 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 19:37:58 2011 Starting background process ASMB Sun Sep 25 19:37:58 2011 ASMB started with pid=20, OS id=12971 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 19:40:24 2011 Starting background process ASMB Sun Sep 25 19:40:24 2011 ASMB started with pid=20, OS id=18240 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 19:41:08 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_18352.trc Sun Sep 25 19:41:08 2011 Trace dumping is performing id=[cdmp_20110925194108] Sun Sep 25 20:05:55 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_716.trc Sun Sep 25 20:05:55 2011 Trace dumping is performing id=[cdmp_20110925200555] Sun Sep 25 20:46:30 2011 Starting background process ASMB Sun Sep 25 20:46:30 2011 ASMB started with pid=22, OS id=20524 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 20:48:55 2011 Starting background process ASMB Sun Sep 25 20:48:55 2011 ASMB started with pid=22, OS id=26932 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 20:51:17 2011 Starting background process ASMB Sun Sep 25 20:51:17 2011 ASMB started with pid=20, OS id=29416 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 20:53:34 2011 Starting background process ASMB Sun Sep 25 20:53:34 2011 ASMB started with pid=20, OS id=32687 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 20:55:26 2011 Starting background process ASMB Sun Sep 25 20:55:26 2011 ASMB started with pid=20, OS id=1806 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 20:56:12 2011 Starting background process ASMB Sun Sep 25 20:56:12 2011 ASMB started with pid=20, OS id=2404 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 20:56:50 2011 Starting background process ASMB Sun Sep 25 20:56:50 2011 ASMB started with pid=20, OS id=3150 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 20:57:31 2011 Starting background process ASMB Sun Sep 25 20:57:31 2011 ASMB started with pid=20, OS id=3727 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 20:57:55 2011 Starting background process ASMB Sun Sep 25 20:57:55 2011 ASMB started with pid=20, OS id=4469 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 20:58:22 2011 Starting background process ASMB Sun Sep 25 20:58:22 2011 ASMB started with pid=20, OS id=4843 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 20:58:47 2011 Starting background process ASMB Sun Sep 25 20:58:47 2011 ASMB started with pid=20, OS id=5340 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 20:59:48 2011 Starting background process ASMB Sun Sep 25 20:59:48 2011 ASMB started with pid=20, OS id=6110 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 21:00:33 2011 Starting background process ASMB Sun Sep 25 21:00:33 2011 ASMB started with pid=20, OS id=6902 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 21:06:42 2011 Starting background process ASMB Sun Sep 25 21:06:42 2011 ASMB started with pid=20, OS id=25240 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 21:07:29 2011 Starting background process ASMB Sun Sep 25 21:07:29 2011 ASMB started with pid=20, OS id=25822 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 21:08:07 2011 Starting background process ASMB Sun Sep 25 21:08:07 2011 ASMB started with pid=20, OS id=26657 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Sep 25 21:36:21 2011 Starting background process ASMB Sun Sep 25 21:36:21 2011 ASMB started with pid=21, OS id=16823 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Sep 26 11:08:23 2011 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26790.trc Mon Sep 26 11:08:23 2011 Trace dumping is performing id=[cdmp_20110926110823] Mon Sep 26 11:11:53 2011 ERROR: Instance termination initiated by instance 2 with reason 1. ERROR: This comes as a down reconfig event from the cluster manager. ERROR: Please check instance 2's alert and LMON trace file for detail. ERROR: Please also examine the CSS log files. Mon Sep 26 11:17:37 2011 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Mon Sep 26 11:17:39 2011 PMON started with pid=2, OS id=8949 Mon Sep 26 11:17:39 2011 VKTM started with pid=3, OS id=8951 at elevated priority VKTM running at (20)ms precision Mon Sep 26 11:17:39 2011 DIAG started with pid=4, OS id=8955 Mon Sep 26 11:17:39 2011 PING started with pid=5, OS id=8957 Mon Sep 26 11:17:39 2011 PSP0 started with pid=6, OS id=8959 Mon Sep 26 11:17:39 2011 DIA0 started with pid=7, OS id=8961 Mon Sep 26 11:17:39 2011 LMON started with pid=8, OS id=8967 Mon Sep 26 11:17:39 2011 LMD0 started with pid=9, OS id=8969 Mon Sep 26 11:17:39 2011 LMS0 started with pid=10, OS id=8987 at elevated priority Mon Sep 26 11:17:39 2011 MMAN started with pid=11, OS id=8994 Mon Sep 26 11:17:39 2011 DBW0 started with pid=12, OS id=8996 Mon Sep 26 11:17:39 2011 LGWR started with pid=13, OS id=9004 Mon Sep 26 11:17:39 2011 CKPT started with pid=14, OS id=9006 Mon Sep 26 11:17:39 2011 SMON started with pid=15, OS id=9010 Mon Sep 26 11:17:39 2011 RBAL started with pid=16, OS id=9012 Mon Sep 26 11:17:39 2011 GMON started with pid=17, OS id=9014 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 32) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * allocate domain 2, invalid = TRUE * allocate domain 3, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 * domain 2 valid = 1 according to instance 1 * domain 3 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Mon Sep 26 11:17:40 2011 LCK0 started with pid=18, OS id=9064 ORACLE_BASE from environment = /u01/app/oracle Mon Sep 26 11:17:41 2011 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc6086872 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc6086872 NOTE: cache registered group FLASH number=2 incarn=0x3a286871 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a286871 NOTE: cache registered group PRD01 number=3 incarn=0xd3f86873 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f86873 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache mounting (not first) group 1/0xC6086872 (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.37637699 ABA NOTE: seq=166 blk=3436 NOTE: cache mounting group 1/0xC6086872 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc6086872 kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A286871 (FLASH) kjbdomatt send to node 1 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 1 for diskgroup 2 NOTE: LGWR mounted thread 1 for disk group 2 NOTE: opening chunk 1 at fcn 0.16788155 ABA NOTE: seq=46 blk=9562 NOTE: cache mounting group 2/0x3A286871 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a286871 kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F86873 (PRD01) kjbdomatt send to node 1 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 1 for diskgroup 3 NOTE: LGWR mounted thread 1 for disk group 3 NOTE: opening chunk 1 at fcn 0.231570 ABA NOTE: seq=6 blk=522 NOTE: cache mounting group 3/0xD3F86873 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f86873 kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Mon Sep 26 11:17:51 2011 Starting background process ASMB Mon Sep 26 11:17:51 2011 ASMB started with pid=20, OS id=9645 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Sep 26 11:18:41 2011 Starting background process ASMB Mon Sep 26 11:18:42 2011 ASMB started with pid=27, OS id=11392 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Sep 26 11:22:49 2011 Starting background process ASMB Mon Sep 26 11:22:49 2011 ASMB started with pid=20, OS id=15213 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Sep 26 11:23:28 2011 Starting background process ASMB Mon Sep 26 11:23:28 2011 ASMB started with pid=20, OS id=16402 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Sep 26 11:27:10 2011 Trace dumping is performing id=[cdmp_20110926112709] Mon Sep 26 11:33:47 2011 Trace dumping is performing id=[cdmp_20110926113347] Mon Sep 26 11:35:39 2011 Trace dumping is performing id=[cdmp_20110926113539] Mon Sep 26 11:40:20 2011 Starting background process ASMB Mon Sep 26 11:40:20 2011 ASMB started with pid=20, OS id=30006 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue Oct 18 11:48:43 2011 SQL> alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/dwh1/temp03.dbf' SUCCESS: alter diskgroup /*ASMCMD*/ "ORAFILES" drop file '+ORAFILES/dwh1/temp03.dbf' Thu Dec 08 15:54:54 2011 SQL> alter diskgroup ORAFILES add disk '/dev/mpath/mpath5p1' WARNING: Deprecated privilege SYSDBA for command 'ALTER DISKGROUP ADD/DROP/REBALANCE' NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) NOTE: requesting all-instance membership refresh for group=1 NOTE: initializing header on grp 1 disk ORAFILES_0002 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: requesting all-instance disk validation for group=1 Thu Dec 08 15:54:56 2011 NOTE: disk validation pending for group 1/0xc6086872 (ORAFILES) Thu Dec 08 15:54:59 2011 WARNING: RDBMS client (dwh11:dwh1°) with version (10.2.0.5.0) is not compatible with a pending operation SUCCESS: validated disks for 1/0xc6086872 (ORAFILES) NOTE: initiating PST update: grp = 1 kfdp_update(): 10 Thu Dec 08 15:54:59 2011 kfdp_updateBg(): 10 NOTE: PST update grp = 1 completed successfully WARNING: RDBMS client (dwh11:dwh1°) with version (10.2.0.5.0) is not compatible with a pending operation NOTE: membership refresh pending for group 1/0xc6086872 (ORAFILES) kfdp_query(ORAFILES): 11 kfdp_queryBg(): 11 kfdp_query(ORAFILES): 12 kfdp_queryBg(): 12 SUCCESS: refreshed membership for 1/0xc6086872 (ORAFILES) Thu Dec 08 15:55:04 2011 SUCCESS: alter diskgroup ORAFILES add disk '/dev/mpath/mpath5p1' NOTE: starting rebalance of group 1/0xc6086872 (ORAFILES) at power 1 Starting background process ARB0 Thu Dec 08 15:55:05 2011 ARB0 started with pid=23, OS id=28905 NOTE: assigning ARB0 to group 1/0xc6086872 (ORAFILES) Thu Dec 08 20:24:22 2011 NOTE: stopping process ARB0 SUCCESS: rebalance completed for group 1/0xc6086872 (ORAFILES) Thu Dec 08 20:24:25 2011 NOTE: requesting all-instance membership refresh for group=1 NOTE: initiating PST update: grp = 1 kfdp_update(): 13 Thu Dec 08 20:24:32 2011 kfdp_updateBg(): 13 NOTE: PST update grp = 1 completed successfully NOTE: initiating PST update: grp = 1 kfdp_update(): 14 kfdp_updateBg(): 14 NOTE: PST update grp = 1 completed successfully Thu Dec 08 20:24:33 2011 NOTE: membership refresh pending for group 1/0xc6086872 (ORAFILES) kfdp_query(ORAFILES): 15 kfdp_queryBg(): 15 SUCCESS: refreshed membership for 1/0xc6086872 (ORAFILES) Tue Dec 20 12:11:57 2011 SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_19/thread_1_seq_59.270.762300535' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_19/thread_1_seq_59.270.762300535' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_19/thread_2_seq_1.271.762300555' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_19/thread_2_seq_1.271.762300555' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_19/thread_2_seq_2.272.762300555' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_19/thread_2_seq_2.272.762300555' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_19/thread_2_seq_3.273.762300555' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_19/thread_2_seq_3.273.762300555' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_19/thread_1_seq_60.274.762300559' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_19/thread_1_seq_60.274.762300559' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_19/thread_1_seq_61.275.762301605' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_19/thread_1_seq_61.275.762301605' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_19/thread_2_seq_4.276.762301999' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_19/thread_2_seq_4.276.762301999' Tue Dec 20 12:12:08 2011 SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_62.277.762349929' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_62.277.762349929' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_5.278.762356891' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_5.278.762356891' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_6.279.762356895' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_6.279.762356895' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_7.280.762356895' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_7.280.762356895' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_63.281.762356897' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_63.281.762356897' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_64.282.762357923' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_64.282.762357923' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_65.283.762357923' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_65.283.762357923' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_66.284.762357923' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_66.284.762357923' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_8.285.762357927' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_8.285.762357927' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_9.286.762358163' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_9.286.762358163' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_10.287.762358167' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_10.287.762358167' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_11.288.762358167' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_11.288.762358167' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_67.289.762358169' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_67.289.762358169' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_68.290.762358365' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_68.290.762358365' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_69.291.762358449' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_69.291.762358449' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_70.292.762358521' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_70.292.762358521' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_12.293.762358523' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_12.293.762358523' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_71.294.762358727' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_71.294.762358727' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_13.295.762359393' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_13.295.762359393' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_14.296.762359395' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_14.296.762359395' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_15.297.762359395' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_15.297.762359395' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_72.298.762359397' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_72.298.762359397' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_73.299.762359435' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_73.299.762359435' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_74.300.762359489' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_74.300.762359489' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_75.301.762359525' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_75.301.762359525' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_76.302.762359767' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_76.302.762359767' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_77.303.762359771' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_77.303.762359771' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_78.304.762359871' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_78.304.762359871' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_16.305.762359873' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_16.305.762359873' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_17.306.762361125' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_17.306.762361125' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_18.307.762361127' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_18.307.762361127' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_19.308.762361127' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_2_seq_19.308.762361127' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_79.309.762361515' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_79.309.762361515' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_80.310.762365065' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_80.310.762365065' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_81.311.762366007' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_81.311.762366007' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_82.312.762366331' Tue Dec 20 12:12:20 2011 SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_82.312.762366331' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_83.313.762366651' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_83.313.762366651' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_84.314.762371407' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_84.314.762371407' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_85.315.762371427' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_85.315.762371427' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_86.316.762371883' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_86.316.762371883' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_87.317.762373667' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_87.317.762373667' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_88.318.762374289' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_88.318.762374289' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_89.319.762374289' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_89.319.762374289' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_90.320.762374313' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_20/thread_1_seq_90.320.762374313' Tue Dec 20 12:12:34 2011 SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_1_seq_4.337.762807735' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_1_seq_4.337.762807735' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_2_seq_1.338.762807753' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_2_seq_1.338.762807753' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_2_seq_2.339.762807753' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_2_seq_2.339.762807753' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_2_seq_3.340.762807753' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_2_seq_3.340.762807753' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_1_seq_5.341.762807757' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_1_seq_5.341.762807757' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_1_seq_6.342.762808169' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_1_seq_6.342.762808169' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_1_seq_7.343.762808263' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_1_seq_7.343.762808263' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_1_seq_8.344.762808341' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_1_seq_8.344.762808341' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_2_seq_4.345.762808345' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_2_seq_4.345.762808345' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_1_seq_9.346.762809101' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_1_seq_9.346.762809101' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_2_seq_5.347.762809261' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_2_seq_5.347.762809261' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_1_seq_10.348.762809505' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_1_seq_10.348.762809505' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_2_seq_6.349.762809637' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_2_seq_6.349.762809637' SQL> alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_2_seq_7.350.762809841' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" drop file '+PRD01/PRD01/ARCHIVELOG/2011_09_25/thread_2_seq_7.350.762809841' Tue Dec 20 12:13:38 2011 SQL> alter diskgroup /*ASMCMD*/ "PRD01" add directory '+PRD01/PRD01/ARCHIVELOG' SUCCESS: alter diskgroup /*ASMCMD*/ "PRD01" add directory '+PRD01/PRD01/ARCHIVELOG' Wed Feb 15 18:52:07 2012 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_diag_8955.trc Wed Feb 15 20:34:07 2012 Received detach msg from node 1 for dom 1 Wed Feb 15 20:34:08 2012 NOTE: PST enabling heartbeating (grp 1) Received detach msg from node 1 for dom 2 NOTE: PST enabling heartbeating (grp 2) Received detach msg from node 1 for dom 3 NOTE: PST enabling heartbeating (grp 3) Wed Feb 15 20:34:11 2012 Reconfiguration started (old inc 32, new inc 34) List of nodes: 0 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Feb 15 20:34:12 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5405 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Wed Feb 15 20:44:02 2012 Reconfiguration started (old inc 34, new inc 36) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Feb 15 20:44:02 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5460 GCS shadows traversed, 2750 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Thu Feb 16 00:00:46 2012 Reconfiguration started (old inc 36, new inc 38) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Thu Feb 16 00:00:46 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5460 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Thu Feb 16 00:00:46 2012 kfdp_query(ORAFILES): 16 Thu Feb 16 00:00:46 2012 kfdp_queryBg(): 16 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=83.5305 group=1 NOTE: advancing ckpt for thread=1 ckpt=83.5305 NOTE: smon did instance recovery for domain 1 kfdp_query(FLASH): 17 kfdp_queryBg(): 17 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=53.3802 group=2 NOTE: advancing ckpt for thread=2 ckpt=53.3802 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 18 kfdp_queryBg(): 18 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=5.4711 group=3 NOTE: advancing ckpt for thread=2 ckpt=5.4711 NOTE: smon did instance recovery for domain 3 Thu Feb 16 00:07:05 2012 Reconfiguration started (old inc 38, new inc 40) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Thu Feb 16 00:07:05 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5458 GCS shadows traversed, 2759 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Thu Feb 16 00:23:04 2012 Trace dumping is performing id=[cdmp_20120216002304] Sat Feb 18 14:44:14 2012 Reconfiguration started (old inc 40, new inc 42) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Feb 18 14:44:14 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5460 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Sat Feb 18 14:44:14 2012 kfdp_query(ORAFILES): 19 Reconfiguration complete Sat Feb 18 14:44:14 2012 kfdp_queryBg(): 19 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=84.5681 group=1 NOTE: advancing ckpt for thread=1 ckpt=84.5681 NOTE: smon did instance recovery for domain 1 kfdp_query(FLASH): 20 kfdp_queryBg(): 20 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=55.1354 group=2 NOTE: advancing ckpt for thread=2 ckpt=55.1354 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 21 kfdp_queryBg(): 21 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=6.4781 group=3 NOTE: advancing ckpt for thread=2 ckpt=6.4781 NOTE: smon did instance recovery for domain 3 Sat Feb 18 15:05:05 2012 Reconfiguration started (old inc 42, new inc 44) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Feb 18 15:05:05 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5458 GCS shadows traversed, 2820 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Feb 18 17:27:23 2012 Reconfiguration started (old inc 44, new inc 46) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Feb 18 17:27:23 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5458 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sat Feb 18 17:27:23 2012 kfdp_query(ORAFILES): 22 Sat Feb 18 17:27:23 2012 kfdp_queryBg(): 22 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=85.5769 group=1 NOTE: advancing ckpt for thread=1 ckpt=85.5769 NOTE: smon did instance recovery for domain 1 kfdp_query(FLASH): 23 kfdp_queryBg(): 23 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=56.1355 group=2 NOTE: advancing ckpt for thread=2 ckpt=56.1355 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 24 kfdp_queryBg(): 24 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=7.4808 group=3 NOTE: advancing ckpt for thread=2 ckpt=7.4808 NOTE: smon did instance recovery for domain 3 Sat Feb 18 17:39:54 2012 Reconfiguration started (old inc 46, new inc 48) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Feb 18 17:39:54 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5456 GCS shadows traversed, 2818 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Feb 18 17:50:03 2012 Reconfiguration started (old inc 48, new inc 50) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Feb 18 17:50:03 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5456 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sat Feb 18 17:50:03 2012 kfdp_query(ORAFILES): 25 Sat Feb 18 17:50:03 2012 kfdp_queryBg(): 25 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=86.5770 group=1 NOTE: advancing ckpt for thread=1 ckpt=86.5770 NOTE: smon did instance recovery for domain 1 kfdp_query(FLASH): 26 kfdp_queryBg(): 26 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=57.1356 group=2 NOTE: advancing ckpt for thread=2 ckpt=57.1356 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 27 kfdp_queryBg(): 27 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=8.4810 group=3 NOTE: advancing ckpt for thread=2 ckpt=8.4810 NOTE: smon did instance recovery for domain 3 Sat Feb 18 17:57:21 2012 Reconfiguration started (old inc 50, new inc 52) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Feb 18 17:57:21 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5454 GCS shadows traversed, 2816 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Feb 18 18:00:08 2012 Reconfiguration started (old inc 52, new inc 54) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Feb 18 18:00:08 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Sat Feb 18 18:00:08 2012 kfdp_query(ORAFILES): 28 LMS 0: 5454 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sat Feb 18 18:00:08 2012 kfdp_queryBg(): 28 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=87.5771 group=1 NOTE: advancing ckpt for thread=1 ckpt=87.5771 NOTE: smon did instance recovery for domain 1 kfdp_query(FLASH): 29 kfdp_queryBg(): 29 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=58.1357 group=2 NOTE: advancing ckpt for thread=2 ckpt=58.1357 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 30 kfdp_queryBg(): 30 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=9.4812 group=3 NOTE: advancing ckpt for thread=2 ckpt=9.4812 NOTE: smon did instance recovery for domain 3 Sat Feb 18 18:07:48 2012 Reconfiguration started (old inc 54, new inc 56) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Feb 18 18:07:49 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5452 GCS shadows traversed, 2814 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Feb 18 18:14:12 2012 Trace dumping is performing id=[cdmp_20120218192353] Trace dumping is performing id=[cdmp_20120218192358] Sat Feb 18 18:14:20 2012 Reconfiguration started (old inc 56, new inc 58) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Feb 18 18:14:20 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5452 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sat Feb 18 18:14:20 2012 kfdp_query(ORAFILES): 31 Sat Feb 18 18:14:20 2012 kfdp_queryBg(): 31 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=88.5859 group=1 NOTE: advancing ckpt for thread=1 ckpt=88.5859 NOTE: smon did instance recovery for domain 1 kfdp_query(FLASH): 32 kfdp_queryBg(): 32 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=59.1366 group=2 NOTE: advancing ckpt for thread=2 ckpt=59.1366 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 33 kfdp_queryBg(): 33 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=10.4831 group=3 NOTE: advancing ckpt for thread=2 ckpt=10.4831 NOTE: smon did instance recovery for domain 3 Sat Feb 18 18:28:52 2012 Reconfiguration started (old inc 58, new inc 60) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Feb 18 18:28:52 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5450 GCS shadows traversed, 2813 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Feb 18 19:33:13 2012 SQL> ALTER DISKGROUP ORAFILES MOUNT WARNING: Deprecated privilege SYSDBA for command 'ALTER DISKGROUP MOUNT' ORA-15032: not all alterations performed ORA-15017: diskgroup "ORAFILES" cannot be mounted ORA-15013: diskgroup "ORAFILES" is already mounted ERROR: ALTER DISKGROUP ORAFILES MOUNT Sat Feb 18 20:42:06 2012 Trace dumping is performing id=[cdmp_20120218204206] Sat Feb 18 20:42:11 2012 Reconfiguration started (old inc 60, new inc 62) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 2 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Feb 18 20:42:11 2012 LMS 0: 1 GCS shadows cancelled, 1 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5450 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sat Feb 18 20:42:11 2012 NOTE: waiting for instance recovery of group 3 Sat Feb 18 20:42:11 2012 kfdp_query(FLASH): 34 Sat Feb 18 20:42:12 2012 kfdp_queryBg(): 34 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=60.1367 group=2 NOTE: advancing ckpt for thread=2 ckpt=60.1367 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 35 kfdp_queryBg(): 35 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=11.4849 group=3 NOTE: advancing ckpt for thread=2 ckpt=11.4849 NOTE: smon did instance recovery for domain 3 Sat Feb 18 20:55:49 2012 Reconfiguration started (old inc 62, new inc 64) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Feb 18 20:55:49 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5448 GCS shadows traversed, 2811 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Feb 18 21:23:28 2012 Reconfiguration started (old inc 64, new inc 66) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Feb 18 21:23:28 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5445 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sat Feb 18 21:23:28 2012 kfdp_query(ORAFILES): 36 Sat Feb 18 21:23:28 2012 kfdp_queryBg(): 36 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=89.5945 group=1 NOTE: advancing ckpt for thread=1 ckpt=89.5945 NOTE: smon did instance recovery for domain 1 kfdp_query(FLASH): 37 kfdp_queryBg(): 37 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=61.1373 group=2 NOTE: advancing ckpt for thread=2 ckpt=61.1373 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 38 kfdp_queryBg(): 38 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=12.4867 group=3 NOTE: advancing ckpt for thread=2 ckpt=12.4867 NOTE: smon did instance recovery for domain 3 Sat Feb 18 21:30:03 2012 Reconfiguration started (old inc 66, new inc 68) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Feb 18 21:30:03 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5445 GCS shadows traversed, 2808 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Feb 25 16:17:44 2012 Received detach msg from node 1 for dom 1 Received detach msg from node 1 for dom 2 Received detach msg from node 1 for dom 3 Sat Feb 25 16:17:48 2012 Reconfiguration started (old inc 68, new inc 70) List of nodes: 0 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Feb 25 16:17:48 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5405 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sat Feb 25 17:01:54 2012 Reconfiguration started (old inc 70, new inc 72) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Feb 25 17:01:54 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5457 GCS shadows traversed, 2829 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Feb 25 17:24:44 2012 Reconfiguration started (old inc 72, new inc 74) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Feb 25 17:24:44 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5457 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sat Feb 25 17:24:44 2012 kfdp_query(ORAFILES): 39 Sat Feb 25 17:24:44 2012 kfdp_queryBg(): 39 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=91.6818 group=1 NOTE: advancing ckpt for thread=1 ckpt=91.6818 NOTE: smon did instance recovery for domain 1 kfdp_query(FLASH): 40 kfdp_queryBg(): 40 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=65.387 group=2 NOTE: advancing ckpt for thread=2 ckpt=65.387 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 41 kfdp_queryBg(): 41 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=14.5121 group=3 NOTE: advancing ckpt for thread=2 ckpt=14.5121 NOTE: smon did instance recovery for domain 3 Sat Feb 25 17:30:10 2012 Reconfiguration started (old inc 74, new inc 76) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Feb 25 17:30:10 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5458 GCS shadows traversed, 2814 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Feb 25 17:31:56 2012 Reconfiguration started (old inc 76, new inc 78) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Feb 25 17:31:56 2012 LMS 0: 1 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Sat Feb 25 17:31:56 2012 kfdp_query(ORAFILES): 42 LMS 0: 5455 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sat Feb 25 17:31:56 2012 NOTE: waiting for instance recovery of group 1 Sat Feb 25 17:31:56 2012 kfdp_queryBg(): 42 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=92.6819 group=1 NOTE: advancing ckpt for thread=1 ckpt=92.6819 NOTE: smon did instance recovery for domain 1 kfdp_query(FLASH): 43 NOTE: waiting for instance recovery of group 2 kfdp_queryBg(): 43 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=66.388 group=2 NOTE: advancing ckpt for thread=2 ckpt=66.388 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 44 NOTE: waiting for instance recovery of group 3 kfdp_queryBg(): 44 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=15.5122 group=3 NOTE: advancing ckpt for thread=2 ckpt=15.5122 NOTE: smon did instance recovery for domain 3 Sat Feb 25 17:40:58 2012 Reconfiguration started (old inc 78, new inc 80) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Feb 25 17:40:58 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5456 GCS shadows traversed, 2813 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Fri Mar 16 22:19:55 2012 Trace dumping is performing id=[cdmp_20120316221955] Trace dumping is performing id=[cdmp_20120316222000] Fri Mar 16 22:20:05 2012 Reconfiguration started (old inc 80, new inc 82) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Mar 16 22:20:05 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5460 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Fri Mar 16 22:20:05 2012 NOTE: waiting for instance recovery of group 3 Fri Mar 16 22:20:05 2012 kfdp_query(ORAFILES): 45 Fri Mar 16 22:20:05 2012 kfdp_queryBg(): 45 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=93.8581 group=1 NOTE: advancing ckpt for thread=1 ckpt=93.8581 NOTE: smon did instance recovery for domain 1 kfdp_query(FLASH): 46 kfdp_queryBg(): 46 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=71.6580 group=2 NOTE: advancing ckpt for thread=2 ckpt=71.6580 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 47 kfdp_queryBg(): 47 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=16.5712 group=3 NOTE: advancing ckpt for thread=2 ckpt=16.5712 NOTE: smon did instance recovery for domain 3 Fri Mar 16 22:26:25 2012 Reconfiguration started (old inc 82, new inc 84) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Mar 16 22:26:25 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5458 GCS shadows traversed, 2772 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Fri Mar 16 22:33:54 2012 Trace dumping is performing id=[cdmp_20120316223353] Trace dumping is performing id=[cdmp_20120316223358] Fri Mar 16 22:34:02 2012 Reconfiguration started (old inc 84, new inc 86) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Fri Mar 16 22:34:02 2012 LMS 0: 1 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Fri Mar 16 22:34:03 2012 kfdp_query(ORAFILES): 48 LMS 0: 5457 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Fri Mar 16 22:34:03 2012 NOTE: waiting for instance recovery of group 3 Fri Mar 16 22:34:03 2012 kfdp_queryBg(): 48 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=94.8671 group=1 NOTE: advancing ckpt for thread=1 ckpt=94.8671 NOTE: smon did instance recovery for domain 1 kfdp_query(FLASH): 49 kfdp_queryBg(): 49 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=72.6595 group=2 NOTE: advancing ckpt for thread=2 ckpt=72.6595 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 50 kfdp_queryBg(): 50 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=17.5730 group=3 NOTE: advancing ckpt for thread=2 ckpt=17.5730 NOTE: smon did instance recovery for domain 3 Sat Mar 17 04:05:12 2012 Reconfiguration started (old inc 86, new inc 88) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Mar 17 04:05:12 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5460 GCS shadows traversed, 2772 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sun Mar 25 02:09:14 2012 Starting background process ASMB Sun Mar 25 02:09:14 2012 ASMB started with pid=22, OS id=26895 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Fri Apr 20 03:19:50 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Fri Apr 20 03:19:52 2012 PMON started with pid=2, OS id=9433 Fri Apr 20 03:19:52 2012 VKTM started with pid=3, OS id=9435 at elevated priority Fri Apr 20 03:19:52 2012 DIAG started with pid=4, OS id=9439 Fri Apr 20 03:19:52 2012 PING started with pid=5, OS id=9441 VKTM running at (20)ms precision Fri Apr 20 03:19:52 2012 PSP0 started with pid=6, OS id=9443 Fri Apr 20 03:19:52 2012 DIA0 started with pid=7, OS id=9445 Fri Apr 20 03:19:53 2012 LMON started with pid=8, OS id=9447 Fri Apr 20 03:19:53 2012 LMD0 started with pid=9, OS id=9449 Fri Apr 20 03:19:53 2012 LMS0 started with pid=10, OS id=9451 at elevated priority Fri Apr 20 03:19:53 2012 MMAN started with pid=11, OS id=9455 Fri Apr 20 03:19:53 2012 DBW0 started with pid=12, OS id=9457 Fri Apr 20 03:19:53 2012 LGWR started with pid=13, OS id=9459 Fri Apr 20 03:19:53 2012 CKPT started with pid=14, OS id=9461 Fri Apr 20 03:19:53 2012 SMON started with pid=15, OS id=9464 Fri Apr 20 03:19:53 2012 RBAL started with pid=16, OS id=9466 Fri Apr 20 03:19:53 2012 GMON started with pid=17, OS id=9468 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 92) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * allocate domain 2, invalid = TRUE * allocate domain 3, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 * domain 2 valid = 1 according to instance 1 * domain 3 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Fri Apr 20 03:19:55 2012 LCK0 started with pid=18, OS id=9599 ORACLE_BASE from environment = /u01/app/oracle Fri Apr 20 03:19:58 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc609016c NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc609016c NOTE: cache registered group FLASH number=2 incarn=0x3a29016b NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a29016b NOTE: cache registered group PRD01 number=3 incarn=0xd3f9016d NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f9016d WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (not first) group 1/0xC609016C (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.38594121 ABA NOTE: seq=199 blk=8788 NOTE: cache mounting group 1/0xC609016C (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc609016c kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A29016B (FLASH) kjbdomatt send to node 1 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 1 for diskgroup 2 NOTE: LGWR mounted thread 1 for disk group 2 NOTE: opening chunk 1 at fcn 0.63181315 ABA NOTE: seq=170 blk=2601 NOTE: cache mounting group 2/0x3A29016B (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a29016b kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F9016D (PRD01) kjbdomatt send to node 1 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 1 for diskgroup 3 NOTE: LGWR mounted thread 1 for disk group 3 NOTE: opening chunk 1 at fcn 0.3239216 ABA NOTE: seq=48 blk=5990 NOTE: cache mounting group 3/0xD3F9016D (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f9016d kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Fri Apr 20 03:20:10 2012 Starting background process ASMB Fri Apr 20 03:20:10 2012 ASMB started with pid=20, OS id=10382 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Fri Apr 20 03:21:01 2012 Starting background process ASMB Fri Apr 20 03:21:01 2012 ASMB started with pid=25, OS id=12695 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Tue May 08 15:50:48 2012 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_11339.trc Tue May 08 15:50:48 2012 Trace dumping is performing id=[cdmp_20120508155048] Tue May 08 15:50:55 2012 Starting background process ASMB Tue May 08 15:50:55 2012 ASMB started with pid=20, OS id=25332 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed May 16 19:47:46 2012 Shutting down instance: further logons disabled Shutting down instance (immediate) License high water mark = 14 SQL> ALTER DISKGROUP ALL DISMOUNT NOTE: cache dismounting group 1/0xC609016C (ORAFILES) Wed May 16 19:47:46 2012 kjbdomdet send to node 1 detach from dom 1, sending detach message to node 1 Wed May 16 19:47:46 2012 NOTE: detached from domain 1 NOTE: cache dismounted group 1/0xC609016C (ORAFILES) kfdp_dismount(): 10 Wed May 16 19:47:47 2012 kfdp_dismountBg(): 10 NOTE: De-assigning number (1,0) from disk (/dev/mpath/mpath1p1) NOTE: De-assigning number (1,1) from disk (/dev/mpath/mpath2p1) NOTE: De-assigning number (1,2) from disk (/dev/mpath/mpath5p1) SUCCESS: diskgroup ORAFILES was dismounted NOTE: cache dismounting group 2/0x3A29016B (FLASH) kjbdomdet send to node 1 detach from dom 2, sending detach message to node 1 NOTE: detached from domain 2 NOTE: cache dismounted group 2/0x3A29016B (FLASH) kfdp_dismount(): 11 kfdp_dismountBg(): 11 NOTE: De-assigning number (2,0) from disk (/dev/mpath/mpath3p1) SUCCESS: diskgroup FLASH was dismounted NOTE: cache dismounting group 3/0xD3F9016D (PRD01) kjbdomdet send to node 1 detach from dom 3, sending detach message to node 1 NOTE: detached from domain 3 NOTE: cache dismounted group 3/0xD3F9016D (PRD01) kfdp_dismount(): 12 kfdp_dismountBg(): 12 NOTE: De-assigning number (3,0) from disk (/dev/mpath/mpath4p1) SUCCESS: diskgroup PRD01 was dismounted SUCCESS: ALTER DISKGROUP ALL DISMOUNT Wed May 16 19:47:51 2012 Stopping background process VKTM: Wed May 16 19:47:55 2012 freeing rdom 3 freeing rdom 2 freeing rdom 1 freeing rdom 0 Wed May 16 19:47:57 2012 Instance shutdown complete Wed May 16 19:56:32 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Wed May 16 19:56:33 2012 PMON started with pid=2, OS id=9897 Wed May 16 19:56:33 2012 VKTM started with pid=3, OS id=9907 at elevated priority Wed May 16 19:56:33 2012 DIAG started with pid=4, OS id=9913 VKTM running at (20)ms precision Wed May 16 19:56:33 2012 PING started with pid=5, OS id=9917 Wed May 16 19:56:33 2012 PSP0 started with pid=6, OS id=9920 Wed May 16 19:56:34 2012 DIA0 started with pid=7, OS id=9922 Wed May 16 19:56:34 2012 LMON started with pid=8, OS id=9924 Wed May 16 19:56:34 2012 LMD0 started with pid=9, OS id=9926 Wed May 16 19:56:34 2012 LMS0 started with pid=10, OS id=9928 at elevated priority Wed May 16 19:56:34 2012 MMAN started with pid=11, OS id=9932 Wed May 16 19:56:34 2012 DBW0 started with pid=12, OS id=9934 Wed May 16 19:56:34 2012 LGWR started with pid=13, OS id=9936 Wed May 16 19:56:34 2012 CKPT started with pid=14, OS id=9938 Wed May 16 19:56:34 2012 SMON started with pid=15, OS id=9940 Wed May 16 19:56:34 2012 RBAL started with pid=16, OS id=9942 Wed May 16 19:56:34 2012 GMON started with pid=17, OS id=9944 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 96) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * allocate domain 2, invalid = TRUE * allocate domain 3, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 * domain 2 valid = 1 according to instance 1 * domain 3 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed May 16 19:56:36 2012 LCK0 started with pid=19, OS id=9981 ORACLE_BASE from environment = /u01/app/oracle Wed May 16 19:56:38 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc6088673 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc6088673 NOTE: cache registered group FLASH number=2 incarn=0x3a288672 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a288672 NOTE: cache registered group PRD01 number=3 incarn=0xd3f88674 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3f88674 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (not first) group 1/0xC6088673 (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.38603796 ABA NOTE: seq=201 blk=951 NOTE: cache mounting group 1/0xC6088673 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc6088673 kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A288672 (FLASH) kjbdomatt send to node 1 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 1 for diskgroup 2 NOTE: LGWR mounted thread 1 for disk group 2 NOTE: opening chunk 1 at fcn 0.68736027 ABA NOTE: seq=186 blk=2515 NOTE: cache mounting group 2/0x3A288672 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a288672 kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3F88674 (PRD01) kjbdomatt send to node 1 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 1 for diskgroup 3 NOTE: LGWR mounted thread 1 for disk group 3 NOTE: opening chunk 1 at fcn 0.3321623 ABA NOTE: seq=49 blk=9229 NOTE: cache mounting group 3/0xD3F88674 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3f88674 kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Wed May 16 19:56:50 2012 Starting background process ASMB Wed May 16 19:56:50 2012 ASMB started with pid=20, OS id=10340 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed May 16 20:24:23 2012 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_11015.trc Wed May 16 20:24:23 2012 Trace dumping is performing id=[cdmp_20120516202423] Wed May 16 20:32:19 2012 LMON (ospid: 9924): terminating the instance due to error 472 Wed May 16 20:32:19 2012 System state dump is made for local instance System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_diag_9913.trc Trace dumping is performing id=[cdmp_20120516203219] Instance terminated by LMON, pid = 9924 Wed May 16 20:48:35 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Wed May 16 20:48:35 2012 PMON started with pid=2, OS id=29669 Wed May 16 20:48:35 2012 VKTM started with pid=3, OS id=29674 at elevated priority VKTM running at (20)ms precision Wed May 16 20:48:35 2012 DIAG started with pid=4, OS id=29681 Wed May 16 20:48:35 2012 PING started with pid=5, OS id=29683 Wed May 16 20:48:35 2012 PSP0 started with pid=6, OS id=29685 Wed May 16 20:48:35 2012 DIA0 started with pid=7, OS id=29687 Wed May 16 20:48:35 2012 LMON started with pid=8, OS id=29689 Wed May 16 20:48:35 2012 LMD0 started with pid=9, OS id=29691 Wed May 16 20:48:35 2012 LMS0 started with pid=10, OS id=29693 at elevated priority Wed May 16 20:48:35 2012 MMAN started with pid=11, OS id=29697 Wed May 16 20:48:35 2012 DBW0 started with pid=12, OS id=29699 Wed May 16 20:48:35 2012 LGWR started with pid=13, OS id=29701 Wed May 16 20:48:35 2012 CKPT started with pid=14, OS id=29703 Wed May 16 20:48:35 2012 SMON started with pid=15, OS id=29705 Wed May 16 20:48:35 2012 RBAL started with pid=16, OS id=29707 Wed May 16 20:48:36 2012 GMON started with pid=17, OS id=29709 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 100) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * allocate domain 2, invalid = TRUE * allocate domain 3, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 * domain 2 valid = 1 according to instance 1 * domain 3 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed May 16 20:48:38 2012 LCK0 started with pid=18, OS id=29726 ORACLE_BASE from environment = /u01/app/oracle Wed May 16 20:48:39 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group ORAFILES number=1 incarn=0xc60d49e2 NOTE: cache began mount (not first) of group ORAFILES number=1 incarn=0xc60d49e2 NOTE: cache registered group FLASH number=2 incarn=0x3a2d49e1 NOTE: cache began mount (not first) of group FLASH number=2 incarn=0x3a2d49e1 NOTE: cache registered group PRD01 number=3 incarn=0xd3fd49e3 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0xd3fd49e3 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (1,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(ORAFILES): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 1: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 1: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (not first) group 1/0xC60D49E2 (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.38603800 ABA NOTE: seq=202 blk=954 NOTE: cache mounting group 1/0xC60D49E2 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=1 incarn=0xc60d49e2 kfdp_query(FLASH): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 2/0x3A2D49E1 (FLASH) kjbdomatt send to node 1 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 1 for diskgroup 2 NOTE: LGWR mounted thread 1 for disk group 2 NOTE: opening chunk 1 at fcn 0.68736027 ABA NOTE: seq=187 blk=2516 NOTE: cache mounting group 2/0x3A2D49E1 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=2 incarn=0x3a2d49e1 kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0xD3FD49E3 (PRD01) kjbdomatt send to node 1 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 1 for diskgroup 3 NOTE: LGWR mounted thread 1 for disk group 3 NOTE: opening chunk 1 at fcn 0.3321687 ABA NOTE: seq=50 blk=9254 NOTE: cache mounting group 3/0xD3FD49E3 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0xd3fd49e3 kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup FLASH was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Wed May 16 20:48:50 2012 Starting background process ASMB Wed May 16 20:48:50 2012 ASMB started with pid=20, OS id=29927 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed May 16 20:49:33 2012 Starting background process ASMB Wed May 16 20:49:33 2012 ASMB started with pid=25, OS id=31138 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu May 17 19:53:06 2012 Received detach msg from node 1 for dom 1 Thu May 17 19:53:06 2012 NOTE: PST enabling heartbeating (grp 1) Received detach msg from node 1 for dom 2 NOTE: PST enabling heartbeating (grp 2) Received detach msg from node 1 for dom 3 NOTE: PST enabling heartbeating (grp 3) Thu May 17 19:53:10 2012 Reconfiguration started (old inc 100, new inc 102) List of nodes: 0 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Thu May 17 19:53:10 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5433 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Thu May 17 20:47:52 2012 Reconfiguration started (old inc 102, new inc 104) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Thu May 17 20:47:52 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5457 GCS shadows traversed, 2825 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Jun 16 18:37:51 2012 Reconfiguration started (old inc 104, new inc 106) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sat Jun 16 18:37:51 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5460 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sat Jun 16 18:37:51 2012 kfdp_query(ORAFILES): 10 Sat Jun 16 18:37:51 2012 kfdp_queryBg(): 10 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=97.6567 group=1 NOTE: advancing ckpt for thread=1 ckpt=97.6567 NOTE: smon did instance recovery for domain 1 kfdp_query(FLASH): 11 kfdp_queryBg(): 11 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=90.10249 group=2 NOTE: advancing ckpt for thread=2 ckpt=90.10249 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 12 kfdp_queryBg(): 12 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=19.8246 group=3 NOTE: advancing ckpt for thread=2 ckpt=19.8246 NOTE: smon did instance recovery for domain 3 Sun Jun 17 00:30:25 2012 Reconfiguration started (old inc 106, new inc 108) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sun Jun 17 00:30:25 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5460 GCS shadows traversed, 2764 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Aug 15 07:28:15 2012 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_diag_29681.trc Wed Aug 15 07:34:07 2012 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_diag_29681.trc Wed Aug 15 08:15:19 2012 NOTE: ASM client dwh11:dwh1 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_31456.trc Wed Aug 15 08:15:19 2012 Trace dumping is performing id=[cdmp_20120815081519] Wed Aug 15 08:16:57 2012 Trace dumping is performing id=[cdmp_20120815081657] Wed Aug 15 08:18:27 2012 Trace dumping is performing id=[cdmp_20120815081827] Wed Aug 15 08:19:42 2012 Reconfiguration started (old inc 108, new inc 110) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Aug 15 08:19:42 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 5450 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Wed Aug 15 08:19:42 2012 kfdp_query(ORAFILES): 13 Wed Aug 15 08:19:42 2012 kfdp_queryBg(): 13 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=1 ckpt=98.10029 group=1 NOTE: advancing ckpt for thread=1 ckpt=98.10029 NOTE: smon did instance recovery for domain 1 kfdp_query(FLASH): 14 kfdp_queryBg(): 14 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=100.7275 group=2 NOTE: advancing ckpt for thread=2 ckpt=100.7275 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 15 kfdp_queryBg(): 15 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=2 ckpt=20.9734 group=3 NOTE: advancing ckpt for thread=2 ckpt=20.9734 NOTE: smon did instance recovery for domain 3 Wed Aug 15 08:20:30 2012 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_diag_29681.trc Wed Aug 15 08:21:49 2012 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_30381.trc Wed Aug 15 08:21:49 2012 Trace dumping is performing id=[cdmp_20120815082149] Wed Aug 15 08:21:52 2012 Reconfiguration started (old inc 110, new inc 112) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Aug 15 08:21:53 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 5450 GCS shadows traversed, 2763 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Aug 15 08:24:09 2012 Error: Cluster Synchronization Service is shutting down Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_29689.trc: ORA-29746: Cluster Synchronization Service is being shut down. LMON (ospid: 29689): terminating the instance due to error 29746 Instance terminated by LMON, pid = 29689 Wed Aug 15 08:33:23 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "/dev/mpath/mpath*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Wed Aug 15 08:33:24 2012 PMON started with pid=2, OS id=10996 Wed Aug 15 08:33:24 2012 VKTM started with pid=3, OS id=10998 at elevated priority VKTM running at (20)ms precision Wed Aug 15 08:33:24 2012 DIAG started with pid=4, OS id=11002 Wed Aug 15 08:33:24 2012 PING started with pid=5, OS id=11004 Wed Aug 15 08:33:24 2012 PSP0 started with pid=6, OS id=11006 Wed Aug 15 08:33:24 2012 DIA0 started with pid=7, OS id=11008 Wed Aug 15 08:33:24 2012 LMON started with pid=8, OS id=11010 Wed Aug 15 08:33:24 2012 LMD0 started with pid=9, OS id=11012 Wed Aug 15 08:33:24 2012 LMS0 started with pid=10, OS id=11014 at elevated priority Wed Aug 15 08:33:24 2012 MMAN started with pid=11, OS id=11019 Wed Aug 15 08:33:24 2012 DBW0 started with pid=12, OS id=11021 Wed Aug 15 08:33:24 2012 LGWR started with pid=13, OS id=11023 Wed Aug 15 08:33:24 2012 CKPT started with pid=14, OS id=11025 Wed Aug 15 08:33:24 2012 SMON started with pid=15, OS id=11027 Wed Aug 15 08:33:24 2012 RBAL started with pid=16, OS id=11029 Wed Aug 15 08:33:24 2012 GMON started with pid=17, OS id=11031 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 4) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Aug 15 08:33:25 2012 LCK0 started with pid=18, OS id=11043 ORACLE_BASE from environment = /u01/app/oracle Wed Aug 15 08:33:27 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group FLASH number=1 incarn=0x9798a2e2 * allocate domain 1, invalid = TRUE NOTE: cache began mount (not first) of group FLASH number=1 incarn=0x9798a2e2 NOTE: cache registered group ORAFILES number=2 incarn=0x97a8a2e3 NOTE: cache began mount (first) of group ORAFILES number=2 incarn=0x97a8a2e3 NOTE: cache registered group PRD01 number=3 incarn=0x97a8a2e4 * allocate domain 3, invalid = TRUE NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0x97a8a2e4 WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,0) to disk (/dev/mpath/mpath3p1) NOTE: Assigning number (2,2) to disk (/dev/mpath/mpath5p1) NOTE: Assigning number (2,1) to disk (/dev/mpath/mpath2p1) NOTE: Assigning number (2,0) to disk (/dev/mpath/mpath1p1) NOTE: Assigning number (3,0) to disk (/dev/mpath/mpath4p1) kfdp_query(FLASH): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: FLASH_0000 path:/dev/mpath/mpath3p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x9798A2E2 (FLASH) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.83805827 ABA NOTE: seq=102 blk=7322 NOTE: cache mounting group 1/0x9798A2E2 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=1 incarn=0x9798a2e2 Wed Aug 15 08:33:37 2012 NOTE: start heartbeating (grp 2) Wed Aug 15 08:33:37 2012 kfdp_query(ORAFILES): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 2: ORAFILES_0000 path:/dev/mpath/mpath1p1 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 2: ORAFILES_0001 path:/dev/mpath/mpath2p1 NOTE: cache opening disk 2 of grp 2: ORAFILES_0002 path:/dev/mpath/mpath5p1 NOTE: cache mounting (first) group 2/0x97A8A2E3 (ORAFILES) Wed Aug 15 08:33:37 2012 * allocate domain 2, invalid = TRUE kjbdomatt send to node 1 Wed Aug 15 08:33:37 2012 NOTE: attached to recovery domain 2 NOTE: starting recovery of thread=1 ckpt=99.10032 group=2 NOTE: advancing ckpt for thread=1 ckpt=99.10032 NOTE: cache recovered group 2 to fcn 0.38655623 Wed Aug 15 08:33:38 2012 NOTE: LGWR attempting to mount thread 1 for diskgroup 2 NOTE: LGWR mounted thread 1 for disk group 2 NOTE: opening chunk 1 at fcn 0.38655299 ABA NOTE: seq=100 blk=10033 NOTE: cache mounting group 2/0x97A8A2E3 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=2 incarn=0x97a8a2e3 kfdp_query(PRD01): 7 kfdp_queryBg(): 7 NOTE: cache opening disk 0 of grp 3: PRD01_0000 path:/dev/mpath/mpath4p1 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0x97A8A2E4 (PRD01) kjbdomatt send to node 1 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3543569 ABA NOTE: seq=22 blk=9773 NOTE: cache mounting group 3/0x97A8A2E4 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0x97a8a2e4 Wed Aug 15 08:33:38 2012 kfdp_query(FLASH): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup FLASH was mounted kfdp_query(ORAFILES): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(PRD01): 10 kfdp_queryBg(): 10 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Wed Aug 15 08:33:47 2012 Starting background process ASMB Wed Aug 15 08:33:47 2012 ASMB started with pid=20, OS id=11360 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed Aug 15 08:34:07 2012 Trace dumping is performing id=[cdmp_20120815083407] Wed Aug 15 08:34:51 2012 Starting background process ASMB Wed Aug 15 08:34:51 2012 ASMB started with pid=26, OS id=12871 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Aug 16 22:06:30 2012 ALTER SYSTEM SET asm_diskstring='ORCL:*' SCOPE=SPFILE; Thu Aug 16 22:06:58 2012 ALTER SYSTEM RESET asm_diskstring SCOPE=SPFILE SID='+ASM1'; ALTER SYSTEM RESET asm_diskstring SCOPE=SPFILE SID='+ASM2'; Thu Aug 16 22:08:12 2012 NOTE: ASM client dwh11:dwh1 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_13019.trc Thu Aug 16 22:08:12 2012 Trace dumping is performing id=[cdmp_20120816220812] Thu Aug 16 22:08:17 2012 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_11838.trc Trace dumping is performing id=[cdmp_20120816220817] Thu Aug 16 22:08:18 2012 Shutting down instance (abort) License high water mark = 15 USER (ospid: 14419): terminating the instance Instance terminated by USER, pid = 14419 Thu Aug 16 22:08:19 2012 Instance shutdown complete Thu Aug 16 22:09:19 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "ORCL:*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Thu Aug 16 22:09:19 2012 PMON started with pid=2, OS id=17058 Thu Aug 16 22:09:19 2012 VKTM started with pid=3, OS id=17060 at elevated priority VKTM running at (20)ms precision Thu Aug 16 22:09:19 2012 DIAG started with pid=4, OS id=17064 Thu Aug 16 22:09:19 2012 PING started with pid=5, OS id=17066 Thu Aug 16 22:09:19 2012 PSP0 started with pid=6, OS id=17068 Thu Aug 16 22:09:19 2012 DIA0 started with pid=7, OS id=17070 Thu Aug 16 22:09:19 2012 LMON started with pid=8, OS id=17072 Thu Aug 16 22:09:19 2012 LMD0 started with pid=9, OS id=17074 Thu Aug 16 22:09:20 2012 LMS0 started with pid=10, OS id=17076 at elevated priority Thu Aug 16 22:09:20 2012 MMAN started with pid=11, OS id=17080 Thu Aug 16 22:09:20 2012 DBW0 started with pid=12, OS id=17082 Thu Aug 16 22:09:20 2012 LGWR started with pid=13, OS id=17084 Thu Aug 16 22:09:20 2012 CKPT started with pid=14, OS id=17086 Thu Aug 16 22:09:20 2012 SMON started with pid=15, OS id=17088 Thu Aug 16 22:09:20 2012 RBAL started with pid=16, OS id=17090 Thu Aug 16 22:09:20 2012 GMON started with pid=17, OS id=17092 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 8) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * allocate domain 3, invalid = TRUE * allocate domain 2, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 * domain 3 valid = 1 according to instance 1 * domain 2 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Thu Aug 16 22:09:20 2012 LCK0 started with pid=18, OS id=17149 ORACLE_BASE from environment = /u01/app/oracle Thu Aug 16 22:09:21 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group FLASH number=1 incarn=0x979723b8 NOTE: cache began mount (not first) of group FLASH number=1 incarn=0x979723b8 NOTE: cache registered group ORAFILES number=2 incarn=0x97a723b9 NOTE: cache began mount (not first) of group ORAFILES number=2 incarn=0x97a723b9 NOTE: cache registered group PRD01 number=3 incarn=0x97a723ba NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0x97a723ba NOTE:Loaded lib: /opt/oracle/extapi/64/asm/orcl/1/libasm.so NOTE: Assigning number (1,0) to disk (ORCL:FLASH0) NOTE: Assigning number (2,0) to disk (ORCL:ORAFILES0) NOTE: Assigning number (2,1) to disk (ORCL:ORAFILES1) NOTE: Assigning number (2,2) to disk (ORCL:ORAFILES2) NOTE: Assigning number (3,0) to disk (ORCL:PRD010) kfdp_query(FLASH): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: FLASH_0000 label:FLASH0 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x979723B8 (FLASH) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.83951265 ABA NOTE: seq=104 blk=383 NOTE: cache mounting group 1/0x979723B8 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=1 incarn=0x979723b8 kfdp_query(ORAFILES): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: ORAFILES_0000 label:ORAFILES0 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 2: ORAFILES_0001 label:ORAFILES1 NOTE: cache opening disk 2 of grp 2: ORAFILES_0002 label:ORAFILES2 NOTE: cache mounting (not first) group 2/0x97A723B9 (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 1 for diskgroup 2 NOTE: LGWR mounted thread 1 for disk group 2 NOTE: opening chunk 1 at fcn 0.38656022 ABA NOTE: seq=101 blk=10186 NOTE: cache mounting group 2/0x97A723B9 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=2 incarn=0x97a723b9 kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 label:PRD010 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0x97A723BA (PRD01) kjbdomatt send to node 1 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3548963 ABA NOTE: seq=23 blk=10041 NOTE: cache mounting group 3/0x97A723BA (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0x97a723ba kfdp_query(FLASH): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup FLASH was mounted kfdp_query(ORAFILES): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Thu Aug 16 22:09:29 2012 Starting background process ASMB Thu Aug 16 22:09:29 2012 ASMB started with pid=21, OS id=17329 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Aug 16 22:10:06 2012 Starting background process ASMB Thu Aug 16 22:10:06 2012 ASMB started with pid=25, OS id=18469 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Thu Aug 16 22:12:43 2012 Trace dumping is performing id=[cdmp_20120816221243] Trace dumping is performing id=[cdmp_20120816221248] Thu Aug 16 22:12:54 2012 Reconfiguration started (old inc 8, new inc 10) List of nodes: 0 Global Resource Directory frozen * dead instance detected - domain 1 invalid = TRUE * dead instance detected - domain 3 invalid = TRUE * dead instance detected - domain 2 invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Thu Aug 16 22:12:54 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 1758 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Thu Aug 16 22:12:54 2012 kfdp_query(FLASH): 10 Thu Aug 16 22:12:54 2012 NOTE: PST enabling heartbeating (grp 2) NOTE: PST enabling heartbeating (grp 3) NOTE: PST enabling heartbeating (grp 1) kfdp_queryBg(): 10 NOTE: SMON starting instance recovery for group 1 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=1 ckpt=228.5362 group=1 NOTE: advancing ckpt for thread=1 ckpt=228.5362 NOTE: smon did instance recovery for domain 1 kfdp_query(ORAFILES): 11 kfdp_queryBg(): 11 NOTE: SMON starting instance recovery for group 2 (mounted) NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: starting recovery of thread=2 ckpt=203.8583 group=2 NOTE: advancing ckpt for thread=2 ckpt=203.8583 NOTE: smon did instance recovery for domain 2 kfdp_query(PRD01): 12 kfdp_queryBg(): 12 NOTE: SMON starting instance recovery for group 3 (mounted) NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: starting recovery of thread=1 ckpt=52.6507 group=3 NOTE: advancing ckpt for thread=1 ckpt=52.6507 NOTE: smon did instance recovery for domain 3 Thu Aug 16 22:13:33 2012 Reconfiguration started (old inc 10, new inc 12) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Thu Aug 16 22:13:33 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 1758 GCS shadows traversed, 1290 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Sep 19 19:53:53 2012 Shutting down instance: further logons disabled Shutting down instance (immediate) License high water mark = 10 SQL> ALTER DISKGROUP ALL DISMOUNT NOTE: cache dismounting group 1/0x979723B8 (FLASH) Wed Sep 19 19:53:53 2012 kjbdomdet send to node 1 detach from dom 1, sending detach message to node 1 Wed Sep 19 19:53:54 2012 NOTE: detached from domain 1 NOTE: cache dismounted group 1/0x979723B8 (FLASH) kfdp_dismount(): 13 Wed Sep 19 19:53:54 2012 kfdp_dismountBg(): 13 NOTE: De-assigning number (1,0) from disk (ORCL:FLASH0) SUCCESS: diskgroup FLASH was dismounted NOTE: cache dismounting group 2/0x97A723B9 (ORAFILES) kjbdomdet send to node 1 detach from dom 2, sending detach message to node 1 NOTE: detached from domain 2 NOTE: cache dismounted group 2/0x97A723B9 (ORAFILES) kfdp_dismount(): 14 kfdp_dismountBg(): 14 NOTE: De-assigning number (2,0) from disk (ORCL:ORAFILES0) NOTE: De-assigning number (2,1) from disk (ORCL:ORAFILES1) NOTE: De-assigning number (2,2) from disk (ORCL:ORAFILES2) SUCCESS: diskgroup ORAFILES was dismounted NOTE: cache dismounting group 3/0x97A723BA (PRD01) kjbdomdet send to node 1 detach from dom 3, sending detach message to node 1 NOTE: detached from domain 3 NOTE: cache dismounted group 3/0x97A723BA (PRD01) kfdp_dismount(): 15 kfdp_dismountBg(): 15 NOTE: De-assigning number (3,0) from disk (ORCL:PRD010) SUCCESS: diskgroup PRD01 was dismounted SUCCESS: ALTER DISKGROUP ALL DISMOUNT Wed Sep 19 19:53:56 2012 Stopping background process VKTM: Wed Sep 19 19:53:57 2012 freeing rdom 2 freeing rdom 3 freeing rdom 1 freeing rdom 0 Wed Sep 19 19:53:59 2012 Instance shutdown complete Wed Sep 19 20:01:34 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "ORCL:*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Wed Sep 19 20:01:37 2012 PMON started with pid=2, OS id=10770 Wed Sep 19 20:01:37 2012 VKTM started with pid=3, OS id=10776 at elevated priority Wed Sep 19 20:01:37 2012 DIAG started with pid=4, OS id=10802 VKTM running at (20)ms precision Wed Sep 19 20:01:37 2012 PING started with pid=5, OS id=10805 Wed Sep 19 20:01:37 2012 PSP0 started with pid=6, OS id=10818 Wed Sep 19 20:01:37 2012 DIA0 started with pid=7, OS id=10827 Wed Sep 19 20:01:37 2012 LMON started with pid=8, OS id=10829 Wed Sep 19 20:01:37 2012 LMD0 started with pid=9, OS id=10831 Wed Sep 19 20:01:37 2012 LMS0 started with pid=10, OS id=10841 at elevated priority Wed Sep 19 20:01:37 2012 MMAN started with pid=11, OS id=10855 Wed Sep 19 20:01:37 2012 DBW0 started with pid=12, OS id=10862 Wed Sep 19 20:01:37 2012 LGWR started with pid=13, OS id=10875 Wed Sep 19 20:01:37 2012 CKPT started with pid=14, OS id=10879 Wed Sep 19 20:01:37 2012 SMON started with pid=15, OS id=10881 Wed Sep 19 20:01:37 2012 RBAL started with pid=16, OS id=10883 Wed Sep 19 20:01:37 2012 GMON started with pid=17, OS id=10885 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 16) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * allocate domain 3, invalid = TRUE * allocate domain 2, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 * domain 3 valid = 1 according to instance 1 * domain 2 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Sep 19 20:01:38 2012 LCK0 started with pid=18, OS id=10982 ORACLE_BASE from environment = /u01/app/oracle Wed Sep 19 20:01:40 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group FLASH number=1 incarn=0x97987f08 NOTE: cache began mount (not first) of group FLASH number=1 incarn=0x97987f08 NOTE: cache registered group ORAFILES number=2 incarn=0x97a87f09 NOTE: cache began mount (not first) of group ORAFILES number=2 incarn=0x97a87f09 NOTE: cache registered group PRD01 number=3 incarn=0x97a87f0a NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0x97a87f0a NOTE:Loaded lib: /opt/oracle/extapi/64/asm/orcl/1/libasm.so NOTE: Assigning number (1,0) to disk (ORCL:FLASH0) NOTE: Assigning number (2,0) to disk (ORCL:ORAFILES0) NOTE: Assigning number (2,1) to disk (ORCL:ORAFILES1) NOTE: Assigning number (2,2) to disk (ORCL:ORAFILES2) NOTE: Assigning number (3,0) to disk (ORCL:PRD010) kfdp_query(FLASH): 4 kfdp_queryBg(): 4 NOTE: cache opening disk 0 of grp 1: FLASH_0000 label:FLASH0 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x97987F08 (FLASH) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.88424362 ABA NOTE: seq=116 blk=2455 NOTE: cache mounting group 1/0x97987F08 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=1 incarn=0x97987f08 kfdp_query(ORAFILES): 5 kfdp_queryBg(): 5 NOTE: cache opening disk 0 of grp 2: ORAFILES_0000 label:ORAFILES0 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 2: ORAFILES_0001 label:ORAFILES1 NOTE: cache opening disk 2 of grp 2: ORAFILES_0002 label:ORAFILES2 NOTE: cache mounting (not first) group 2/0x97A87F09 (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 1 for diskgroup 2 NOTE: LGWR mounted thread 1 for disk group 2 NOTE: opening chunk 1 at fcn 0.38661260 ABA NOTE: seq=103 blk=1634 NOTE: cache mounting group 2/0x97A87F09 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=2 incarn=0x97a87f09 kfdp_query(PRD01): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 3: PRD01_0000 label:PRD010 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0x97A87F0A (PRD01) kjbdomatt send to node 1 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3646354 ABA NOTE: seq=25 blk=2924 NOTE: cache mounting group 3/0x97A87F0A (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0x97a87f0a kfdp_query(FLASH): 7 kfdp_queryBg(): 7 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup FLASH was mounted kfdp_query(ORAFILES): 8 kfdp_queryBg(): 8 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(PRD01): 9 kfdp_queryBg(): 9 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Wed Sep 19 20:01:57 2012 Starting background process ASMB Wed Sep 19 20:01:57 2012 ASMB started with pid=20, OS id=11408 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed Sep 19 20:02:45 2012 Starting background process ASMB Wed Sep 19 20:02:45 2012 ASMB started with pid=25, OS id=12874 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed Sep 19 20:21:24 2012 Received detach msg from node 1 for dom 1 Wed Sep 19 20:21:25 2012 NOTE: PST enabling heartbeating (grp 1) Received detach msg from node 1 for dom 2 NOTE: PST enabling heartbeating (grp 2) Received detach msg from node 1 for dom 3 NOTE: PST enabling heartbeating (grp 3) Wed Sep 19 20:21:28 2012 Reconfiguration started (old inc 16, new inc 18) List of nodes: 0 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Sep 19 20:21:29 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR LMS 0: 4988 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Wed Sep 19 20:30:35 2012 Reconfiguration started (old inc 18, new inc 20) List of nodes: 0 1 Global Resource Directory frozen Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Sep 19 20:30:36 2012 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 4988 GCS shadows traversed, 2882 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Sep 19 20:36:44 2012 SQL> create diskgroup redoa external redundancy disk 'ORCL:REDOA0' WARNING: Deprecated privilege SYSDBA for command 'CREATE DISKGROUP' NOTE: Assigning number (4,0) to disk (ORCL:REDOA0) NOTE: initializing header on grp 4 disk REDOA0 NOTE: initiating PST update: grp = 4 kfdp_update(): 10 Wed Sep 19 20:36:45 2012 kfdp_updateBg(): 10 NOTE: group REDOA: initial PST location: disk 0000 (PST copy 0) NOTE: PST update grp = 4 completed successfully NOTE: cache registered group REDOA number=4 incarn=0x67887f0e NOTE: cache began mount (first) of group REDOA number=4 incarn=0x67887f0e NOTE: cache opening disk 0 of grp 4: REDOA0 label:REDOA0 Wed Sep 19 20:36:45 2012 * allocate domain 4, invalid = TRUE kjbdomatt send to node 1 Wed Sep 19 20:36:45 2012 NOTE: attached to recovery domain 4 NOTE: cache creating group 4/0x67887F0E (REDOA) NOTE: cache mounting group 4/0x67887F0E (REDOA) succeeded NOTE: allocating F1X0 on grp 4 disk REDOA0 NOTE: diskgroup must now be re-mounted prior to first use NOTE: cache dismounting group 4/0x67887F0E (REDOA) NOTE: lgwr not being msg'd to dismount kjbdomdet send to node 1 detach from dom 4, sending detach message to node 1 freeing rdom 4 NOTE: detached from domain 4 NOTE: cache dismounted group 4/0x67887F0E (REDOA) NOTE: cache ending mount (fail) of group REDOA number=4 incarn=0x67887f0e kfdp_dismount(): 11 kfdp_dismountBg(): 11 kfdp_dismount(): 12 kfdp_dismountBg(): 12 NOTE: De-assigning number (4,0) from disk (ORCL:REDOA0) SUCCESS: diskgroup REDOA was created NOTE: cache registered group REDOA number=4 incarn=0x79f87f10 NOTE: cache began mount (first) of group REDOA number=4 incarn=0x79f87f10 NOTE: Assigning number (4,0) to disk (ORCL:REDOA0) NOTE: start heartbeating (grp 4) kfdp_query(REDOA): 15 kfdp_queryBg(): 15 NOTE: cache opening disk 0 of grp 4: REDOA0 label:REDOA0 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (first) group 4/0x79F87F10 (REDOA) * allocate domain 4, invalid = TRUE kjbdomatt send to node 1 NOTE: attached to recovery domain 4 NOTE: cache recovered group 4 to fcn 0.0 Wed Sep 19 20:36:52 2012 NOTE: LGWR attempting to mount thread 1 for diskgroup 4 NOTE: LGWR mounted thread 1 for disk group 4 NOTE: opening chunk 1 at fcn 0.0 ABA NOTE: seq=2 blk=0 NOTE: cache mounting group 4/0x79F87F10 (REDOA) succeeded NOTE: cache ending mount (success) of group REDOA number=4 incarn=0x79f87f10 Wed Sep 19 20:36:52 2012 kfdp_query(REDOA): 16 kfdp_queryBg(): 16 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 4 SUCCESS: diskgroup REDOA was mounted SUCCESS: create diskgroup redoa external redundancy disk 'ORCL:REDOA0' Wed Sep 19 20:37:09 2012 NOTE: enlarging ACD for group 4/0x79f87f10 (REDOA) SUCCESS: ACD enlarged for group 4/0x79f87f10 (REDOA) Wed Sep 19 20:37:18 2012 SQL> create diskgroup redob external redundancy disk 'ORCL:REDOB0' WARNING: Deprecated privilege SYSDBA for command 'CREATE DISKGROUP' NOTE: Assigning number (5,0) to disk (ORCL:REDOB0) NOTE: initializing header on grp 5 disk REDOB0 NOTE: initiating PST update: grp = 5 kfdp_update(): 17 Wed Sep 19 20:37:19 2012 kfdp_updateBg(): 17 NOTE: group REDOB: initial PST location: disk 0000 (PST copy 0) NOTE: PST update grp = 5 completed successfully NOTE: cache registered group REDOB number=5 incarn=0x3cd87f13 NOTE: cache began mount (first) of group REDOB number=5 incarn=0x3cd87f13 NOTE: cache opening disk 0 of grp 5: REDOB0 label:REDOB0 Wed Sep 19 20:37:19 2012 * allocate domain 5, invalid = TRUE kjbdomatt send to node 1 Wed Sep 19 20:37:19 2012 NOTE: attached to recovery domain 5 NOTE: cache creating group 5/0x3CD87F13 (REDOB) NOTE: cache mounting group 5/0x3CD87F13 (REDOB) succeeded NOTE: allocating F1X0 on grp 5 disk REDOB0 NOTE: diskgroup must now be re-mounted prior to first use NOTE: cache dismounting group 5/0x3CD87F13 (REDOB) NOTE: lgwr not being msg'd to dismount kjbdomdet send to node 1 detach from dom 5, sending detach message to node 1 freeing rdom 5 NOTE: detached from domain 5 NOTE: cache dismounted group 5/0x3CD87F13 (REDOB) NOTE: cache ending mount (fail) of group REDOB number=5 incarn=0x3cd87f13 kfdp_dismount(): 18 kfdp_dismountBg(): 18 kfdp_dismount(): 19 kfdp_dismountBg(): 19 NOTE: De-assigning number (5,0) from disk (ORCL:REDOB0) SUCCESS: diskgroup REDOB was created NOTE: cache registered group REDOB number=5 incarn=0x48587f15 NOTE: cache began mount (first) of group REDOB number=5 incarn=0x48587f15 NOTE: Assigning number (5,0) to disk (ORCL:REDOB0) NOTE: start heartbeating (grp 5) kfdp_query(REDOB): 22 kfdp_queryBg(): 22 NOTE: cache opening disk 0 of grp 5: REDOB0 label:REDOB0 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (first) group 5/0x48587F15 (REDOB) * allocate domain 5, invalid = TRUE kjbdomatt send to node 1 NOTE: attached to recovery domain 5 NOTE: cache recovered group 5 to fcn 0.0 Wed Sep 19 20:37:25 2012 NOTE: LGWR attempting to mount thread 1 for diskgroup 5 NOTE: LGWR mounted thread 1 for disk group 5 NOTE: opening chunk 1 at fcn 0.0 ABA NOTE: seq=2 blk=0 NOTE: cache mounting group 5/0x48587F15 (REDOB) succeeded NOTE: cache ending mount (success) of group REDOB number=5 incarn=0x48587f15 Wed Sep 19 20:37:25 2012 kfdp_query(REDOB): 23 kfdp_queryBg(): 23 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 5 SUCCESS: diskgroup REDOB was mounted SUCCESS: create diskgroup redob external redundancy disk 'ORCL:REDOB0' NOTE: enlarging ACD for group 5/0x48587f15 (REDOB) SUCCESS: ACD enlarged for group 5/0x48587f15 (REDOB) Wed Sep 19 20:41:45 2012 Shutting down instance: further logons disabled Shutting down instance (immediate) License high water mark = 8 SQL> ALTER DISKGROUP ALL DISMOUNT NOTE: cache dismounting group 1/0x97987F08 (FLASH) Wed Sep 19 20:41:45 2012 kjbdomdet send to node 1 detach from dom 1, sending detach message to node 1 Wed Sep 19 20:41:45 2012 NOTE: detached from domain 1 NOTE: cache dismounted group 1/0x97987F08 (FLASH) kfdp_dismount(): 24 Wed Sep 19 20:41:45 2012 kfdp_dismountBg(): 24 NOTE: De-assigning number (1,0) from disk (ORCL:FLASH0) SUCCESS: diskgroup FLASH was dismounted NOTE: cache dismounting group 2/0x97A87F09 (ORAFILES) kjbdomdet send to node 1 detach from dom 2, sending detach message to node 1 NOTE: detached from domain 2 NOTE: cache dismounted group 2/0x97A87F09 (ORAFILES) kfdp_dismount(): 25 kfdp_dismountBg(): 25 NOTE: De-assigning number (2,0) from disk (ORCL:ORAFILES0) NOTE: De-assigning number (2,1) from disk (ORCL:ORAFILES1) NOTE: De-assigning number (2,2) from disk (ORCL:ORAFILES2) SUCCESS: diskgroup ORAFILES was dismounted NOTE: cache dismounting group 3/0x97A87F0A (PRD01) kjbdomdet send to node 1 detach from dom 3, sending detach message to node 1 NOTE: detached from domain 3 NOTE: cache dismounted group 3/0x97A87F0A (PRD01) kfdp_dismount(): 26 kfdp_dismountBg(): 26 NOTE: De-assigning number (3,0) from disk (ORCL:PRD010) SUCCESS: diskgroup PRD01 was dismounted NOTE: cache dismounting group 4/0x79F87F10 (REDOA) kjbdomdet send to node 1 detach from dom 4, sending detach message to node 1 NOTE: detached from domain 4 NOTE: cache dismounted group 4/0x79F87F10 (REDOA) kfdp_dismount(): 27 kfdp_dismountBg(): 27 NOTE: De-assigning number (4,0) from disk (ORCL:REDOA0) SUCCESS: diskgroup REDOA was dismounted NOTE: cache dismounting group 5/0x48587F15 (REDOB) kjbdomdet send to node 1 detach from dom 5, sending detach message to node 1 NOTE: detached from domain 5 NOTE: cache dismounted group 5/0x48587F15 (REDOB) kfdp_dismount(): 28 kfdp_dismountBg(): 28 NOTE: De-assigning number (5,0) from disk (ORCL:REDOB0) SUCCESS: diskgroup REDOB was dismounted SUCCESS: ALTER DISKGROUP ALL DISMOUNT Wed Sep 19 20:41:47 2012 Stopping background process VKTM: Wed Sep 19 20:41:49 2012 freeing rdom 5 freeing rdom 4 freeing rdom 2 freeing rdom 3 freeing rdom 1 freeing rdom 0 Wed Sep 19 20:41:50 2012 Instance shutdown complete Wed Sep 19 20:51:06 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "ORCL:*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" asm_diskgroups = "REDOA" asm_diskgroups = "REDOB" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Wed Sep 19 20:51:07 2012 PMON started with pid=2, OS id=24609 Wed Sep 19 20:51:07 2012 VKTM started with pid=3, OS id=24611 at elevated priority VKTM running at (20)ms precision Wed Sep 19 20:51:07 2012 DIAG started with pid=4, OS id=24615 Wed Sep 19 20:51:07 2012 PING started with pid=5, OS id=24617 Wed Sep 19 20:51:07 2012 PSP0 started with pid=6, OS id=24619 Wed Sep 19 20:51:07 2012 DIA0 started with pid=7, OS id=24621 Wed Sep 19 20:51:07 2012 LMON started with pid=8, OS id=24623 Wed Sep 19 20:51:07 2012 LMD0 started with pid=9, OS id=24625 Wed Sep 19 20:51:07 2012 LMS0 started with pid=10, OS id=24627 at elevated priority Wed Sep 19 20:51:07 2012 MMAN started with pid=11, OS id=24631 Wed Sep 19 20:51:07 2012 DBW0 started with pid=12, OS id=24633 Wed Sep 19 20:51:07 2012 LGWR started with pid=13, OS id=24635 Wed Sep 19 20:51:07 2012 CKPT started with pid=14, OS id=24637 Wed Sep 19 20:51:07 2012 SMON started with pid=15, OS id=24639 Wed Sep 19 20:51:07 2012 RBAL started with pid=16, OS id=24641 Wed Sep 19 20:51:07 2012 GMON started with pid=17, OS id=24643 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 4) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * allocate domain 2, invalid = TRUE * allocate domain 3, invalid = TRUE * allocate domain 4, invalid = TRUE * allocate domain 5, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 * domain 2 valid = 1 according to instance 1 * domain 3 valid = 1 according to instance 1 * domain 4 valid = 1 according to instance 1 * domain 5 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Sep 19 20:51:08 2012 LCK0 started with pid=18, OS id=24654 ORACLE_BASE from environment = /u01/app/oracle Wed Sep 19 20:51:09 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group FLASH number=1 incarn=0x496d0764 NOTE: cache began mount (not first) of group FLASH number=1 incarn=0x496d0764 NOTE: cache registered group ORAFILES number=2 incarn=0x496d0765 NOTE: cache began mount (not first) of group ORAFILES number=2 incarn=0x496d0765 NOTE: cache registered group PRD01 number=3 incarn=0x496d0766 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0x496d0766 NOTE: cache registered group REDOA number=4 incarn=0x496d0767 NOTE: cache began mount (not first) of group REDOA number=4 incarn=0x496d0767 NOTE: cache registered group REDOB number=5 incarn=0x497d0768 NOTE: cache began mount (not first) of group REDOB number=5 incarn=0x497d0768 NOTE:Loaded lib: /opt/oracle/extapi/64/asm/orcl/1/libasm.so NOTE: Assigning number (1,0) to disk (ORCL:FLASH0) NOTE: Assigning number (2,0) to disk (ORCL:ORAFILES0) NOTE: Assigning number (2,1) to disk (ORCL:ORAFILES1) NOTE: Assigning number (2,2) to disk (ORCL:ORAFILES2) NOTE: Assigning number (3,0) to disk (ORCL:PRD010) NOTE: Assigning number (4,0) to disk (ORCL:REDOA0) NOTE: Assigning number (5,0) to disk (ORCL:REDOB0) kfdp_query(FLASH): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 1: FLASH_0000 label:FLASH0 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x496D0764 (FLASH) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.88424362 ABA NOTE: seq=117 blk=2455 NOTE: cache mounting group 1/0x496D0764 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=1 incarn=0x496d0764 kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: cache opening disk 0 of grp 2: ORAFILES_0000 label:ORAFILES0 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 2: ORAFILES_0001 label:ORAFILES1 NOTE: cache opening disk 2 of grp 2: ORAFILES_0002 label:ORAFILES2 NOTE: cache mounting (not first) group 2/0x496D0765 (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.38660957 ABA NOTE: seq=206 blk=10376 NOTE: cache mounting group 2/0x496D0765 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=2 incarn=0x496d0765 kfdp_query(PRD01): 8 kfdp_queryBg(): 8 NOTE: cache opening disk 0 of grp 3: PRD01_0000 label:PRD010 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0x496D0766 (PRD01) kjbdomatt send to node 1 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3646394 ABA NOTE: seq=26 blk=2945 NOTE: cache mounting group 3/0x496D0766 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0x496d0766 kfdp_query(REDOA): 9 kfdp_queryBg(): 9 NOTE: cache opening disk 0 of grp 4: REDOA0 label:REDOA0 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 4/0x496D0767 (REDOA) kjbdomatt send to node 1 NOTE: attached to recovery domain 4 NOTE: LGWR attempting to mount thread 2 for diskgroup 4 NOTE: LGWR mounted thread 2 for disk group 4 NOTE: opening chunk 2 at fcn 0.0 ABA NOTE: seq=3 blk=0 NOTE: cache mounting group 4/0x496D0767 (REDOA) succeeded NOTE: cache ending mount (success) of group REDOA number=4 incarn=0x496d0767 kfdp_query(REDOB): 10 kfdp_queryBg(): 10 NOTE: cache opening disk 0 of grp 5: REDOB0 label:REDOB0 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 5/0x497D0768 (REDOB) kjbdomatt send to node 1 NOTE: attached to recovery domain 5 NOTE: LGWR attempting to mount thread 2 for diskgroup 5 NOTE: LGWR mounted thread 2 for disk group 5 NOTE: opening chunk 2 at fcn 0.0 ABA NOTE: seq=3 blk=0 NOTE: cache mounting group 5/0x497D0768 (REDOB) succeeded NOTE: cache ending mount (success) of group REDOB number=5 incarn=0x497d0768 kfdp_query(FLASH): 11 kfdp_queryBg(): 11 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup FLASH was mounted kfdp_query(ORAFILES): 12 kfdp_queryBg(): 12 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(PRD01): 13 kfdp_queryBg(): 13 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted kfdp_query(REDOA): 14 kfdp_queryBg(): 14 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 4 SUCCESS: diskgroup REDOA was mounted kfdp_query(REDOB): 15 kfdp_queryBg(): 15 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 5 SUCCESS: diskgroup REDOB was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Wed Sep 19 20:51:32 2012 Starting background process ASMB Wed Sep 19 20:51:32 2012 ASMB started with pid=21, OS id=25007 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Wed Sep 19 20:52:08 2012 Starting background process ASMB Wed Sep 19 20:52:08 2012 ASMB started with pid=25, OS id=26090 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sat Nov 24 13:34:01 2012 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_diag_24615.trc Sat Nov 24 13:34:35 2012 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_diag_24615.trc Sat Nov 24 14:06:25 2012 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ckpt_24637.trc Sat Nov 24 14:06:36 2012 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ckpt_24637.trc: ORA-15082: ASM failed to communicate with database instance CKPT (ospid: 24637): terminating the instance due to error 15082 Sat Nov 24 14:06:36 2012 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_diag_24615.trc Sat Nov 24 14:06:36 2012 ORA-1092 : opitsk aborting process Sat Nov 24 14:06:37 2012 ORA-1092 : opitsk aborting process Sat Nov 24 14:06:37 2012 License high water mark = 18 Instance terminated by CKPT, pid = 24637 USER (ospid: 22258): terminating the instance Instance terminated by USER, pid = 22258 Sat Nov 24 14:06:47 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "ORCL:*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" asm_diskgroups = "REDOA" asm_diskgroups = "REDOB" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sat Nov 24 14:06:47 2012 PMON started with pid=2, OS id=24081 Sat Nov 24 14:06:47 2012 VKTM started with pid=3, OS id=24083 at elevated priority VKTM running at (20)ms precision Sat Nov 24 14:06:47 2012 DIAG started with pid=4, OS id=24087 Sat Nov 24 14:06:47 2012 PING started with pid=5, OS id=24089 Sat Nov 24 14:06:47 2012 PSP0 started with pid=6, OS id=24091 Sat Nov 24 14:06:47 2012 DIA0 started with pid=7, OS id=24093 Sat Nov 24 14:06:47 2012 LMON started with pid=8, OS id=24095 Sat Nov 24 14:06:47 2012 LMD0 started with pid=9, OS id=24097 Sat Nov 24 14:06:48 2012 LMS0 started with pid=10, OS id=24099 at elevated priority Sat Nov 24 14:06:48 2012 MMAN started with pid=11, OS id=24103 Sat Nov 24 14:06:48 2012 DBW0 started with pid=12, OS id=24105 Sat Nov 24 14:06:48 2012 LGWR started with pid=13, OS id=24107 Sat Nov 24 14:06:48 2012 CKPT started with pid=14, OS id=24109 Sat Nov 24 14:06:48 2012 SMON started with pid=15, OS id=24111 Sat Nov 24 14:06:48 2012 RBAL started with pid=16, OS id=24113 Sat Nov 24 14:06:48 2012 GMON started with pid=17, OS id=24115 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 8) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * allocate domain 2, invalid = TRUE * allocate domain 3, invalid = TRUE * allocate domain 4, invalid = TRUE * allocate domain 5, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 * domain 2 valid = 1 according to instance 1 * domain 3 valid = 1 according to instance 1 * domain 4 valid = 1 according to instance 1 * domain 5 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sat Nov 24 14:06:49 2012 LCK0 started with pid=18, OS id=24134 ORACLE_BASE from environment = /u01/app/oracle Sat Nov 24 14:06:50 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group FLASH number=1 incarn=0x49695bb0 NOTE: cache began mount (not first) of group FLASH number=1 incarn=0x49695bb0 NOTE: cache registered group ORAFILES number=2 incarn=0x49695bb1 NOTE: cache began mount (not first) of group ORAFILES number=2 incarn=0x49695bb1 NOTE: cache registered group PRD01 number=3 incarn=0x49695bb2 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0x49695bb2 NOTE: cache registered group REDOA number=4 incarn=0x49695bb3 NOTE: cache began mount (not first) of group REDOA number=4 incarn=0x49695bb3 NOTE: cache registered group REDOB number=5 incarn=0x49795bb4 NOTE: cache began mount (not first) of group REDOB number=5 incarn=0x49795bb4 NOTE:Loaded lib: /opt/oracle/extapi/64/asm/orcl/1/libasm.so NOTE: Assigning number (1,0) to disk (ORCL:FLASH0) NOTE: Assigning number (2,0) to disk (ORCL:ORAFILES0) NOTE: Assigning number (2,1) to disk (ORCL:ORAFILES1) NOTE: Assigning number (2,2) to disk (ORCL:ORAFILES2) NOTE: Assigning number (3,0) to disk (ORCL:PRD010) NOTE: Assigning number (4,0) to disk (ORCL:REDOA0) NOTE: Assigning number (5,0) to disk (ORCL:REDOB0) kfdp_query(FLASH): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 1: FLASH_0000 label:FLASH0 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x49695BB0 (FLASH) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.97252787 ABA NOTE: seq=141 blk=1766 NOTE: cache mounting group 1/0x49695BB0 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=1 incarn=0x49695bb0 kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: cache opening disk 0 of grp 2: ORAFILES_0000 label:ORAFILES0 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 2: ORAFILES_0001 label:ORAFILES1 NOTE: cache opening disk 2 of grp 2: ORAFILES_0002 label:ORAFILES2 NOTE: cache mounting (not first) group 2/0x49695BB1 (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.38666953 ABA NOTE: seq=208 blk=723 NOTE: cache mounting group 2/0x49695BB1 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=2 incarn=0x49695bb1 kfdp_query(PRD01): 8 kfdp_queryBg(): 8 NOTE: cache opening disk 0 of grp 3: PRD01_0000 label:PRD010 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0x49695BB2 (PRD01) kjbdomatt send to node 1 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.3784877 ABA NOTE: seq=27 blk=7942 NOTE: cache mounting group 3/0x49695BB2 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0x49695bb2 kfdp_query(REDOA): 9 kfdp_queryBg(): 9 NOTE: cache opening disk 0 of grp 4: REDOA0 label:REDOA0 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 4/0x49695BB3 (REDOA) kjbdomatt send to node 1 NOTE: attached to recovery domain 4 NOTE: LGWR attempting to mount thread 2 for diskgroup 4 NOTE: LGWR mounted thread 2 for disk group 4 NOTE: opening chunk 2 at fcn 0.56635 ABA NOTE: seq=4 blk=4855 NOTE: cache mounting group 4/0x49695BB3 (REDOA) succeeded NOTE: cache ending mount (success) of group REDOA number=4 incarn=0x49695bb3 kfdp_query(REDOB): 10 kfdp_queryBg(): 10 NOTE: cache opening disk 0 of grp 5: REDOB0 label:REDOB0 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 5/0x49795BB4 (REDOB) kjbdomatt send to node 1 NOTE: attached to recovery domain 5 NOTE: LGWR attempting to mount thread 2 for diskgroup 5 NOTE: LGWR mounted thread 2 for disk group 5 NOTE: opening chunk 2 at fcn 0.59053 ABA NOTE: seq=4 blk=4879 NOTE: cache mounting group 5/0x49795BB4 (REDOB) succeeded NOTE: cache ending mount (success) of group REDOB number=5 incarn=0x49795bb4 kfdp_query(FLASH): 11 kfdp_queryBg(): 11 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup FLASH was mounted kfdp_query(ORAFILES): 12 kfdp_queryBg(): 12 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(PRD01): 13 kfdp_queryBg(): 13 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted kfdp_query(REDOA): 14 kfdp_queryBg(): 14 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 4 SUCCESS: diskgroup REDOA was mounted kfdp_query(REDOB): 15 kfdp_queryBg(): 15 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 5 SUCCESS: diskgroup REDOB was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sat Nov 24 14:06:59 2012 Starting background process ASMB Sat Nov 24 14:06:59 2012 ASMB started with pid=20, OS id=25218 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Jan 27 02:31:27 2013 NOTE: ASM client dwh11:dwh1 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_25847.trc Sun Jan 27 02:31:28 2013 Trace dumping is performing id=[cdmp_20130127023128] Sun Jan 27 02:31:33 2013 Starting background process ASMB Sun Jan 27 02:31:33 2013 ASMB started with pid=21, OS id=25730 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Mon Mar 04 16:02:54 2013 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_diag_24087.trc Mon Mar 04 16:15:51 2013 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_diag_24087.trc Sun Apr 21 02:32:04 2013 Immediate Kill Session#: 78, Serial#: 3 Immediate Kill Session: sess: 0x70f8e920 OS pid: 29440 Sun Apr 21 02:33:22 2013 Immediate Kill Session#: 76, Serial#: 4 Immediate Kill Session: sess: 0x70f89290 OS pid: 30069 Sun Apr 21 02:37:24 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 02:38:52 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 02:40:25 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 02:41:53 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 02:43:25 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 02:44:54 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 02:46:26 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 02:47:55 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 02:49:25 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 02:50:55 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 02:52:25 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 02:53:55 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 02:55:27 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 02:56:56 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 02:58:26 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 02:59:57 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:01:26 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:02:57 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:04:25 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:05:57 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:07:30 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:08:57 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:10:27 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:11:58 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:13:27 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:14:58 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:16:27 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:17:59 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:19:27 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:21:00 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:22:27 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:24:01 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:25:01 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:26:03 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:27:12 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:28:27 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:30:03 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:31:28 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:33:04 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:34:29 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:36:04 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:37:29 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:39:05 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:40:29 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:42:05 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:43:29 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:45:05 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:46:29 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:48:06 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:49:30 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:51:06 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:52:29 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:54:05 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:55:30 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:56:33 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:58:38 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 03:59:42 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 04:00:48 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 04:01:51 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 04:03:53 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 04:08:59 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 04:10:45 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 04:14:04 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 04:15:13 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 04:19:09 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 04:24:14 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 04:25:46 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 04:29:21 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 04:32:05 2013 ORA-00020: maximum number of processes 0 exceeded ORA-20 errors will not be written to the alert log for the next minute. Please look at trace files to see all the ORA-20 errors. Sun Apr 21 04:33:11 2013 NOTE: ASM client dwh11:dwh1 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_25868.trc Sun Apr 21 04:33:11 2013 Trace dumping is performing id=[cdmp_20130421043311] Sun Apr 21 04:41:02 2013 NOTE: ASM client PRD011:PRD01 died unexpectedly. NOTE: Process state recorded in trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_26751.trc Sun Apr 21 04:41:02 2013 Trace dumping is performing id=[cdmp_20130421044102] Sun Apr 21 04:41:18 2013 PSP0 (ospid: 24091): terminating the instance due to error 472 Sun Apr 21 04:41:18 2013 System state dump is made for local instance System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_diag_24087.trc Trace dumping is performing id=[cdmp_20130421044118] Instance terminated by PSP0, pid = 24091 Sun Apr 21 04:56:58 2013 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 bond1 10.1.1.0 configured from OCR for use as a cluster interconnect Interface type 1 bond0 10.30.5.0 configured from OCR for use as a public interface Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/db_1/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side spfile /dev/mpath/mpath0p3 System parameters with non-default values: large_pool_size = 12M spfile = "/dev/mpath/mpath0p3" instance_type = "asm" cluster_database = TRUE instance_number = 1 asm_diskstring = "ORCL:*" asm_diskgroups = "ORAFILES" asm_diskgroups = "FLASH" asm_diskgroups = "PRD01" asm_diskgroups = "REDOA" asm_diskgroups = "REDOB" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.1.1.185 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Sun Apr 21 04:57:01 2013 PMON started with pid=2, OS id=9892 Sun Apr 21 04:57:01 2013 VKTM started with pid=3, OS id=9909 at elevated priority Sun Apr 21 04:57:01 2013 DIAG started with pid=4, OS id=9922 Sun Apr 21 04:57:01 2013 PING started with pid=5, OS id=9924 VKTM running at (20)ms precision Sun Apr 21 04:57:01 2013 PSP0 started with pid=6, OS id=9926 Sun Apr 21 04:57:01 2013 DIA0 started with pid=7, OS id=9928 Sun Apr 21 04:57:01 2013 LMON started with pid=8, OS id=9933 Sun Apr 21 04:57:01 2013 LMD0 started with pid=9, OS id=9941 Sun Apr 21 04:57:01 2013 LMS0 started with pid=10, OS id=9943 at elevated priority Sun Apr 21 04:57:01 2013 MMAN started with pid=11, OS id=9947 Sun Apr 21 04:57:01 2013 DBW0 started with pid=12, OS id=9949 Sun Apr 21 04:57:01 2013 LGWR started with pid=13, OS id=9951 Sun Apr 21 04:57:01 2013 CKPT started with pid=14, OS id=9953 Sun Apr 21 04:57:01 2013 SMON started with pid=15, OS id=9956 Sun Apr 21 04:57:01 2013 RBAL started with pid=16, OS id=9958 Sun Apr 21 04:57:01 2013 GMON started with pid=17, OS id=9960 lmon registered with NM - instance id 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 12) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * allocate domain 1, invalid = TRUE * allocate domain 2, invalid = TRUE * allocate domain 3, invalid = TRUE * allocate domain 4, invalid = TRUE * allocate domain 5, invalid = TRUE * domain 0 valid = 1 according to instance 1 * domain 1 valid = 1 according to instance 1 * domain 2 valid = 1 according to instance 1 * domain 3 valid = 1 according to instance 1 * domain 4 valid = 1 according to instance 1 * domain 5 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Sun Apr 21 04:57:02 2013 LCK0 started with pid=18, OS id=10039 ORACLE_BASE from environment = /u01/app/oracle Sun Apr 21 04:57:04 2013 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group FLASH number=1 incarn=0x496943e1 NOTE: cache began mount (not first) of group FLASH number=1 incarn=0x496943e1 NOTE: cache registered group ORAFILES number=2 incarn=0x496943e2 NOTE: cache began mount (not first) of group ORAFILES number=2 incarn=0x496943e2 NOTE: cache registered group PRD01 number=3 incarn=0x496943e3 NOTE: cache began mount (not first) of group PRD01 number=3 incarn=0x496943e3 NOTE: cache registered group REDOA number=4 incarn=0x496943e4 NOTE: cache began mount (not first) of group REDOA number=4 incarn=0x496943e4 NOTE: cache registered group REDOB number=5 incarn=0x497943e5 NOTE: cache began mount (not first) of group REDOB number=5 incarn=0x497943e5 NOTE:Loaded lib: /opt/oracle/extapi/64/asm/orcl/1/libasm.so NOTE: Assigning number (1,0) to disk (ORCL:FLASH0) NOTE: Assigning number (2,0) to disk (ORCL:ORAFILES0) NOTE: Assigning number (2,1) to disk (ORCL:ORAFILES1) NOTE: Assigning number (2,2) to disk (ORCL:ORAFILES2) NOTE: Assigning number (3,0) to disk (ORCL:PRD010) NOTE: Assigning number (4,0) to disk (ORCL:REDOA0) NOTE: Assigning number (5,0) to disk (ORCL:REDOB0) kfdp_query(FLASH): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 1: FLASH_0000 label:FLASH0 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 1/0x496943E1 (FLASH) kjbdomatt send to node 1 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 NOTE: LGWR mounted thread 2 for disk group 1 NOTE: opening chunk 2 at fcn 0.118253327 ABA NOTE: seq=197 blk=3667 NOTE: cache mounting group 1/0x496943E1 (FLASH) succeeded NOTE: cache ending mount (success) of group FLASH number=1 incarn=0x496943e1 kfdp_query(ORAFILES): 7 kfdp_queryBg(): 7 NOTE: cache opening disk 0 of grp 2: ORAFILES_0000 label:ORAFILES0 NOTE: F1X0 found on disk 0 fcn 0.667 NOTE: cache opening disk 1 of grp 2: ORAFILES_0001 label:ORAFILES1 NOTE: cache opening disk 2 of grp 2: ORAFILES_0002 label:ORAFILES2 NOTE: cache mounting (not first) group 2/0x496943E2 (ORAFILES) kjbdomatt send to node 1 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 2 for diskgroup 2 NOTE: LGWR mounted thread 2 for disk group 2 NOTE: opening chunk 2 at fcn 0.38708417 ABA NOTE: seq=209 blk=5094 NOTE: cache mounting group 2/0x496943E2 (ORAFILES) succeeded NOTE: cache ending mount (success) of group ORAFILES number=2 incarn=0x496943e2 kfdp_query(PRD01): 8 kfdp_queryBg(): 8 NOTE: cache opening disk 0 of grp 3: PRD01_0000 label:PRD010 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 3/0x496943E3 (PRD01) kjbdomatt send to node 1 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 2 for diskgroup 3 NOTE: LGWR mounted thread 2 for disk group 3 NOTE: opening chunk 2 at fcn 0.4940641 ABA NOTE: seq=32 blk=7104 NOTE: cache mounting group 3/0x496943E3 (PRD01) succeeded NOTE: cache ending mount (success) of group PRD01 number=3 incarn=0x496943e3 kfdp_query(REDOA): 9 kfdp_queryBg(): 9 NOTE: cache opening disk 0 of grp 4: REDOA0 label:REDOA0 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 4/0x496943E4 (REDOA) kjbdomatt send to node 1 NOTE: attached to recovery domain 4 NOTE: LGWR attempting to mount thread 2 for diskgroup 4 NOTE: LGWR mounted thread 2 for disk group 4 NOTE: opening chunk 2 at fcn 0.75230 ABA NOTE: seq=6 blk=2967 NOTE: cache mounting group 4/0x496943E4 (REDOA) succeeded NOTE: cache ending mount (success) of group REDOA number=4 incarn=0x496943e4 kfdp_query(REDOB): 10 kfdp_queryBg(): 10 NOTE: cache opening disk 0 of grp 5: REDOB0 label:REDOB0 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache mounting (not first) group 5/0x497943E5 (REDOB) kjbdomatt send to node 1 NOTE: attached to recovery domain 5 NOTE: LGWR attempting to mount thread 2 for diskgroup 5 NOTE: LGWR mounted thread 2 for disk group 5 NOTE: opening chunk 2 at fcn 0.77573 ABA NOTE: seq=6 blk=2991 NOTE: cache mounting group 5/0x497943E5 (REDOB) succeeded NOTE: cache ending mount (success) of group REDOB number=5 incarn=0x497943e5 kfdp_query(FLASH): 11 kfdp_queryBg(): 11 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup FLASH was mounted kfdp_query(ORAFILES): 12 kfdp_queryBg(): 12 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup ORAFILES was mounted kfdp_query(PRD01): 13 kfdp_queryBg(): 13 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup PRD01 was mounted kfdp_query(REDOA): 14 kfdp_queryBg(): 14 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 4 SUCCESS: diskgroup REDOA was mounted kfdp_query(REDOB): 15 kfdp_queryBg(): 15 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 5 SUCCESS: diskgroup REDOB was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Sun Apr 21 04:57:18 2013 Starting background process ASMB Sun Apr 21 04:57:18 2013 ASMB started with pid=20, OS id=10659 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds Sun Apr 21 04:58:06 2013 Starting background process ASMB Sun Apr 21 04:58:06 2013 ASMB started with pid=27, OS id=12014 NOTE: ASMB process exiting due to lack of ASM file activity for 5 seconds