2010-07-15 05:32:51.221 [client(29676)]CRS-1006:The OCR location /dev/mpath/mpath0p1 is inaccessible. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/client/ocrconfig_29676.log. 2010-07-15 05:32:51.258 [client(29676)]CRS-1006:The OCR location /dev/mpath/mpath0p1 is inaccessible. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/client/ocrconfig_29676.log. 2010-07-15 05:32:51.278 [client(29676)]CRS-1006:The OCR location /dev/mpath/mpath0p1 is inaccessible. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/client/ocrconfig_29676.log. 2010-07-15 05:32:51.298 [client(29676)]CRS-1006:The OCR location /dev/mpath/mpath0p1 is inaccessible. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/client/ocrconfig_29676.log. 2010-07-15 05:32:51.432 [client(29676)]CRS-1013:The OCR at /dev/mpath/mpath0p1 was successfully formatted using version 2. Ignore earlier CRS-1006 messages if any. 2010-07-15 05:32:55.231 [client(29754)]CRS-1801:Cluster prod-rac configured with nodes prod-rac01 prod-rac02 . 2010-07-15 05:33:10.089 [cssd(31116)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. [cssd(31116)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 . [cssd(31116)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2010-07-15 05:37:26.264 [crsd(30653)]CRS-1012:The OCR service started on node prod-rac01. 2010-07-15 05:37:26.267 [evmd(30551)]CRS-1401:EVMD started on node prod-rac01. 2010-07-15 05:37:27.648 [crsd(30653)]CRS-1201:CRSD started on node prod-rac01. 2010-07-16 00:26:07.957 [evmd(10071)]CRS-1401:EVMD started on node prod-rac01. [cssd(31116)]CRS-1603:CSSD on node prod-rac01 shutdown by user. 2010-07-16 04:36:31.876 [cssd(5486)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2010-07-16 04:39:53.509 [cssd(5486)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 . 2010-07-16 04:39:55.026 [crsd(4880)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599488 to 185599488. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2010-07-16 04:39:55.027 [crsd(4880)]CRS-1012:The OCR service started on node prod-rac01. 2010-07-16 04:39:55.615 [evmd(4746)]CRS-1401:EVMD started on node prod-rac01. 2010-07-16 04:39:56.896 [crsd(4880)]CRS-1201:CRSD started on node prod-rac01. 2010-07-16 04:40:01.080 [crsd(4880)]CRS-1205:Auto-start failed for the CRS resource . Details in prod-rac01. 2010-07-16 04:40:01.092 [crsd(4880)]CRS-1205:Auto-start failed for the CRS resource . Details in prod-rac01. 2010-07-16 04:40:22.723 [client(7548)]CRS-1804:Node prod-rac01 upgraded to version 11.1.0.7.0. 2010-07-16 04:40:25.666 [cssd(5486)]CRS-1603:CSSD on node prod-rac01 shutdown by user. 2010-07-16 04:41:00.985 [cssd(9592)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2010-07-16 04:44:21.935 [cssd(9592)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 . 2010-07-16 04:44:23.742 [crsd(8770)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599488 to 185599488. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2010-07-16 04:44:23.742 [crsd(8770)]CRS-1012:The OCR service started on node prod-rac01. 2010-07-16 04:44:24.700 [evmd(8734)]CRS-1401:EVMD started on node prod-rac01. 2010-07-16 04:44:26.868 [crsd(8770)]CRS-1201:CRSD started on node prod-rac01. 2010-07-16 04:44:30.794 [crsd(8770)]CRS-1205:Auto-start failed for the CRS resource . Details in prod-rac01. 2010-07-16 04:44:30.814 [crsd(8770)]CRS-1205:Auto-start failed for the CRS resource . Details in prod-rac01. 2010-07-16 04:48:38.489 [cssd(9592)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2010-07-16 04:48:40.371 [crsd(8770)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599488 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2010-07-19 00:21:27.485 [cssd(9592)]CRS-1603:CSSD on node prod-rac01 shutdown by user. 2010-07-19 00:23:01.914 [cssd(23766)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2010-07-19 00:23:02.674 [cssd(23766)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2010-07-19 00:23:03.599 [evmd(23079)]CRS-1401:EVMD started on node prod-rac01. 2010-07-19 00:23:03.797 [crsd(23091)]CRS-1012:The OCR service started on node prod-rac01. 2010-07-19 00:23:06.431 [crsd(23091)]CRS-1201:CRSD started on node prod-rac01. 2010-07-19 04:26:10.992 [cssd(23766)]CRS-1603:CSSD on node prod-rac01 shutdown by user. 2010-07-19 04:28:15.462 [cssd(12631)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2010-07-19 04:31:36.395 [cssd(12631)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 . 2010-07-19 04:31:37.598 [crsd(11949)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2010-07-19 04:31:37.599 [crsd(11949)]CRS-1012:The OCR service started on node prod-rac01. 2010-07-19 04:31:38.325 [evmd(11835)]CRS-1401:EVMD started on node prod-rac01. 2010-07-19 04:31:40.199 [crsd(11949)]CRS-1201:CRSD started on node prod-rac01. 2010-07-19 04:31:41.135 [crsd(11949)]CRS-1205:Auto-start failed for the CRS resource . Details in prod-rac01. 2010-07-19 04:31:41.162 [crsd(11949)]CRS-1205:Auto-start failed for the CRS resource . Details in prod-rac01. 2010-07-19 04:32:46.054 [crsd(11949)]CRS-1205:Auto-start failed for the CRS resource . Details in prod-rac01. 2010-07-19 04:32:46.063 [crsd(11949)]CRS-1205:Auto-start failed for the CRS resource . Details in prod-rac01. 2010-07-19 04:32:50.017 [crsd(11949)]CRS-1205:Auto-start failed for the CRS resource . Details in prod-rac01. 2010-07-19 04:35:31.910 [cssd(12631)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2010-08-22 01:01:30.987 [cssd(12631)]CRS-1604:CSSD voting file is offline: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2010-08-22 01:01:31.311 [crsd(11949)]CRS-1006:The OCR location /dev/mpath/mpath0p1 is inaccessible. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2010-08-22 01:02:01.355 [cssd(12631)]CRS-1615:voting device hang at 50% fatal, termination in 99420 ms, disk (0//dev/mpath/mpath0p2) 2010-08-22 01:02:07.056 [cssd(12631)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2010-08-22 01:02:42.125 [crsd(22943)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2010-08-22 01:02:42.126 [crsd(22943)]CRS-1012:The OCR service started on node prod-rac01. 2010-08-22 01:02:53.501 [crsd(22943)]CRS-1201:CRSD started on node prod-rac01. 2010-08-22 01:03:26.009 [crsd(22943)]CRS-1205:Auto-start failed for the CRS resource . Details in prod-rac01. 2010-10-21 15:50:23.020 [cssd(12843)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2010-10-21 15:50:24.332 [cssd(12843)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2010-10-21 15:50:25.123 [crsd(12251)]CRS-1012:The OCR service started on node prod-rac01. 2010-10-21 15:50:25.713 [evmd(12253)]CRS-1401:EVMD started on node prod-rac01. 2010-10-21 15:50:28.074 [crsd(12251)]CRS-1201:CRSD started on node prod-rac01. 2010-10-21 15:53:03.669 [crsd(12251)]CRS-1205:Auto-start failed for the CRS resource . Details in prod-rac01. 2010-10-21 15:56:35.594 [crsd(12251)]CRS-1205:Auto-start failed for the CRS resource . Details in prod-rac01. 2010-11-19 02:27:51.341 [cssd(12830)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2010-11-19 02:27:52.373 [cssd(12830)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2010-11-19 02:27:53.170 [crsd(12205)]CRS-1012:The OCR service started on node prod-rac01. 2010-11-19 02:27:54.067 [evmd(12207)]CRS-1401:EVMD started on node prod-rac01. 2010-11-19 02:27:56.029 [crsd(12205)]CRS-1201:CRSD started on node prod-rac01. 2010-12-23 21:21:09.294 [cssd(12830)]CRS-1612:node prod-rac02 (2) at 50% heartbeat fatal, eviction in 14.840 seconds 2010-12-23 21:21:10.296 [cssd(12830)]CRS-1612:node prod-rac02 (2) at 50% heartbeat fatal, eviction in 13.840 seconds 2010-12-23 21:21:17.310 [cssd(12830)]CRS-1611:node prod-rac02 (2) at 75% heartbeat fatal, eviction in 6.820 seconds 2010-12-23 21:21:21.318 [cssd(12830)]CRS-1610:node prod-rac02 (2) at 90% heartbeat fatal, eviction in 2.820 seconds 2010-12-23 21:21:22.320 [cssd(12830)]CRS-1610:node prod-rac02 (2) at 90% heartbeat fatal, eviction in 1.820 seconds 2010-12-23 21:21:23.322 [cssd(12830)]CRS-1610:node prod-rac02 (2) at 90% heartbeat fatal, eviction in 0.810 seconds 2010-12-23 21:21:24.135 [cssd(12830)]CRS-1607:CSSD evicting node prod-rac02. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2010-12-23 21:21:55.202 [cssd(12830)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 . 2010-12-23 21:21:55.418 [crsd(12205)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2010-12-23 21:22:03.159 [crsd(12205)]CRS-1204:Recovering CRS resources for node prod-rac02. 2010-12-23 21:26:02.736 [cssd(12830)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2011-02-25 02:26:32.477 [cssd(13513)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2011-02-25 02:26:34.690 [cssd(13513)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2011-02-25 02:26:35.570 [crsd(12900)]CRS-1012:The OCR service started on node prod-rac01. 2011-02-25 02:26:36.310 [evmd(12879)]CRS-1401:EVMD started on node prod-rac01. 2011-02-25 02:26:39.761 [crsd(12900)]CRS-1201:CRSD started on node prod-rac01. 2011-02-25 04:05:27.189 [cssd(13513)]CRS-1603:CSSD on node prod-rac01 shutdown by user. 2011-02-25 04:10:16.196 [cssd(13738)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2011-02-25 04:10:17.116 [cssd(13738)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2011-02-25 04:10:17.738 [evmd(13105)]CRS-1401:EVMD started on node prod-rac01. 2011-02-25 04:10:17.791 [crsd(13119)]CRS-1012:The OCR service started on node prod-rac01. 2011-02-25 04:10:21.228 [crsd(13119)]CRS-1201:CRSD started on node prod-rac01. 2011-09-18 01:05:41.250 [cssd(13738)]CRS-1615:voting device hang at 50% fatal, termination in 99500 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:06:31.300 [cssd(13738)]CRS-1614:voting device hang at 75% fatal, termination in 49460 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:01.330 [cssd(13738)]CRS-1613:voting device hang at 90% fatal, termination in 19440 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:02.331 [cssd(13738)]CRS-1613:voting device hang at 90% fatal, termination in 18430 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:03.332 [cssd(13738)]CRS-1613:voting device hang at 90% fatal, termination in 17430 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:04.333 [cssd(13738)]CRS-1613:voting device hang at 90% fatal, termination in 16430 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:05.334 [cssd(13738)]CRS-1613:voting device hang at 90% fatal, termination in 15430 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:06.335 [cssd(13738)]CRS-1613:voting device hang at 90% fatal, termination in 14430 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:07.336 [cssd(13738)]CRS-1613:voting device hang at 90% fatal, termination in 13430 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:08.337 [cssd(13738)]CRS-1613:voting device hang at 90% fatal, termination in 12430 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:09.338 [cssd(13738)]CRS-1613:voting device hang at 90% fatal, termination in 11430 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:10.339 [cssd(13738)]CRS-1613:voting device hang at 90% fatal, termination in 10430 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:11.340 [cssd(13738)]CRS-1613:voting device hang at 90% fatal, termination in 9430 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:12.341 [cssd(13738)]CRS-1613:voting device hang at 90% fatal, termination in 8430 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:13.342 [cssd(13738)]CRS-1613:voting device hang at 90% fatal, termination in 7430 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:14.343 [cssd(13738)]CRS-1613:voting device hang at 90% fatal, termination in 6420 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:15.344 [cssd(13738)]CRS-1613:voting device hang at 90% fatal, termination in 5420 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:16.345 [cssd(13738)]CRS-1613:voting device hang at 90% fatal, termination in 4420 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:17.346 [cssd(13738)]CRS-1613:voting device hang at 90% fatal, termination in 3420 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:13:05.857 [cssd(13539)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2011-09-18 01:13:07.336 [cssd(13539)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2011-09-18 01:13:07.824 [crsd(12929)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2011-09-18 01:13:07.825 [crsd(12929)]CRS-1012:The OCR service started on node prod-rac01. 2011-09-18 01:13:07.829 [evmd(12917)]CRS-1401:EVMD started on node prod-rac01. 2011-09-18 01:13:09.456 [crsd(12929)]CRS-1201:CRSD started on node prod-rac01. 2011-09-25 03:03:54.832 [cssd(13539)]CRS-1603:CSSD on node prod-rac01 shutdown by user. 2011-09-25 03:08:44.972 [cssd(13596)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2011-09-25 03:08:46.274 [cssd(13596)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2011-09-25 03:08:46.954 [crsd(12998)]CRS-1012:The OCR service started on node prod-rac01. 2011-09-25 03:08:46.960 [evmd(12997)]CRS-1401:EVMD started on node prod-rac01. 2011-09-25 03:08:49.267 [crsd(12998)]CRS-1201:CRSD started on node prod-rac01. 2011-09-25 04:36:09.904 [cssd(6192)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2011-09-25 04:36:10.550 [cssd(6192)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2011-09-25 04:36:10.957 [crsd(5559)]CRS-1012:The OCR service started on node prod-rac01. 2011-09-25 04:36:10.981 [evmd(5558)]CRS-1401:EVMD started on node prod-rac01. 2011-09-25 04:36:15.650 [crsd(5559)]CRS-1201:CRSD started on node prod-rac01. 2011-09-25 05:13:32.858 [cssd(6192)]CRS-1612:node prod-rac02 (2) at 50% heartbeat fatal, eviction in 15.000 seconds 2011-09-25 05:13:33.858 [cssd(6192)]CRS-1612:node prod-rac02 (2) at 50% heartbeat fatal, eviction in 14.000 seconds 2011-09-25 05:13:33.867 [cssd(6192)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 . 2011-09-25 05:13:34.149 [crsd(5559)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2011-09-25 05:13:34.413 [crsd(5559)]CRS-1204:Recovering CRS resources for node prod-rac02. 2011-09-25 05:17:59.956 [cssd(6192)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2011-12-16 03:16:17.400 [cssd(6192)]CRS-1615:voting device hang at 50% fatal, termination in 99310 ms, disk (0//dev/mpath/mpath0p2) 2012-04-09 11:47:07.553 [cssd(6192)]CRS-1612:node prod-rac02 (2) at 50% heartbeat fatal, eviction in 14.550 seconds 2012-04-09 11:47:08.554 [cssd(6192)]CRS-1612:node prod-rac02 (2) at 50% heartbeat fatal, eviction in 13.550 seconds 2012-04-09 11:47:15.555 [cssd(6192)]CRS-1611:node prod-rac02 (2) at 75% heartbeat fatal, eviction in 6.550 seconds 2012-04-09 11:47:19.556 [cssd(6192)]CRS-1610:node prod-rac02 (2) at 90% heartbeat fatal, eviction in 2.550 seconds 2012-04-09 11:47:20.556 [cssd(6192)]CRS-1610:node prod-rac02 (2) at 90% heartbeat fatal, eviction in 1.550 seconds 2012-04-09 11:47:21.556 [cssd(6192)]CRS-1610:node prod-rac02 (2) at 90% heartbeat fatal, eviction in 0.550 seconds 2012-04-09 11:47:35.109 [cssd(6192)]CRS-1607:CSSD evicting node prod-rac02. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2012-04-09 11:47:38.142 [cssd(6192)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 . 2012-04-09 11:47:41.658 [crsd(5559)]CRS-1204:Recovering CRS resources for node prod-rac02. 2012-04-09 11:51:38.217 [cssd(6192)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2012-04-21 12:28:09.483 [cssd(6192)]CRS-1603:CSSD on node prod-rac01 shutdown by user. 2012-04-21 12:32:59.783 [cssd(6333)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2012-04-21 12:33:01.228 [cssd(6333)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2012-04-21 12:33:01.703 [crsd(5721)]CRS-1012:The OCR service started on node prod-rac01. 2012-04-21 12:33:02.584 [evmd(5668)]CRS-1401:EVMD started on node prod-rac01. 2012-04-21 12:33:04.111 [crsd(5721)]CRS-1201:CRSD started on node prod-rac01. 2012-05-27 03:39:13.139 [cssd(6333)]CRS-1603:CSSD on node prod-rac01 shutdown by user. 2012-05-27 03:45:13.580 [cssd(6298)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2012-05-27 03:45:15.186 [cssd(6298)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2012-05-27 03:45:15.738 [crsd(5687)]CRS-1012:The OCR service started on node prod-rac01. 2012-05-27 03:45:16.603 [evmd(5671)]CRS-1401:EVMD started on node prod-rac01. 2012-05-27 03:45:18.801 [crsd(5687)]CRS-1201:CRSD started on node prod-rac01. 2012-05-27 03:49:07.953 [cssd(6298)]CRS-1603:CSSD on node prod-rac01 shutdown by user. 2012-05-27 03:54:43.506 [cssd(6269)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2012-05-27 03:54:44.260 [cssd(6269)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2012-05-27 03:54:44.691 [crsd(5664)]CRS-1012:The OCR service started on node prod-rac01. 2012-05-27 03:54:45.455 [evmd(5634)]CRS-1401:EVMD started on node prod-rac01. 2012-05-27 03:54:46.984 [crsd(5664)]CRS-1201:CRSD started on node prod-rac01. 2012-05-27 04:28:40.509 [crsd(5664)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2012-05-27 04:28:44.440 [cssd(6269)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 . 2012-05-27 04:28:48.323 [crsd(5664)]CRS-1204:Recovering CRS resources for node prod-rac02. 2012-05-27 04:33:52.554 [cssd(6269)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2012-05-27 04:44:02.731 [cssd(6269)]CRS-1603:CSSD on node prod-rac01 shutdown by user. 2012-05-27 05:07:49.131 [cssd(16786)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2012-05-27 05:07:50.300 [cssd(16786)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2012-05-27 05:07:50.939 [crsd(15924)]CRS-1012:The OCR service started on node prod-rac01. 2012-05-27 05:07:52.070 [evmd(15746)]CRS-1401:EVMD started on node prod-rac01. 2012-05-27 05:07:53.296 [crsd(15924)]CRS-1201:CRSD started on node prod-rac01. 2012-05-27 05:10:09.509 [crsd(15924)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2012-05-27 05:10:10.898 [cssd(16786)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 . 2012-05-27 05:10:15.545 [crsd(15924)]CRS-1204:Recovering CRS resources for node prod-rac02. 2012-05-27 05:13:11.986 [cssd(16786)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2012-06-08 01:18:17.840 [cssd(16786)]CRS-1615:voting device hang at 50% fatal, termination in 99100 ms, disk (0//dev/mpath/mpath0p2) 2012-07-16 01:27:32.111 [cssd(16786)]CRS-1615:voting device hang at 50% fatal, termination in 99290 ms, disk (0//dev/mpath/mpath0p2) 2012-09-13 03:29:03.821 [crsd(4271)]CRS-1012:The OCR service started on node prod-rac01. 2012-09-13 03:29:08.838 [crsd(4271)]CRS-1201:CRSD started on node prod-rac01. 2012-09-16 02:15:20.339 [cssd(16786)]CRS-1603:CSSD on node prod-rac01 shutdown by user. 2012-09-16 02:20:21.271 [cssd(7710)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2012-09-16 02:20:23.021 [cssd(7710)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2012-09-16 02:20:23.773 [crsd(6822)]CRS-1012:The OCR service started on node prod-rac01. 2012-09-16 02:20:24.562 [evmd(6824)]CRS-1401:EVMD started on node prod-rac01. 2012-09-16 02:20:25.761 [crsd(6822)]CRS-1201:CRSD started on node prod-rac01. 2012-09-16 02:23:58.723 [crsd(6822)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2012-09-16 02:23:59.904 [cssd(7710)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 . 2012-09-16 02:24:03.838 [crsd(6822)]CRS-1204:Recovering CRS resources for node prod-rac02. 2012-09-16 02:28:54.016 [cssd(7710)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2013-01-16 11:29:05.473 [cssd(7710)]CRS-1615:voting device hang at 50% fatal, termination in 99230 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:29:55.499 [cssd(7710)]CRS-1614:voting device hang at 75% fatal, termination in 49220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:25.502 [cssd(7710)]CRS-1613:voting device hang at 90% fatal, termination in 19220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:26.503 [cssd(7710)]CRS-1613:voting device hang at 90% fatal, termination in 18220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:27.503 [cssd(7710)]CRS-1613:voting device hang at 90% fatal, termination in 17220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:28.503 [cssd(7710)]CRS-1613:voting device hang at 90% fatal, termination in 16220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:29.503 [cssd(7710)]CRS-1613:voting device hang at 90% fatal, termination in 15220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:30.503 [cssd(7710)]CRS-1613:voting device hang at 90% fatal, termination in 14220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:31.503 [cssd(7710)]CRS-1613:voting device hang at 90% fatal, termination in 13220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:32.504 [cssd(7710)]CRS-1613:voting device hang at 90% fatal, termination in 12220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:33.504 [cssd(7710)]CRS-1613:voting device hang at 90% fatal, termination in 11220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:34.504 [cssd(7710)]CRS-1613:voting device hang at 90% fatal, termination in 10220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:35.504 [cssd(7710)]CRS-1613:voting device hang at 90% fatal, termination in 9220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:36.504 [cssd(7710)]CRS-1613:voting device hang at 90% fatal, termination in 8220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:37.504 [cssd(7710)]CRS-1613:voting device hang at 90% fatal, termination in 7220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:38.504 [cssd(7710)]CRS-1613:voting device hang at 90% fatal, termination in 6220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:39.504 [cssd(7710)]CRS-1613:voting device hang at 90% fatal, termination in 5220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:40.505 [cssd(7710)]CRS-1613:voting device hang at 90% fatal, termination in 4220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:41.505 [cssd(7710)]CRS-1613:voting device hang at 90% fatal, termination in 3220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:42.505 [cssd(7710)]CRS-1613:voting device hang at 90% fatal, termination in 2220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:43.505 [cssd(7710)]CRS-1613:voting device hang at 90% fatal, termination in 1220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:44.505 [cssd(7710)]CRS-1613:voting device hang at 90% fatal, termination in 220 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:35:28.988 [cssd(7452)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2013-01-16 11:35:55.231 [cssd(7452)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2013-01-16 11:35:56.784 [crsd(6567)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2013-01-16 11:35:56.784 [crsd(6567)]CRS-1012:The OCR service started on node prod-rac01. 2013-01-16 11:35:56.793 [evmd(6566)]CRS-1401:EVMD started on node prod-rac01. 2013-01-16 11:35:58.395 [crsd(6567)]CRS-1201:CRSD started on node prod-rac01. 2013-01-16 11:49:07.311 [cssd(7452)]CRS-1615:voting device hang at 50% fatal, termination in 99470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:49:57.318 [cssd(7452)]CRS-1614:voting device hang at 75% fatal, termination in 49470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:27.320 [cssd(7452)]CRS-1613:voting device hang at 90% fatal, termination in 19470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:28.320 [cssd(7452)]CRS-1613:voting device hang at 90% fatal, termination in 18470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:29.320 [cssd(7452)]CRS-1613:voting device hang at 90% fatal, termination in 17470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:30.321 [cssd(7452)]CRS-1613:voting device hang at 90% fatal, termination in 16470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:31.321 [cssd(7452)]CRS-1613:voting device hang at 90% fatal, termination in 15470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:32.321 [cssd(7452)]CRS-1613:voting device hang at 90% fatal, termination in 14470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:33.321 [cssd(7452)]CRS-1613:voting device hang at 90% fatal, termination in 13470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:34.321 [cssd(7452)]CRS-1613:voting device hang at 90% fatal, termination in 12470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:35.321 [cssd(7452)]CRS-1613:voting device hang at 90% fatal, termination in 11470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:36.321 [cssd(7452)]CRS-1613:voting device hang at 90% fatal, termination in 10470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:37.322 [cssd(7452)]CRS-1613:voting device hang at 90% fatal, termination in 9470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:38.322 [cssd(7452)]CRS-1613:voting device hang at 90% fatal, termination in 8470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:39.322 [cssd(7452)]CRS-1613:voting device hang at 90% fatal, termination in 7470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:40.322 [cssd(7452)]CRS-1613:voting device hang at 90% fatal, termination in 6470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:41.322 [cssd(7452)]CRS-1613:voting device hang at 90% fatal, termination in 5470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:42.322 [cssd(7452)]CRS-1613:voting device hang at 90% fatal, termination in 4470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:43.322 [cssd(7452)]CRS-1613:voting device hang at 90% fatal, termination in 3470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:44.322 [cssd(7452)]CRS-1613:voting device hang at 90% fatal, termination in 2470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:45.323 [cssd(7452)]CRS-1613:voting device hang at 90% fatal, termination in 1470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:46.323 [cssd(7452)]CRS-1613:voting device hang at 90% fatal, termination in 470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:14:51.051 [cssd(8710)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2013-01-16 12:14:58.082 [cssd(8710)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2013-01-16 12:14:58.611 [crsd(7838)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2013-01-16 12:14:58.612 [crsd(7838)]CRS-1012:The OCR service started on node prod-rac01. 2013-01-16 12:14:59.364 [evmd(7782)]CRS-1401:EVMD started on node prod-rac01. 2013-01-16 12:15:01.214 [crsd(7838)]CRS-1201:CRSD started on node prod-rac01. 2013-01-16 12:15:17.304 [crsd(7838)]CRS-1205:Auto-start failed for the CRS resource ora.HWPROD.HWPROD1.inst. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2013-01-16 12:15:18.611 [crsd(7838)]CRS-1205:Auto-start failed for the CRS resource ora.HWPROD.HWPROD2.inst. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2013-01-16 12:20:33.590 [crsd(7838)]CRS-1205:Auto-start failed for the CRS resource ora.HWPROD.db. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2013-01-16 12:33:49.777 [cssd(8710)]CRS-1615:voting device hang at 50% fatal, termination in 99950 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:34:39.785 [cssd(8710)]CRS-1614:voting device hang at 75% fatal, termination in 49950 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:09.828 [cssd(8710)]CRS-1613:voting device hang at 90% fatal, termination in 19920 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:10.829 [cssd(8710)]CRS-1613:voting device hang at 90% fatal, termination in 18920 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:11.829 [cssd(8710)]CRS-1613:voting device hang at 90% fatal, termination in 17920 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:12.829 [cssd(8710)]CRS-1613:voting device hang at 90% fatal, termination in 16920 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:13.829 [cssd(8710)]CRS-1613:voting device hang at 90% fatal, termination in 15910 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:14.829 [cssd(8710)]CRS-1613:voting device hang at 90% fatal, termination in 14910 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:15.839 [cssd(8710)]CRS-1613:voting device hang at 90% fatal, termination in 13910 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:16.840 [cssd(8710)]CRS-1613:voting device hang at 90% fatal, termination in 12910 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:17.840 [cssd(8710)]CRS-1613:voting device hang at 90% fatal, termination in 11910 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:18.840 [cssd(8710)]CRS-1613:voting device hang at 90% fatal, termination in 10910 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:19.840 [cssd(8710)]CRS-1613:voting device hang at 90% fatal, termination in 9900 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:20.840 [cssd(8710)]CRS-1613:voting device hang at 90% fatal, termination in 8900 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:21.840 [cssd(8710)]CRS-1613:voting device hang at 90% fatal, termination in 7900 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:22.851 [cssd(8710)]CRS-1613:voting device hang at 90% fatal, termination in 6900 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:23.851 [cssd(8710)]CRS-1613:voting device hang at 90% fatal, termination in 5900 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:24.851 [cssd(8710)]CRS-1613:voting device hang at 90% fatal, termination in 4890 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:25.851 [cssd(8710)]CRS-1613:voting device hang at 90% fatal, termination in 3890 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:26.851 [cssd(8710)]CRS-1613:voting device hang at 90% fatal, termination in 2890 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:27.851 [cssd(8710)]CRS-1613:voting device hang at 90% fatal, termination in 1890 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:28.851 [cssd(8710)]CRS-1613:voting device hang at 90% fatal, termination in 890 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:40:36.193 [cssd(6898)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2013-01-16 12:40:39.784 [cssd(6898)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2013-01-16 12:40:40.217 [crsd(6059)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2013-01-16 12:40:40.218 [crsd(6059)]CRS-1012:The OCR service started on node prod-rac01. 2013-01-16 12:40:40.978 [evmd(6061)]CRS-1401:EVMD started on node prod-rac01. 2013-01-16 12:40:42.073 [crsd(6059)]CRS-1201:CRSD started on node prod-rac01. 2013-01-16 12:41:23.005 [crsd(6059)]CRS-1205:Auto-start failed for the CRS resource ora.HWPROD.HWPROD1.inst. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2013-01-16 12:42:08.371 [crsd(6059)]CRS-1205:Auto-start failed for the CRS resource ora.HWPROD.HWPROD2.inst. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2013-01-16 12:42:53.159 [crsd(6059)]CRS-1205:Auto-start failed for the CRS resource ora.HWPROD.db. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2013-05-05 02:14:42.625 [cssd(6898)]CRS-1603:CSSD on node prod-rac01 shutdown by user. 2013-05-05 02:20:21.094 [cssd(8662)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/cssd/ocssd.log. 2013-05-05 02:20:22.407 [cssd(8662)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2013-05-05 02:20:22.970 [crsd(7671)]CRS-1012:The OCR service started on node prod-rac01. 2013-05-05 02:20:23.974 [evmd(7638)]CRS-1401:EVMD started on node prod-rac01. 2013-05-05 02:20:25.726 [crsd(7671)]CRS-1201:CRSD started on node prod-rac01. 2013-05-05 02:27:09.248 [crsd(7671)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac01/crsd/crsd.log. 2013-05-05 02:27:09.624 [cssd(8662)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 . 2013-05-05 02:27:13.394 [crsd(7671)]CRS-1204:Recovering CRS resources for node prod-rac02. 2013-05-05 02:32:38.731 [cssd(8662)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2013-05-23 11:05:17.264 [cssd(8662)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 . 2013-05-23 11:05:20.811 [crsd(7671)]CRS-1204:Recovering CRS resources for node prod-rac02. 2013-05-23 11:10:39.367 [cssd(8662)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 .