2010-07-15 05:37:24.297 [cssd(27783)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. [cssd(27783)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2010-07-15 05:37:25.572 [crsd(27319)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599488 to 185599488. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/crsd/crsd.log. 2010-07-15 05:37:25.574 [crsd(27319)]CRS-1012:The OCR service started on node prod-rac02. 2010-07-15 05:37:26.425 [evmd(27227)]CRS-1401:EVMD started on node prod-rac02. 2010-07-15 05:37:27.851 [crsd(27319)]CRS-1201:CRSD started on node prod-rac02. 2010-07-16 02:55:44.244 [cssd(27783)]CRS-1612:node prod-rac01 (1) at 50% heartbeat fatal, eviction in 14.094 seconds 2010-07-16 02:55:51.258 [cssd(27783)]CRS-1611:node prod-rac01 (1) at 75% heartbeat fatal, eviction in 7.074 seconds 2010-07-16 02:55:52.260 [cssd(27783)]CRS-1611:node prod-rac01 (1) at 75% heartbeat fatal, eviction in 6.074 seconds 2010-07-16 02:55:56.268 [cssd(27783)]CRS-1610:node prod-rac01 (1) at 90% heartbeat fatal, eviction in 2.064 seconds 2010-07-16 02:55:57.270 [cssd(27783)]CRS-1610:node prod-rac01 (1) at 90% heartbeat fatal, eviction in 1.064 seconds 2010-07-16 02:55:58.272 [cssd(27783)]CRS-1610:node prod-rac01 (1) at 90% heartbeat fatal, eviction in 0.064 seconds 2010-07-16 02:55:58.595 [cssd(27783)]CRS-1607:CSSD evicting node prod-rac01. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. [cssd(27783)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac02 . 2010-07-16 02:55:58.623 [crsd(27319)]CRS-1204:Recovering CRS resources for node prod-rac01. [cssd(27783)]CRS-1603:CSSD on node prod-rac02 shutdown by user. 2010-07-16 04:48:37.293 [cssd(4076)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. 2010-07-16 04:48:38.488 [cssd(4076)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2010-07-16 04:48:40.353 [crsd(3443)]CRS-1012:The OCR service started on node prod-rac02. 2010-07-16 04:48:40.950 [evmd(3341)]CRS-1401:EVMD started on node prod-rac02. 2010-07-16 04:48:43.659 [crsd(3443)]CRS-1201:CRSD started on node prod-rac02. 2010-07-16 04:49:06.867 [client(4565)]CRS-1804:Node prod-rac02 upgraded to version 11.1.0.7.0. 2010-07-19 00:21:27.621 [crsd(3443)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/crsd/crsd.log. 2010-07-19 00:21:30.649 [cssd(4076)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac02 . 2010-07-19 00:21:34.256 [crsd(3443)]CRS-1204:Recovering CRS resources for node prod-rac01. 2010-07-19 00:23:02.673 [cssd(4076)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2010-07-19 04:26:13.824 [cssd(4076)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac02 . 2010-07-19 04:26:17.255 [crsd(3443)]CRS-1204:Recovering CRS resources for node prod-rac01. 2010-07-19 04:27:54.632 [cssd(4076)]CRS-1603:CSSD on node prod-rac02 shutdown by user. 2010-07-19 04:35:30.506 [cssd(15426)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. 2010-07-19 04:35:31.908 [cssd(15426)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2010-07-19 04:35:32.510 [crsd(14688)]CRS-1012:The OCR service started on node prod-rac02. 2010-07-19 04:35:32.513 [evmd(14747)]CRS-1401:EVMD started on node prod-rac02. 2010-07-19 04:35:34.628 [crsd(14688)]CRS-1201:CRSD started on node prod-rac02. 2010-08-22 01:01:30.605 [cssd(15426)]CRS-1604:CSSD voting file is offline: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. 2010-08-22 01:02:01.868 [cssd(15426)]CRS-1615:voting device hang at 50% fatal, termination in 99200 ms, disk (0//dev/mpath/mpath0p2) 2010-08-22 01:02:06.672 [cssd(15426)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. 2010-08-22 01:02:42.970 [crsd(8201)]CRS-1012:The OCR service started on node prod-rac02. 2010-08-22 01:02:59.498 [crsd(8201)]CRS-1201:CRSD started on node prod-rac02. 2010-10-21 15:45:46.769 [cssd(15426)]CRS-1607:CSSD evicting node prod-rac01. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. 2010-10-21 15:45:47.775 [cssd(15426)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac02 . 2010-10-21 15:45:47.953 [crsd(8201)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/crsd/crsd.log. 2010-10-21 15:45:54.397 [crsd(8201)]CRS-1204:Recovering CRS resources for node prod-rac01. 2010-10-21 15:50:24.339 [cssd(15426)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2010-11-19 02:23:15.143 [cssd(15426)]CRS-1607:CSSD evicting node prod-rac01. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. 2010-11-19 02:23:16.150 [cssd(15426)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac02 . 2010-11-19 02:23:16.251 [crsd(8201)]CRS-1204:Recovering CRS resources for node prod-rac01. 2010-11-19 02:27:52.384 [cssd(15426)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2010-12-23 21:21:22.834 [cssd(15426)]CRS-1612:node prod-rac01 (1) at 50% heartbeat fatal, eviction in 14.470 seconds 2010-12-23 21:26:01.254 [cssd(12829)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. 2010-12-23 21:26:02.723 [cssd(12829)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2010-12-23 21:26:03.273 [crsd(12210)]CRS-1012:The OCR service started on node prod-rac02. 2010-12-23 21:26:03.303 [evmd(12209)]CRS-1401:EVMD started on node prod-rac02. 2010-12-23 21:26:05.454 [crsd(12210)]CRS-1201:CRSD started on node prod-rac02. 2011-02-25 02:21:48.349 [cssd(12829)]CRS-1607:CSSD evicting node prod-rac01. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. 2011-02-25 02:21:49.355 [cssd(12829)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac02 . 2011-02-25 02:21:49.620 [crsd(12210)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/crsd/crsd.log. 2011-02-25 02:21:53.047 [crsd(12210)]CRS-1204:Recovering CRS resources for node prod-rac01. 2011-02-25 02:26:34.689 [cssd(12829)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2011-02-25 04:05:41.566 [cssd(12829)]CRS-1612:node prod-rac01 (1) at 50% heartbeat fatal, eviction in 14.160 seconds 2011-02-25 04:05:42.566 [cssd(12829)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac02 . 2011-02-25 04:05:42.615 [crsd(12210)]CRS-1204:Recovering CRS resources for node prod-rac01. 2011-02-25 04:10:16.623 [cssd(12829)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2011-09-18 01:05:41.124 [cssd(12829)]CRS-1615:voting device hang at 50% fatal, termination in 99860 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:06:31.174 [cssd(12829)]CRS-1614:voting device hang at 75% fatal, termination in 49820 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:01.204 [cssd(12829)]CRS-1613:voting device hang at 90% fatal, termination in 19790 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:02.205 [cssd(12829)]CRS-1613:voting device hang at 90% fatal, termination in 18790 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:03.206 [cssd(12829)]CRS-1613:voting device hang at 90% fatal, termination in 17790 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:04.207 [cssd(12829)]CRS-1613:voting device hang at 90% fatal, termination in 16790 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:05.208 [cssd(12829)]CRS-1613:voting device hang at 90% fatal, termination in 15790 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:06.209 [cssd(12829)]CRS-1613:voting device hang at 90% fatal, termination in 14790 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:07.210 [cssd(12829)]CRS-1613:voting device hang at 90% fatal, termination in 13790 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:08.211 [cssd(12829)]CRS-1613:voting device hang at 90% fatal, termination in 12790 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:09.212 [cssd(12829)]CRS-1613:voting device hang at 90% fatal, termination in 11790 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:10.213 [cssd(12829)]CRS-1613:voting device hang at 90% fatal, termination in 10790 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:11.214 [cssd(12829)]CRS-1613:voting device hang at 90% fatal, termination in 9790 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:12.215 [cssd(12829)]CRS-1613:voting device hang at 90% fatal, termination in 8780 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:13.216 [cssd(12829)]CRS-1613:voting device hang at 90% fatal, termination in 7780 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:14.217 [cssd(12829)]CRS-1613:voting device hang at 90% fatal, termination in 6780 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:15.218 [cssd(12829)]CRS-1613:voting device hang at 90% fatal, termination in 5780 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:16.219 [cssd(12829)]CRS-1613:voting device hang at 90% fatal, termination in 4780 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:17.220 [cssd(12829)]CRS-1613:voting device hang at 90% fatal, termination in 3780 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:07:18.221 [cssd(12829)]CRS-1613:voting device hang at 90% fatal, termination in 2780 ms, disk (0//dev/mpath/mpath0p2) 2011-09-18 01:12:59.803 [cssd(13179)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. 2011-09-18 01:13:07.337 [cssd(13179)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2011-09-18 01:13:07.799 [evmd(12557)]CRS-1401:EVMD started on node prod-rac02. 2011-09-18 01:13:07.928 [crsd(12558)]CRS-1012:The OCR service started on node prod-rac02. 2011-09-18 01:13:09.384 [crsd(12558)]CRS-1201:CRSD started on node prod-rac02. 2011-09-25 03:04:08.727 [cssd(13179)]CRS-1612:node prod-rac01 (1) at 50% heartbeat fatal, eviction in 14.890 seconds 2011-09-25 03:04:09.729 [cssd(13179)]CRS-1612:node prod-rac01 (1) at 50% heartbeat fatal, eviction in 13.890 seconds 2011-09-25 03:04:09.738 [cssd(13179)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac02 . 2011-09-25 03:04:09.881 [crsd(12558)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/crsd/crsd.log. 2011-09-25 03:04:14.856 [crsd(12558)]CRS-1204:Recovering CRS resources for node prod-rac01. 2011-09-25 03:08:45.796 [cssd(13179)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2011-09-25 04:31:43.513 [cssd(13179)]CRS-1612:node prod-rac01 (1) at 50% heartbeat fatal, eviction in 14.280 seconds 2011-09-25 04:31:44.523 [cssd(13179)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac02 . 2011-09-25 04:31:48.055 [crsd(12558)]CRS-1204:Recovering CRS resources for node prod-rac01. 2011-09-25 04:36:10.578 [cssd(13179)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2011-09-25 05:13:18.865 [cssd(13179)]CRS-1603:CSSD on node prod-rac02 shutdown by user. 2011-09-25 05:17:58.767 [cssd(5966)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. 2011-09-25 05:17:59.932 [cssd(5966)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2011-09-25 05:18:00.454 [crsd(5389)]CRS-1012:The OCR service started on node prod-rac02. 2011-09-25 05:18:01.269 [evmd(5368)]CRS-1401:EVMD started on node prod-rac02. 2011-09-25 05:18:03.041 [crsd(5389)]CRS-1201:CRSD started on node prod-rac02. 2012-02-13 21:51:26.805 [crsd(29364)]CRS-1012:The OCR service started on node prod-rac02. 2012-02-13 21:51:27.699 [crsd(29364)]CRS-1201:CRSD started on node prod-rac02. 2012-04-09 11:51:36.623 [cssd(6154)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. 2012-04-09 11:51:38.196 [cssd(6154)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2012-04-09 11:51:38.706 [crsd(5555)]CRS-1012:The OCR service started on node prod-rac02. 2012-04-09 11:51:39.624 [evmd(5523)]CRS-1401:EVMD started on node prod-rac02. 2012-04-09 11:51:41.236 [crsd(5555)]CRS-1201:CRSD started on node prod-rac02. 2012-04-21 12:28:23.176 [cssd(6154)]CRS-1612:node prod-rac01 (1) at 50% heartbeat fatal, eviction in 14.360 seconds 2012-04-21 12:28:24.185 [cssd(6154)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac02 . 2012-04-21 12:28:24.497 [crsd(5555)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/crsd/crsd.log. 2012-04-21 12:28:28.724 [crsd(5555)]CRS-1204:Recovering CRS resources for node prod-rac01. 2012-04-21 12:33:01.244 [cssd(6154)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2012-05-27 03:39:17.142 [cssd(6154)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac02 . 2012-05-27 03:39:17.182 [crsd(5555)]CRS-1204:Recovering CRS resources for node prod-rac01. 2012-05-27 03:45:15.205 [cssd(6154)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2012-05-27 03:49:12.224 [cssd(6154)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac02 . 2012-05-27 03:49:15.806 [crsd(5555)]CRS-1204:Recovering CRS resources for node prod-rac01. 2012-05-27 03:54:44.295 [cssd(6154)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2012-05-27 04:28:40.300 [cssd(6154)]CRS-1603:CSSD on node prod-rac02 shutdown by user. 2012-05-27 04:33:51.292 [cssd(5898)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. 2012-05-27 04:33:52.522 [cssd(5898)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2012-05-27 04:33:53.168 [crsd(5290)]CRS-1012:The OCR service started on node prod-rac02. 2012-05-27 04:33:53.910 [evmd(5289)]CRS-1401:EVMD started on node prod-rac02. 2012-05-27 04:33:56.175 [crsd(5290)]CRS-1201:CRSD started on node prod-rac02. 2012-05-27 04:44:02.810 [crsd(5290)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/crsd/crsd.log. 2012-05-27 04:44:06.048 [cssd(5898)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac02 . 2012-05-27 04:44:10.231 [crsd(5290)]CRS-1204:Recovering CRS resources for node prod-rac01. 2012-05-27 05:07:50.246 [cssd(5898)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2012-05-27 05:10:09.102 [cssd(5898)]CRS-1603:CSSD on node prod-rac02 shutdown by user. 2012-05-27 05:13:10.298 [cssd(584)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. 2012-05-27 05:13:11.923 [cssd(584)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2012-05-27 05:13:12.290 [crsd(32099)]CRS-1012:The OCR service started on node prod-rac02. 2012-05-27 05:13:13.128 [evmd(32096)]CRS-1401:EVMD started on node prod-rac02. 2012-05-27 05:13:14.046 [crsd(32099)]CRS-1201:CRSD started on node prod-rac02. 2012-09-10 01:02:23.789 [cssd(584)]CRS-1615:voting device hang at 50% fatal, termination in 99830 ms, disk (0//dev/mpath/mpath0p2) 2012-09-13 03:29:02.575 [crsd(32099)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/crsd/crsd.log. 2012-09-16 02:15:20.960 [cssd(584)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac02 . 2012-09-16 02:15:24.635 [crsd(32099)]CRS-1204:Recovering CRS resources for node prod-rac01. 2012-09-16 02:20:23.037 [cssd(584)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2012-09-16 02:23:58.547 [cssd(584)]CRS-1603:CSSD on node prod-rac02 shutdown by user. 2012-09-16 02:28:53.128 [cssd(7413)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. 2012-09-16 02:28:54.007 [cssd(7413)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2012-09-16 02:28:54.617 [crsd(6513)]CRS-1012:The OCR service started on node prod-rac02. 2012-09-16 02:28:55.488 [evmd(6526)]CRS-1401:EVMD started on node prod-rac02. 2012-09-16 02:28:56.808 [crsd(6513)]CRS-1201:CRSD started on node prod-rac02. 2012-10-01 09:53:40.962 [crsd(7598)]CRS-1012:The OCR service started on node prod-rac02. 2012-10-01 09:53:43.908 [crsd(7598)]CRS-1201:CRSD started on node prod-rac02. 2013-01-10 06:15:36.106 [crsd(8155)]CRS-1012:The OCR service started on node prod-rac02. 2013-01-10 06:15:39.821 [crsd(8155)]CRS-1201:CRSD started on node prod-rac02. 2013-01-16 11:29:06.351 [cssd(7413)]CRS-1615:voting device hang at 50% fatal, termination in 99470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:29:56.360 [cssd(7413)]CRS-1614:voting device hang at 75% fatal, termination in 49470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:26.365 [cssd(7413)]CRS-1613:voting device hang at 90% fatal, termination in 19470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:27.365 [cssd(7413)]CRS-1613:voting device hang at 90% fatal, termination in 18470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:28.365 [cssd(7413)]CRS-1613:voting device hang at 90% fatal, termination in 17470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:29.365 [cssd(7413)]CRS-1613:voting device hang at 90% fatal, termination in 16470 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:30.365 [cssd(7413)]CRS-1613:voting device hang at 90% fatal, termination in 15460 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:31.376 [cssd(7413)]CRS-1613:voting device hang at 90% fatal, termination in 14460 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:32.376 [cssd(7413)]CRS-1613:voting device hang at 90% fatal, termination in 13460 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:33.376 [cssd(7413)]CRS-1613:voting device hang at 90% fatal, termination in 12460 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:34.376 [cssd(7413)]CRS-1613:voting device hang at 90% fatal, termination in 11460 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:35.376 [cssd(7413)]CRS-1613:voting device hang at 90% fatal, termination in 10450 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:36.376 [cssd(7413)]CRS-1613:voting device hang at 90% fatal, termination in 9450 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:37.376 [cssd(7413)]CRS-1613:voting device hang at 90% fatal, termination in 8450 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:38.387 [cssd(7413)]CRS-1613:voting device hang at 90% fatal, termination in 7450 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:39.387 [cssd(7413)]CRS-1613:voting device hang at 90% fatal, termination in 6450 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:40.387 [cssd(7413)]CRS-1613:voting device hang at 90% fatal, termination in 5450 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:41.387 [cssd(7413)]CRS-1613:voting device hang at 90% fatal, termination in 4440 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:42.387 [cssd(7413)]CRS-1613:voting device hang at 90% fatal, termination in 3440 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:43.387 [cssd(7413)]CRS-1613:voting device hang at 90% fatal, termination in 2440 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:44.398 [cssd(7413)]CRS-1613:voting device hang at 90% fatal, termination in 1440 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:30:45.398 [cssd(7413)]CRS-1613:voting device hang at 90% fatal, termination in 440 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:35:52.665 [cssd(7234)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. 2013-01-16 11:35:55.246 [cssd(7234)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2013-01-16 11:35:57.007 [crsd(6332)]CRS-1012:The OCR service started on node prod-rac02. 2013-01-16 11:35:58.085 [evmd(6341)]CRS-1401:EVMD started on node prod-rac02. 2013-01-16 11:36:00.841 [crsd(6332)]CRS-1201:CRSD started on node prod-rac02. 2013-01-16 11:49:08.332 [cssd(7234)]CRS-1615:voting device hang at 50% fatal, termination in 99110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:49:58.339 [cssd(7234)]CRS-1614:voting device hang at 75% fatal, termination in 49110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:28.344 [cssd(7234)]CRS-1613:voting device hang at 90% fatal, termination in 19110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:29.344 [cssd(7234)]CRS-1613:voting device hang at 90% fatal, termination in 18110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:30.344 [cssd(7234)]CRS-1613:voting device hang at 90% fatal, termination in 17110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:31.344 [cssd(7234)]CRS-1613:voting device hang at 90% fatal, termination in 16110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:32.344 [cssd(7234)]CRS-1613:voting device hang at 90% fatal, termination in 15110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:33.345 [cssd(7234)]CRS-1613:voting device hang at 90% fatal, termination in 14110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:34.345 [cssd(7234)]CRS-1613:voting device hang at 90% fatal, termination in 13110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:35.345 [cssd(7234)]CRS-1613:voting device hang at 90% fatal, termination in 12110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:36.345 [cssd(7234)]CRS-1613:voting device hang at 90% fatal, termination in 11110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:37.345 [cssd(7234)]CRS-1613:voting device hang at 90% fatal, termination in 10110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:38.345 [cssd(7234)]CRS-1613:voting device hang at 90% fatal, termination in 9110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:39.346 [cssd(7234)]CRS-1613:voting device hang at 90% fatal, termination in 8110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:40.346 [cssd(7234)]CRS-1613:voting device hang at 90% fatal, termination in 7110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:41.346 [cssd(7234)]CRS-1613:voting device hang at 90% fatal, termination in 6110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:42.346 [cssd(7234)]CRS-1613:voting device hang at 90% fatal, termination in 5110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:43.346 [cssd(7234)]CRS-1613:voting device hang at 90% fatal, termination in 4110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:44.346 [cssd(7234)]CRS-1613:voting device hang at 90% fatal, termination in 3110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:45.346 [cssd(7234)]CRS-1613:voting device hang at 90% fatal, termination in 2110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:46.347 [cssd(7234)]CRS-1613:voting device hang at 90% fatal, termination in 1110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 11:50:47.347 [cssd(7234)]CRS-1613:voting device hang at 90% fatal, termination in 110 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:14:48.864 [cssd(8418)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. 2013-01-16 12:14:58.084 [cssd(8418)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2013-01-16 12:14:58.696 [crsd(7526)]CRS-1012:The OCR service started on node prod-rac02. 2013-01-16 12:14:59.484 [evmd(7525)]CRS-1401:EVMD started on node prod-rac02. 2013-01-16 12:15:01.291 [crsd(7526)]CRS-1201:CRSD started on node prod-rac02. 2013-01-16 12:20:36.553 [crsd(7526)]CRS-1205:Auto-start failed for the CRS resource ora.HWPROD.HWPROD1.inst. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/crsd/crsd.log. 2013-01-16 12:20:36.587 [crsd(7526)]CRS-1205:Auto-start failed for the CRS resource ora.HWPROD.HWPROD2.inst. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/crsd/crsd.log. 2013-01-16 12:20:45.529 [crsd(7526)]CRS-1205:Auto-start failed for the CRS resource ora.HWPROD.db. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/crsd/crsd.log. 2013-01-16 12:33:50.244 [cssd(8418)]CRS-1615:voting device hang at 50% fatal, termination in 99620 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:34:40.248 [cssd(8418)]CRS-1614:voting device hang at 75% fatal, termination in 49620 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:10.272 [cssd(8418)]CRS-1613:voting device hang at 90% fatal, termination in 19610 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:11.272 [cssd(8418)]CRS-1613:voting device hang at 90% fatal, termination in 18610 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:12.272 [cssd(8418)]CRS-1613:voting device hang at 90% fatal, termination in 17610 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:13.272 [cssd(8418)]CRS-1613:voting device hang at 90% fatal, termination in 16610 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:14.272 [cssd(8418)]CRS-1613:voting device hang at 90% fatal, termination in 15610 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:15.272 [cssd(8418)]CRS-1613:voting device hang at 90% fatal, termination in 14600 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:16.272 [cssd(8418)]CRS-1613:voting device hang at 90% fatal, termination in 13600 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:17.282 [cssd(8418)]CRS-1613:voting device hang at 90% fatal, termination in 12600 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:18.282 [cssd(8418)]CRS-1613:voting device hang at 90% fatal, termination in 11600 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:19.283 [cssd(8418)]CRS-1613:voting device hang at 90% fatal, termination in 10600 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:20.283 [cssd(8418)]CRS-1613:voting device hang at 90% fatal, termination in 9600 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:21.283 [cssd(8418)]CRS-1613:voting device hang at 90% fatal, termination in 8600 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:22.283 [cssd(8418)]CRS-1613:voting device hang at 90% fatal, termination in 7600 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:23.283 [cssd(8418)]CRS-1613:voting device hang at 90% fatal, termination in 6600 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:24.283 [cssd(8418)]CRS-1613:voting device hang at 90% fatal, termination in 5600 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:25.283 [cssd(8418)]CRS-1613:voting device hang at 90% fatal, termination in 4600 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:26.283 [cssd(8418)]CRS-1613:voting device hang at 90% fatal, termination in 3600 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:27.283 [cssd(8418)]CRS-1613:voting device hang at 90% fatal, termination in 2600 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:28.283 [cssd(8418)]CRS-1613:voting device hang at 90% fatal, termination in 1600 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:35:29.283 [cssd(8418)]CRS-1613:voting device hang at 90% fatal, termination in 600 ms, disk (0//dev/mpath/mpath0p2) 2013-01-16 12:40:30.615 [cssd(6723)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. 2013-01-16 12:40:39.784 [cssd(6723)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2013-01-16 12:40:40.285 [crsd(5909)]CRS-1012:The OCR service started on node prod-rac02. 2013-01-16 12:40:41.270 [evmd(5893)]CRS-1401:EVMD started on node prod-rac02. 2013-01-16 12:40:42.601 [crsd(5909)]CRS-1201:CRSD started on node prod-rac02. 2013-01-16 12:43:36.219 [crsd(5909)]CRS-1205:Auto-start failed for the CRS resource ora.HWPROD.HWPROD2.inst. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/crsd/crsd.log. 2013-01-16 12:44:27.166 [crsd(5909)]CRS-1205:Auto-start failed for the CRS resource ora.HWPROD.db. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/crsd/crsd.log. 2013-03-04 19:27:20.023 [cssd(6723)]CRS-1615:voting device hang at 50% fatal, termination in 99490 ms, disk (0//dev/mpath/mpath0p2) 2013-04-07 10:15:35.997 [cssd(6723)]CRS-1615:voting device hang at 50% fatal, termination in 99320 ms, disk (0//dev/mpath/mpath0p2) 2013-05-05 02:14:42.773 [crsd(5909)]CRS-1005:The OCR upgrade was completed. Version has changed from 185599744 to 185599744. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/crsd/crsd.log. 2013-05-05 02:14:43.351 [cssd(6723)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac02 . 2013-05-05 02:14:46.951 [crsd(5909)]CRS-1204:Recovering CRS resources for node prod-rac01. 2013-05-05 02:20:22.425 [cssd(6723)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2013-05-05 02:27:09.106 [cssd(6723)]CRS-1603:CSSD on node prod-rac02 shutdown by user. 2013-05-05 02:32:37.569 [cssd(8260)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. 2013-05-05 02:32:38.657 [cssd(8260)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2013-05-05 02:32:39.090 [crsd(7255)]CRS-1012:The OCR service started on node prod-rac02. 2013-05-05 02:32:39.254 [evmd(7213)]CRS-1401:EVMD started on node prod-rac02. 2013-05-05 02:32:41.210 [crsd(7255)]CRS-1201:CRSD started on node prod-rac02. 2013-05-05 04:11:21.394 [cssd(8260)]CRS-1615:voting device hang at 50% fatal, termination in 99760 ms, disk (0//dev/mpath/mpath0p2) 2013-05-23 11:05:16.564 [cssd(8260)]CRS-1603:CSSD on node prod-rac02 shutdown by user. 2013-05-23 11:10:38.274 [cssd(8530)]CRS-1605:CSSD voting file is online: /dev/mpath/mpath0p2. Details in /u01/app/oracle/product/11.1.0/crs/log/prod-rac02/cssd/ocssd.log. 2013-05-23 11:10:39.307 [cssd(8530)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod-rac01 prod-rac02 . 2013-05-23 11:10:39.747 [crsd(7536)]CRS-1012:The OCR service started on node prod-rac02. 2013-05-23 11:10:40.631 [evmd(7566)]CRS-1401:EVMD started on node prod-rac02. 2013-05-23 11:10:41.736 [crsd(7536)]CRS-1201:CRSD started on node prod-rac02.