Task 13589540

Name	hadcm3n_yaax_1900_40_007520917_4
Workunit	7718392
Created	3 Nov 2011, 8:38:30 UTC
Sent	3 Nov 2011, 8:44:30 UTC
Report deadline	2 Feb 2012, 16:11:41 UTC
Received	27 Nov 2011, 3:24:38 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1161345
Run time	4 days 13 hours 24 min 50 sec
CPU time	4 days 9 hours 9 min 55 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.65 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3400, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5076, iMonCtr=1 Model crash detected, will try to restart... 17:57:35 (4196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:59:34 (4664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1220, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Cforrtl: Access is denied. CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3960, selfPID=3960, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CNo Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4980, selfPID=4980, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CNo Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2472, selfPID=2472, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Cforrtl: Access is denied. CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1456, selfPID=1456, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:02:13 (3996): No heartbeat from core client for 30 sec - exiting 18:02:15 (3996): No heartbeat from core client for 30 sec - exiting 18:02:16 (3996): No heartbeat from core client for 30 sec - exiting 18:02:17 (3996): No heartbeat from core client for 30 sec - exiting 18:02:18 (3996): No heartbeat from core client for 30 sec - exiting 18:02:19 (3996): No heartbeat from core client for 30 sec - exiting 18:02:20 (3996): No heartbeat from core client for 30 sec - exiting 18:02:21 (3996): No heartbeat from core client for 30 sec - exiting 18:02:22 (3996): No heartbeat from core client for 30 sec - exiting 18:02:23 (3996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CWorker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4256, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4256, iMonCtr=1 Model crash detected, will try to restart... 12:19:09 (4024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CNo Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6032, selfPID=6032, iMonCtr=1 16:24:48 (2748): No heartbeat from core client for 30 sec - exiting 16:24:49 (2748): No heartbeat from core client for 30 sec - exiting 16:24:50 (2748): No heartbeat from core client for 30 sec - exiting 16:24:51 (2748): No heartbeat from core client for 30 sec - exiting 16:24:53 (2748): No heartbeat from core client for 30 sec - exiting 16:24:54 (2748): No heartbeat from core client for 30 sec - exiting 16:24:55 (2748): No heartbeat from core client for 30 sec - exiting 16:24:56 (2748): No heartbeat from core client for 30 sec - exiting 16:24:57 (2748): No heartbeat from core client for 30 sec - exiting 16:24:58 (2748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:23:48 (4356): No heartbeat from core client for 30 sec - exiting 19:23:49 (4356): No heartbeat from core client for 30 sec - exiting 19:23:50 (4356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:23:51 (4356): No heartbeat from core client for 30 sec - exiting 17:59:25 (3956): No heartbeat from core client for 30 sec - exiting 17:59:27 (3956): No heartbeat from core client for 30 sec - exiting 17:59:28 (3956): No heartbeat from core client for 30 sec - exiting 17:59:29 (3956): No heartbeat from core client for 30 sec - exiting 17:59:30 (3956): No heartbeat from core client for 30 sec - exiting 17:59:31 (3956): No heartbeat from core client for 30 sec - exiting 17:59:32 (3956): No heartbeat from core client for 30 sec - exiting 17:59:33 (3956): No heartbeat from core client for 30 sec - exiting 17:59:35 (3956): No heartbeat from core client for 30 sec - exiting 17:59:36 (3956): No heartbeat from core client for 30 sec - exiting 17:59:37 (3956): No heartbeat from core client for 30 sec - exiting 17:59:38 (3956): No heartbeat from core client for 30 sec - exiting 17:59:39 (3956): No heartbeat from core client for 30 sec - exiting 17:59:40 (3956): No heartbeat from core client for 30 sec - exiting 17:59:41 (3956): No heartbeat from core client for 30 sec - exiting 17:59:42 (3956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:56:10 (4364): No heartbeat from core client for 30 sec - exiting 17:56:12 (4364): No heartbeat from core client for 30 sec - exiting 17:56:13 (4364): No heartbeat from core client for 30 sec - exiting 17:56:14 (4364): No heartbeat from core client for 30 sec - exiting 17:56:15 (4364): No heartbeat from core client for 30 sec - exiting 17:56:16 (4364): No heartbeat from core client for 30 sec - exiting 17:56:17 (4364): No heartbeat from core client for 30 sec - exiting 17:56:18 (4364): No heartbeat from core client for 30 sec - exiting 17:56:19 (4364): No heartbeat from core client for 30 sec - exiting 17:56:20 (4364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3768, iMonCtr=1 Model crash detected, will try to restart... 17:33:38 (2964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:08:42 (4488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Nov 2011 05:53:55	1161345	13589540	hadcm3n_yaax_1900_40_007520917_4	259,200	378,588	1.4606
25 Nov 2011 10:21:21	1161345	13589540	hadcm3n_yaax_1900_40_007520917_4	233,280	340,568	1.4599
24 Nov 2011 06:51:03	1161345	13589540	hadcm3n_yaax_1900_40_007520917_4	207,360	302,396	1.4583
22 Nov 2011 11:09:21	1161345	13589540	hadcm3n_yaax_1900_40_007520917_4	181,440	264,265	1.4565
21 Nov 2011 07:48:12	1161345	13589540	hadcm3n_yaax_1900_40_007520917_4	155,520	225,370	1.4491
19 Nov 2011 08:33:35	1161345	13589540	hadcm3n_yaax_1900_40_007520917_4	129,600	187,495	1.4467
17 Nov 2011 12:54:27	1161345	13589540	hadcm3n_yaax_1900_40_007520917_4	103,680	150,000	1.4468
16 Nov 2011 07:59:35	1161345	13589540	hadcm3n_yaax_1900_40_007520917_4	77,760	112,305	1.4443
16 Nov 2011 05:58:29	1161345	13589540	hadcm3n_yaax_1900_40_007520917_4	51,840	74,873	1.4443
10 Nov 2011 06:56:10	1161345	13589540	hadcm3n_yaax_1900_40_007520917_4	25,920	37,172	1.4341