Task 15810118

Name	hadcm3n_o18d_1940_40_008379222_0
Workunit	8530081
Created	31 May 2013, 17:21:44 UTC
Sent	18 Jun 2013, 18:50:23 UTC
Report deadline	18 Sep 2013, 2:17:34 UTC
Received	8 Jul 2013, 8:32:34 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1285155
Run time	7 days 1 hours 9 min 8 sec
CPU time	5 days 19 hours 55 min 7 sec
Validate state	Invalid
Credit	2,488.32
Device peak FLOPS	2.02 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.1.17</core_client_version> <![CDATA[ <message> Enheten känner inte igen kommandot. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:52:42 (3816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1060, iMonCtr=1 Model crash detected, will try to restart... 20:12:10 (2736): No heartbeat from core client for 30 sec - exiting 20:12:11 (2736): No heartbeat from core client for 30 sec - exiting 20:12:13 (2736): No heartbeat from core client for 30 sec - exiting 20:12:14 (2736): No heartbeat from core client for 30 sec - exiting 20:12:15 (2736): No heartbeat from core client for 30 sec - exiting 20:12:16 (2736): No heartbeat from core client for 30 sec - exiting 20:12:17 (2736): No heartbeat from core client for 30 sec - exiting 20:12:18 (2736): No heartbeat from core client for 30 sec - exiting 20:12:19 (2736): No heartbeat from core client for 30 sec - exiting 20:12:20 (2736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:14:33 (3384): No heartbeat from core client for 30 sec - exiting 11:14:34 (3384): No heartbeat from core client for 30 sec - exiting 11:14:35 (3384): No heartbeat from core client for 30 sec - exiting 11:14:36 (3384): No heartbeat from core client for 30 sec - exiting 11:14:37 (3384): No heartbeat from core client for 30 sec - exiting 11:14:38 (3384): No heartbeat from core client for 30 sec - exiting 11:14:39 (3384): No heartbeat from core client for 30 sec - exiting 11:14:40 (3384): No heartbeat from core client for 30 sec - exiting 11:14:41 (3384): No heartbeat from core client for 30 sec - exiting 11:14:43 (3384): No heartbeat from core client for 30 sec - exiting 11:14:44 (3384): No heartbeat from core client for 30 sec - exiting 11:14:45 (3384): No heartbeat from core client for 30 sec - exiting 11:14:46 (3384): No heartbeat from core client for 30 sec - exiting 11:14:47 (3384): No heartbeat from core client for 30 sec - exiting 11:14:48 (3384): No heartbeat from core client for 30 sec - exiting 11:14:49 (3384): No heartbeat from core client for 30 sec - exiting 11:14:50 (3384): No heartbeat from core client for 30 sec - exiting 11:14:51 (3384): No heartbeat from core client for 30 sec - exiting 11:14:52 (3384): No heartbeat from core client for 30 sec - exiting 11:14:53 (3384): No heartbeat from core client for 30 sec - exiting 11:14:54 (3384): No heartbeat from core client for 30 sec - exiting 11:14:55 (3384): No heartbeat from core client for 30 sec - exiting 11:14:56 (3384): No heartbeat from core client for 30 sec - exiting 11:14:57 (3384): No heartbeat from core client for 30 sec - exiting 11:14:58 (3384): No heartbeat from core client for 30 sec - exiting 11:14:59 (3384): No heartbeat from core client for 30 sec - exiting 11:15:00 (3384): No heartbeat from core client for 30 sec - exiting 11:15:01 (3384): No heartbeat from core client for 30 sec - exiting 11:15:02 (3384): No heartbeat from core client for 30 sec - exiting 11:15:03 (3384): No heartbeat from core client for 30 sec - exiting 11:15:04 (3384): No heartbeat from core client for 30 sec - exiting 11:15:05 (3384): No heartbeat from core client for 30 sec - exiting 11:15:06 (3384): No heartbeat from core client for 30 sec - exiting 11:15:07 (3384): No heartbeat from core client for 30 sec - exiting 11:15:08 (3384): No heartbeat from core client for 30 sec - exiting 11:15:09 (3384): No heartbeat from core client for 30 sec - exiting 11:15:10 (3384): No heartbeat from core client for 30 sec - exiting 11:15:11 (3384): No heartbeat from core client for 30 sec - exiting 11:15:12 (3384): No heartbeat from core client for 30 sec - exiting 11:15:13 (3384): No heartbeat from core client for 30 sec - exiting 11:15:14 (3384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:15:15 (3384): No heartbeat from core client for 30 sec - exiting 11:15:16 (3384): No heartbeat from core client for 30 sec - exiting 12:09:05 (3460): No heartbeat from core client for 30 sec - exiting 12:09:06 (3460): No heartbeat from core client for 30 sec - exiting 12:09:07 (3460): No heartbeat from core client for 30 sec - exiting 12:09:08 (3460): No heartbeat from core client for 30 sec - exiting 12:09:09 (3460): No heartbeat from core client for 30 sec - exiting 12:09:10 (3460): No heartbeat from core client for 30 sec - exiting 12:09:11 (3460): No heartbeat from core client for 30 sec - exiting 12:09:12 (3460): No heartbeat from core client for 30 sec - exiting 12:09:13 (3460): No heartbeat from core client for 30 sec - exiting 12:09:14 (3460): No heartbeat from core client for 30 sec - exiting 12:09:15 (3460): No heartbeat from core client for 30 sec - exiting 12:14:32 (3508): No heartbeat from core client for 30 sec - exiting 12:14:33 (3508): No heartbeat from core client for 30 sec - exiting 12:14:34 (3508): No heartbeat from core client for 30 sec - exiting 12:14:35 (3508): No heartbeat from core client for 30 sec - exiting 12:14:36 (3508): No heartbeat from core client for 30 sec - exiting 12:14:37 (3508): No heartbeat from core client for 30 sec - exiting 12:14:38 (3508): No heartbeat from core client for 30 sec - exiting 12:14:39 (3508): No heartbeat from core client for 30 sec - exiting 12:14:40 (3508): No heartbeat from core client for 30 sec - exiting 12:14:41 (3508): No heartbeat from core client for 30 sec - exiting 12:14:42 (3508): No heartbeat from core client for 30 sec - exiting 12:14:44 (3508): No heartbeat from core client for 30 sec - exiting 12:14:45 (3508): No heartbeat from core client for 30 sec - exiting 12:14:46 (3508): No heartbeat from core client for 30 sec - exiting 12:14:47 (3508): No heartbeat from core client for 30 sec - exiting 12:14:48 (3508): No heartbeat from core client for 30 sec - exiting 12:14:49 (3508): No heartbeat from core client for 30 sec - exiting 12:14:50 (3508): No heartbeat from core client for 30 sec - exiting 12:14:51 (3508): No heartbeat from core client for 30 sec - exiting 12:14:52 (3508): No heartbeat from core client for 30 sec - exiting 12:14:53 (3508): No heartbeat from core client for 30 sec - exiting 12:14:54 (3508): No heartbeat from core client for 30 sec - exiting 12:14:56 (3508): No heartbeat from core client for 30 sec - exiting 12:14:57 (3508): No heartbeat from core client for 30 sec - exiting 12:14:58 (3508): No heartbeat from core client for 30 sec - exiting 12:14:59 (3508): No heartbeat from core client for 30 sec - exiting 12:15:00 (3508): No heartbeat from core client for 30 sec - exiting 12:15:01 (3508): No heartbeat from core client for 30 sec - exiting 12:15:02 (3508): No heartbeat from core client for 30 sec - exiting 12:15:03 (3508): No heartbeat from core client for 30 sec - exiting 12:15:04 (3508): No heartbeat from core client for 30 sec - exiting 12:15:05 (3508): No heartbeat from core client for 30 sec - exiting 12:15:06 (3508): No heartbeat from core client for 30 sec - exiting 12:15:08 (3508): No heartbeat from core client for 30 sec - exiting 12:15:09 (3508): No heartbeat from core client for 30 sec - exiting 12:15:10 (3508): No heartbeat from core client for 30 sec - exiting 12:15:11 (3508): No heartbeat from core client for 30 sec - exiting 12:15:12 (3508): No heartbeat from core client for 30 sec - exiting 12:15:13 (3508): No heartbeat from core client for 30 sec - exiting 12:15:14 (3508): No heartbeat from core client for 30 sec - exiting 12:15:15 (3508): No heartbeat from core client for 30 sec - exiting 12:15:16 (3508): No heartbeat from core client for 30 sec - exiting 12:15:17 (3508): No heartbeat from core client for 30 sec - exiting 12:15:18 (3508): No heartbeat from core client for 30 sec - exiting 12:15:20 (3508): No heartbeat from core client for 30 sec - exiting 12:15:21 (3508): No heartbeat from core client for 30 sec - exiting 12:15:22 (3508): No heartbeat from core client for 30 sec - exiting 12:15:23 (3508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4532, iMonCtr=1 Model crash detected, will try to restart... 18:08:02 (3368): No heartbeat from core client for 30 sec - exiting 18:08:04 (3368): No heartbeat from core client for 30 sec - exiting 18:08:05 (3368): No heartbeat from core client for 30 sec - exiting 18:08:06 (3368): No heartbeat from core client for 30 sec - exiting 18:08:07 (3368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3528, iMonCtr=1 Model crash detected, will try to restart... 19:40:38 (3352): No heartbeat from core client for 30 sec - exiting 19:40:40 (3352): No heartbeat from core client for 30 sec - exiting 19:40:41 (3352): No heartbeat from core client for 30 sec - exiting 19:40:42 (3352): No heartbeat from core client for 30 sec - exiting 19:40:43 (3352): No heartbeat from core client for 30 sec - exiting 19:40:44 (3352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:40:45 (3352): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=428, iMonCtr=1 Model crash detected, will try to restart... 20:01:59 (3304): No heartbeat from core client for 30 sec - exiting 20:02:01 (3304): No heartbeat from core client for 30 sec - exiting 20:02:02 (3304): No heartbeat from core client for 30 sec - exiting 20:02:03 (3304): No heartbeat from core client for 30 sec - exiting 20:02:04 (3304): No heartbeat from core client for 30 sec - exiting 20:02:05 (3304): No heartbeat from core client for 30 sec - exiting 20:02:06 (3304): No heartbeat from core client for 30 sec - exiting 20:02:07 (3304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:14:37 (2844): No heartbeat from core client for 30 sec - exiting 20:14:38 (2844): No heartbeat from core client for 30 sec - exiting 20:14:39 (2844): No heartbeat from core client for 30 sec - exiting 20:14:40 (2844): No heartbeat from core client for 30 sec - exiting 20:14:41 (2844): No heartbeat from core client for 30 sec - exiting 20:14:42 (2844): No heartbeat from core client for 30 sec - exiting 20:14:44 (2844): No heartbeat from core client for 30 sec - exiting 20:14:45 (2844): No heartbeat from core client for 30 sec - exiting 20:14:46 (2844): No heartbeat from core client for 30 sec - exiting 20:14:47 (2844): No heartbeat from core client for 30 sec - exiting 20:14:48 (2844): No heartbeat from core client for 30 sec - exiting 20:14:49 (2844): No heartbeat from core client for 30 sec - exiting 20:14:50 (2844): No heartbeat from core client for 30 sec - exiting 20:14:51 (2844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:22:40 (3000): No heartbeat from core client for 30 sec - exiting 21:22:42 (3000): No heartbeat from core client for 30 sec - exiting 21:22:43 (3000): No heartbeat from core client for 30 sec - exiting 21:22:44 (3000): No heartbeat from core client for 30 sec - exiting 21:22:45 (3000): No heartbeat from core client for 30 sec - exiting 21:22:46 (3000): No heartbeat from core client for 30 sec - exiting 21:22:47 (3000): No heartbeat from core client for 30 sec - exiting 21:22:48 (3000): No heartbeat from core client for 30 sec - exiting 21:22:49 (3000): No heartbeat from core client for 30 sec - exiting 21:22:50 (3000): No heartbeat from core client for 30 sec - exiting 21:22:51 (3000): No heartbeat from core client for 30 sec - exiting 21:22:53 (3000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:22:54 (3000): No heartbeat from core client for 30 sec - exiting 11:10:16 (2156): No heartbeat from core client for 30 sec - exiting 11:10:17 (2156): No heartbeat from core client for 30 sec - exiting 11:10:18 (2156): No heartbeat from core client for 30 sec - exiting 11:10:19 (2156): No heartbeat from core client for 30 sec - exiting 11:10:20 (2156): No heartbeat from core client for 30 sec - exiting 11:10:21 (2156): No heartbeat from core client for 30 sec - exiting 11:10:22 (2156): No heartbeat from core client for 30 sec - exiting 11:10:23 (2156): No heartbeat from core client for 30 sec - exiting 11:10:25 (2156): No heartbeat from core client for 30 sec - exiting 11:10:26 (2156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:34:05 (2892): No heartbeat from core client for 30 sec - exiting 16:34:06 (2892): No heartbeat from core client for 30 sec - exiting 16:34:07 (2892): No heartbeat from core client for 30 sec - exiting 16:34:08 (2892): No heartbeat from core client for 30 sec - exiting 16:34:09 (2892): No heartbeat from core client for 30 sec - exiting 16:34:11 (2892): No heartbeat from core client for 30 sec - exiting 16:34:12 (2892): No heartbeat from core client for 30 sec - exiting 16:34:13 (2892): No heartbeat from core client for 30 sec - exiting 16:34:14 (2892): No heartbeat from core client for 30 sec - exiting 16:34:15 (2892): No heartbeat from core client for 30 sec - exiting 16:34:16 (2892): No heartbeat from core client for 30 sec - exiting 16:34:17 (2892): No heartbeat from core client for 30 sec - exiting 16:34:18 (2892): No heartbeat from core client for 30 sec - exiting 16:34:19 (2892): No heartbeat from core client for 30 sec - exiting 16:34:20 (2892): No heartbeat from core client for 30 sec - exiting 16:34:21 (2892): No heartbeat from core client for 30 sec - exiting 16:34:23 (2892): No heartbeat from core client for 30 sec - exiting 16:34:24 (2892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3656, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3748, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4088, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
08 Jul 2013 07:32:04	1285155	15810118	hadcm3n_o18d_1940_40_008379222_0	207,360	508,637	2.4529
06 Jul 2013 22:25:44	1285155	15810118	hadcm3n_o18d_1940_40_008379222_0	181,440	445,475	2.4552
04 Jul 2013 14:24:36	1285155	15810118	hadcm3n_o18d_1940_40_008379222_0	155,520	382,333	2.4584
03 Jul 2013 01:03:36	1285155	15810118	hadcm3n_o18d_1940_40_008379222_0	129,600	319,178	2.4628
02 Jul 2013 09:56:13	1285155	15810118	hadcm3n_o18d_1940_40_008379222_0	103,680	255,649	2.4658
28 Jun 2013 04:01:23	1285155	15810118	hadcm3n_o18d_1940_40_008379222_0	77,760	191,415	2.4616
21 Jun 2013 16:40:12	1285155	15810118	hadcm3n_o18d_1940_40_008379222_0	51,840	127,605	2.4615
20 Jun 2013 14:00:13	1285155	15810118	hadcm3n_o18d_1940_40_008379222_0	25,920	63,634	2.4550