Task 13544433

Name	hadcm3n_ycgm_1900_40_007519375_1
Workunit	7716850
Created	28 Oct 2011, 13:02:37 UTC
Sent	19 Nov 2011, 10:32:23 UTC
Report deadline	18 Feb 2012, 17:59:34 UTC
Received	7 Dec 2011, 16:44:29 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1130254
Run time	13 days 20 hours 5 min 22 sec
CPU time	12 days 19 hours 55 min 11 sec
Validate state	Invalid
Credit	4,043.52
Device peak FLOPS	1.41 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.56</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC...Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1900, selfPID=1900, iMonCtr=1 CPDN Monitor - Quit request from BOINC... 14:31:26 (2860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:02:51 (3532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:19:54 (312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:44:50 (4064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:10:42 (1296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:54:37 (1352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:45:13 (2432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:02:25 (1136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:18:30 (136): No heartbeat from core client for 30 sec - exiting 23:18:31 (136): No heartbeat from core client for 30 sec - exiting 23:18:32 (136): No heartbeat from core client for 30 sec - exiting 23:18:34 (136): No heartbeat from core client for 30 sec - exiting 23:18:35 (136): No heartbeat from core client for 30 sec - exiting 23:18:36 (136): No heartbeat from core client for 30 sec - exiting 23:18:37 (136): No heartbeat from core client for 30 sec - exiting 23:18:38 (136): No heartbeat from core client for 30 sec - exiting 23:18:39 (136): No heartbeat from core client for 30 sec - exiting 23:18:41 (136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:07:10 (812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:53:43 (696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:18:30 (1352): No heartbeat from core client for 30 sec - exiting 23:18:31 (1352): No heartbeat from core client for 30 sec - exiting 23:18:32 (1352): No heartbeat from core client for 30 sec - exiting 23:18:33 (1352): No heartbeat from core client for 30 sec - exiting 23:18:35 (1352): No heartbeat from core client for 30 sec - exiting 23:18:36 (1352): No heartbeat from core client for 30 sec - exiting 23:18:37 (1352): No heartbeat from core client for 30 sec - exiting 23:18:38 (1352): No heartbeat from core client for 30 sec - exiting 23:18:39 (1352): No heartbeat from core client for 30 sec - exiting 23:18:40 (1352): No heartbeat from core client for 30 sec - exiting 23:18:42 (1352): No heartbeat from core client for 30 sec - exiting 23:18:43 (1352): No heartbeat from core client for 30 sec - exiting 23:18:44 (1352): No heartbeat from core client for 30 sec - exiting 23:18:45 (1352): No heartbeat from core client for 30 sec - exiting 23:18:46 (1352): No heartbeat from core client for 30 sec - exiting 23:18:47 (1352): No heartbeat from core client for 30 sec - exiting 23:18:49 (1352): No heartbeat from core client for 30 sec - exiting 23:18:50 (1352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:07:54 (256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:07:55 (256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:12:24 (2980): No heartbeat from core client for 30 sec - exiting 18:12:25 (2980): No heartbeat from core client for 30 sec - exiting 18:12:26 (2980): No heartbeat from core client for 30 sec - exiting 18:12:27 (2980): No heartbeat from core client for 30 sec - exiting 18:12:28 (2980): No heartbeat from core client for 30 sec - exiting 18:12:29 (2980): No heartbeat from core client for 30 sec - exiting 18:12:30 (2980): No heartbeat from core client for 30 sec - exiting 18:12:31 (2980): No heartbeat from core client for 30 sec - exiting 18:12:32 (2980): No heartbeat from core client for 30 sec - exiting 18:12:33 (2980): No heartbeat from core client for 30 sec - exiting 18:12:34 (2980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:12:35 (2980): No heartbeat from core client for 30 sec - exiting 18:12:36 (2980): No heartbeat from core client for 30 sec - exiting 18:12:37 (2980): No heartbeat from core client for 30 sec - exiting 18:12:38 (2980): No heartbeat from core client for 30 sec - exiting 18:12:39 (2980): No heartbeat from core client for 30 sec - exiting 18:12:40 (2980): No heartbeat from core client for 30 sec - exiting 18:12:41 (2980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3248, selfPID=3248, iMonCtr=1 22:12:58 (136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:44:14 (3312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Dec 2011 21:07:47	1130254	13544433	hadcm3n_ycgm_1900_40_007519375_1	336,960	1,055,627	3.1328
02 Dec 2011 16:34:03	1130254	13544433	hadcm3n_ycgm_1900_40_007519375_1	311,040	974,064	3.1316
01 Dec 2011 10:01:42	1130254	13544433	hadcm3n_ycgm_1900_40_007519375_1	285,120	891,994	3.1285
30 Nov 2011 05:20:28	1130254	13544433	hadcm3n_ycgm_1900_40_007519375_1	259,200	810,282	3.1261
28 Nov 2011 23:32:55	1130254	13544433	hadcm3n_ycgm_1900_40_007519375_1	233,280	727,810	3.1199
27 Nov 2011 22:37:17	1130254	13544433	hadcm3n_ycgm_1900_40_007519375_1	207,360	645,914	3.1149
26 Nov 2011 23:12:50	1130254	13544433	hadcm3n_ycgm_1900_40_007519375_1	181,440	565,276	3.1155
26 Nov 2011 00:22:28	1130254	13544433	hadcm3n_ycgm_1900_40_007519375_1	155,520	485,343	3.1208
25 Nov 2011 01:08:53	1130254	13544433	hadcm3n_ycgm_1900_40_007519375_1	129,600	404,918	3.1244
24 Nov 2011 01:54:33	1130254	13544433	hadcm3n_ycgm_1900_40_007519375_1	103,680	324,325	3.1281
23 Nov 2011 01:30:50	1130254	13544433	hadcm3n_ycgm_1900_40_007519375_1	77,760	244,180	3.1402
22 Nov 2011 00:21:35	1130254	13544433	hadcm3n_ycgm_1900_40_007519375_1	51,840	163,282	3.1497
20 Nov 2011 11:48:30	1130254	13544433	hadcm3n_ycgm_1900_40_007519375_1	25,920	81,774	3.1549