Task 15581977

Name	hadcm3n_3g4z_1940_40_008263040_4
Workunit	8418164
Created	3 Feb 2013, 16:02:39 UTC
Sent	3 Feb 2013, 16:02:42 UTC
Report deadline	5 May 2013, 23:29:53 UTC
Received	2 Mar 2013, 17:39:47 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1077037
Run time	21 days 6 hours 1 min 43 sec
CPU time	16 days 3 hours 38 min 38 sec
Validate state	Invalid
Credit	5,287.68
Device peak FLOPS	2.26 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... 11:31:14 (5968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:31:16 (5968): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:43:39 (8268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:43:40 (8268): No heartbeat from core client for 30 sec - exiting 22:43:41 (8268): No heartbeat from core client for 30 sec - exiting 22:43:42 (8268): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 01:22:57 (10140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:25:48 (9192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:06:30 (6512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:12:44 (7564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:53:43 (5540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:53:44 (5540): No heartbeat from core client for 30 sec - exiting 09:59:29 (2208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:02:47 (5540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:22:12 (8148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:22:14 (8148): No heartbeat from core client for 30 sec - exiting 23:39:19 (5608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:41:58 (9576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:18:14 (6200): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 19:18:16 (6200): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 10:12:30 (4480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:12:31 (4480): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 09:45:21 (7428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:26:17 (2544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:14:30 (8296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=34740, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:41:00 (5708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:34:17 (8644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:36:23 (51632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:54:46 (20264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:58:18 (31080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=50392, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Mar 2013 15:55:59	1077037	15581977	hadcm3n_3g4z_1940_40_008263040_4	440,640	1,456,394	3.3052
28 Feb 2013 16:00:34	1077037	15581977	hadcm3n_3g4z_1940_40_008263040_4	414,720	1,360,165	3.2797
26 Feb 2013 21:18:13	1077037	15581977	hadcm3n_3g4z_1940_40_008263040_4	388,800	1,259,318	3.2390
25 Feb 2013 16:57:15	1077037	15581977	hadcm3n_3g4z_1940_40_008263040_4	362,880	1,161,665	3.2012
24 Feb 2013 00:32:01	1077037	15581977	hadcm3n_3g4z_1940_40_008263040_4	336,960	1,062,818	3.1541
22 Feb 2013 15:25:34	1077037	15581977	hadcm3n_3g4z_1940_40_008263040_4	311,040	979,467	3.1490
21 Feb 2013 08:54:41	1077037	15581977	hadcm3n_3g4z_1940_40_008263040_4	285,120	889,603	3.1201
19 Feb 2013 16:46:11	1077037	15581977	hadcm3n_3g4z_1940_40_008263040_4	259,200	800,706	3.0891
16 Feb 2013 02:47:42	1077037	15581977	hadcm3n_3g4z_1940_40_008263040_4	233,280	715,122	3.0655
13 Feb 2013 15:55:43	1077037	15581977	hadcm3n_3g4z_1940_40_008263040_4	207,360	625,518	3.0166
12 Feb 2013 13:45:38	1077037	15581977	hadcm3n_3g4z_1940_40_008263040_4	181,440	556,831	3.0690
11 Feb 2013 22:01:43	1077037	15581977	hadcm3n_3g4z_1940_40_008263040_4	155,520	502,101	3.2285
11 Feb 2013 12:55:10	1077037	15581977	hadcm3n_3g4z_1940_40_008263040_4	129,600	449,100	3.4653
09 Feb 2013 11:57:36	1077037	15581977	hadcm3n_3g4z_1940_40_008263040_4	103,680	358,720	3.4599
07 Feb 2013 17:37:37	1077037	15581977	hadcm3n_3g4z_1940_40_008263040_4	77,760	268,125	3.4481
05 Feb 2013 23:46:05	1077037	15581977	hadcm3n_3g4z_1940_40_008263040_4	51,840	176,841	3.4113
04 Feb 2013 17:41:34	1077037	15581977	hadcm3n_3g4z_1940_40_008263040_4	25,920	87,348	3.3699