Task 16203860

Name	hadcm3n_39mn_1980_40_008400181_2
Workunit	8551037
Created	9 Jan 2014, 8:54:58 UTC
Sent	9 Jan 2014, 8:55:11 UTC
Report deadline	10 Apr 2014, 16:22:22 UTC
Received	25 Jan 2014, 9:59:46 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1090757
Run time	4 days 16 hours 6 min 32 sec
CPU time	2 days 6 hours 30 min 3 sec
Validate state	Invalid
Credit	1,244.16
Device peak FLOPS	2.53 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:08:03 (3844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:12:35 (2620): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 07:44:47 (4072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:27:59 (3824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:06:19 (5504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:07:34 (6016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:07:51 (6016): No heartbeat from core client for 30 sec - exiting 18:07:52 (6016): No heartbeat from core client for 30 sec - exiting 18:07:53 (6016): No heartbeat from core client for 30 sec - exiting 18:14:05 (3076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:14:07 (3076): No heartbeat from core client for 30 sec - exiting 18:14:08 (3076): No heartbeat from core client for 30 sec - exiting 18:14:09 (3076): No heartbeat from core client for 30 sec - exiting 18:15:28 (2164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:20:18 (5808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:22:08 (4960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:22:12 (4960): No heartbeat from core client for 30 sec - exiting 18:22:13 (4960): No heartbeat from core client for 30 sec - exiting 18:22:14 (4960): No heartbeat from core client for 30 sec - exiting 18:22:15 (4960): No heartbeat from core client for 30 sec - exiting 18:22:17 (4960): No heartbeat from core client for 30 sec - exiting 18:28:54 (5484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:28:57 (5484): No heartbeat from core client for 30 sec - exiting 18:28:58 (5484): No heartbeat from core client for 30 sec - exiting 18:28:59 (5484): No heartbeat from core client for 30 sec - exiting 18:30:00 (5108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:31:00 (5960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:33:43 (5728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 19:00:26 (4820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:29:31 (4384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:35:37 (4116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:04:47 (3684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:34:24 (3816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:34:23 (4748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:53:33 (3860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:54:38 (4596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:54:39 (4596): No heartbeat from core client for 30 sec - exiting 07:54:40 (4596): No heartbeat from core client for 30 sec - exiting 07:54:41 (4596): No heartbeat from core client for 30 sec - exiting 07:54:42 (4596): No heartbeat from core client for 30 sec - exiting 07:56:16 (3164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:57:13 (4688): No heartbeat from core client for 30 sec - exiting 07:57:18 (4688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:57:19 (4688): No heartbeat from core client for 30 sec - exiting 07:57:20 (4688): No heartbeat from core client for 30 sec - exiting 07:58:18 (4452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:59:49 (192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:01:41 (1124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:02:25 (4900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:04:07 (1260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:09:58 (936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:10:40 (2644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:11:24 (3892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:12:01 (2264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:12:59 (1280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:14:04 (4236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:14:48 (3252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:15:54 (3924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:17:26 (3676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:19:25 (4868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:20:36 (3080): No heartbeat from core client for 30 sec - exiting 08:20:37 (3080): No heartbeat from core client for 30 sec - exiting 08:20:38 (3080): No heartbeat from core client for 30 sec - exiting 08:20:39 (3080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:20:40 (3080): No heartbeat from core client for 30 sec - exiting 08:20:41 (3080): No heartbeat from core client for 30 sec - exiting 08:20:42 (3080): No heartbeat from core client for 30 sec - exiting 08:20:44 (3080): No heartbeat from core client for 30 sec - exiting 08:22:12 (4480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:23:04 (5112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:23:08 (5112): No heartbeat from core client for 30 sec - exiting 08:23:09 (5112): No heartbeat from core client for 30 sec - exiting 08:23:10 (5112): No heartbeat from core client for 30 sec - exiting 08:23:11 (5112): No heartbeat from core client for 30 sec - exiting 08:23:13 (5112): No heartbeat from core client for 30 sec - exiting 08:23:14 (5112): No heartbeat from core client for 30 sec - exiting 08:23:15 (5112): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:29:40 (3912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:28:55 (3696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:30:05 (4612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:32:28 (2144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:32:40 (2144): No heartbeat from core client for 30 sec - exiting 22:32:41 (2144): No heartbeat from core client for 30 sec - exiting 22:32:43 (2144): No heartbeat from core client for 30 sec - exiting 22:32:44 (2144): No heartbeat from core client for 30 sec - exiting 22:32:45 (2144): No heartbeat from core client for 30 sec - exiting 22:32:46 (2144): No heartbeat from core client for 30 sec - exiting 22:32:47 (2144): No heartbeat from core client for 30 sec - exiting 22:32:48 (2144): No heartbeat from core client for 30 sec - exiting 22:32:49 (2144): No heartbeat from core client for 30 sec - exiting 22:32:50 (2144): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 18:24:27 (3084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:25:12 (4060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:25:21 (4060): No heartbeat from core client for 30 sec - exiting 18:25:22 (4060): No heartbeat from core client for 30 sec - exiting 18:25:23 (4060): No heartbeat from core client for 30 sec - exiting 18:25:24 (4060): No heartbeat from core client for 30 sec - exiting 18:25:25 (4060): No heartbeat from core client for 30 sec - exiting 18:25:26 (4060): No heartbeat from core client for 30 sec - exiting 18:25:27 (4060): No heartbeat from core client for 30 sec - exiting 18:25:28 (4060): No heartbeat from core client for 30 sec - exiting 18:25:29 (4060): No heartbeat from core client for 30 sec - exiting 18:25:30 (4060): No heartbeat from core client for 30 sec - exiting 18:26:45 (6008): No heartbeat from core client for 30 sec - exiting 18:26:47 (6008): No heartbeat from core client for 30 sec - exiting 18:26:48 (6008): No heartbeat from core client for 30 sec - exiting 18:26:49 (6008): No heartbeat from core client for 30 sec - exiting 18:26:50 (6008): No heartbeat from core client for 30 sec - exiting 18:26:51 (6008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:27:57 (2664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5816, iMonCtr=1 Model crash detected, will try to restart... 18:29:03 (5816): No heartbeat from core client for 30 sec - exiting 18:29:04 (5816): No heartbeat from core client for 30 sec - exiting 18:29:05 (5816): No heartbeat from core client for 30 sec - exiting 18:29:06 (5816): No heartbeat from core client for 30 sec - exiting 18:29:07 (5816): No heartbeat from core client for 30 sec - exiting 18:29:08 (5816): No heartbeat from core client for 30 sec - exiting 18:29:09 (5816): No heartbeat from core client for 30 sec - exiting 18:29:10 (5816): No heartbeat from core client for 30 sec - exiting 18:29:11 (5816): No heartbeat from core client for 30 sec - exiting 18:29:12 (5816): No heartbeat from core client for 30 sec - exiting 18:29:13 (5816): No heartbeat from core client for 30 sec - exiting 18:29:14 (5816): No heartbeat from core client for 30 sec - exiting 18:29:15 (5816): No heartbeat from core client for 30 sec - exiting 18:29:16 (5816): No heartbeat from core client for 30 sec - exiting 18:29:17 (5816): No heartbeat from core client for 30 sec - exiting 18:29:18 (5816): No heartbeat from core client for 30 sec - exiting 18:29:19 (5816): No heartbeat from core client for 30 sec - exiting 18:29:20 (5816): No heartbeat from core client for 30 sec - exiting 18:29:21 (5816): No heartbeat from core client for 30 sec - exiting 18:29:22 (5816): No heartbeat from core client for 30 sec - exiting 18:29:23 (5816): No heartbeat from core client for 30 sec - exiting 18:29:24 (5816): No heartbeat from core client for 30 sec - exiting 18:29:25 (5816): No heartbeat from core client for 30 sec - exiting 18:29:26 (5816): No heartbeat from core client for 30 sec - exiting 18:29:27 (5816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:34:58 (5312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:35:50 (2956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5792, iMonCtr=1 Model crash detected, will try to restart... 18:51:45 (5792): No heartbeat from core client for 30 sec - exiting 18:51:46 (5792): No heartbeat from core client for 30 sec - exiting 18:51:47 (5792): No heartbeat from core client for 30 sec - exiting 18:51:48 (5792): No heartbeat from core client for 30 sec - exiting 18:51:49 (5792): No heartbeat from core client for 30 sec - exiting 18:51:50 (5792): No heartbeat from core client for 30 sec - exiting 18:51:51 (5792): No heartbeat from core client for 30 sec - exiting 18:51:52 (5792): No heartbeat from core client for 30 sec - exiting 18:51:53 (5792): No heartbeat from core client for 30 sec - exiting 18:51:54 (5792): No heartbeat from core client for 30 sec - exiting 18:51:58 (5792): No heartbeat from core client for 30 sec - exiting 18:52:01 (5792): No heartbeat from core client for 30 sec - exiting 18:52:02 (5792): No heartbeat from core client for 30 sec - exiting 18:52:03 (5792): No heartbeat from core client for 30 sec - exiting 18:52:04 (5792): No heartbeat from core client for 30 sec - exiting 18:52:06 (5792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:54:05 (4872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4484, iMonCtr=1 Model crash detected, will try to restart... 18:57:08 (4484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:57:51 (5992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4256, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4256, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4256, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Jan 2014 10:15:36	1090757	16203860	hadcm3n_39mn_1980_40_008400181_2	103,680	190,486	1.8372
19 Jan 2014 11:20:41	1090757	16203860	hadcm3n_39mn_1980_40_008400181_2	77,760	142,382	1.8310
14 Jan 2014 16:54:27	1090757	16203860	hadcm3n_39mn_1980_40_008400181_2	51,840	94,052	1.8143
12 Jan 2014 13:19:33	1090757	16203860	hadcm3n_39mn_1980_40_008400181_2	25,920	48,150	1.8576