Task 15444455

Name	hadcm3n_ze0q_1880_40_008247345_0
Workunit	8402469
Created	21 Nov 2012, 7:03:28 UTC
Sent	21 Nov 2012, 7:03:49 UTC
Report deadline	20 Feb 2013, 14:31:00 UTC
Received	26 Nov 2012, 5:29:23 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1244719
Run time	4 days 17 hours 54 min 2 sec
CPU time	4 days 9 hours 9 min 3 sec
Validate state	Invalid
Credit	3,421.44
Device peak FLOPS	3.18 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... 18:30:32 (6140): No heartbeat from core client for 30 sec - exiting 18:30:33 (6140): No heartbeat from core client for 30 sec - exiting 18:30:34 (6140): No heartbeat from core client for 30 sec - exiting 18:30:35 (6140): No heartbeat from core client for 30 sec - exiting 18:30:36 (6140): No heartbeat from core client for 30 sec - exiting 18:30:37 (6140): No heartbeat from core client for 30 sec - exiting 18:30:38 (6140): No heartbeat from core client for 30 sec - exiting 18:30:39 (6140): No heartbeat from core client for 30 sec - exiting 18:30:40 (6140): No heartbeat from core client for 30 sec - exiting 18:30:41 (6140): No heartbeat from core client for 30 sec - exiting 18:30:43 (6140): No heartbeat from core client for 30 sec - exiting 18:30:44 (6140): No heartbeat from core client for 30 sec - exiting 18:30:45 (6140): No heartbeat from core client for 30 sec - exiting 18:30:46 (6140): No heartbeat from core client for 30 sec - exiting 18:30:47 (6140): No heartbeat from core client for 30 sec - exiting 18:30:48 (6140): No heartbeat from core client for 30 sec - exiting 18:30:49 (6140): No heartbeat from core client for 30 sec - exiting 18:30:50 (6140): No heartbeat from core client for 30 sec - exiting 18:30:51 (6140): No heartbeat from core client for 30 sec - exiting 18:30:52 (6140): No heartbeat from core client for 30 sec - exiting 18:30:53 (6140): No heartbeat from core client for 30 sec - exiting 18:30:55 (6140): No heartbeat from core client for 30 sec - exiting 18:30:56 (6140): No heartbeat from core client for 30 sec - exiting 18:30:57 (6140): No heartbeat from core client for 30 sec - exiting 18:30:58 (6140): No heartbeat from core client for 30 sec - exiting 18:30:59 (6140): No heartbeat from core client for 30 sec - exiting 18:31:00 (6140): No heartbeat from core client for 30 sec - exiting 18:31:01 (6140): No heartbeat from core client for 30 sec - exiting 18:31:02 (6140): No heartbeat from core client for 30 sec - exiting 18:31:03 (6140): No heartbeat from core client for 30 sec - exiting 18:31:04 (6140): No heartbeat from core client for 30 sec - exiting 18:31:05 (6140): No heartbeat from core client for 30 sec - exiting 18:31:06 (6140): No heartbeat from core client for 30 sec - exiting 18:31:07 (6140): No heartbeat from core client for 30 sec - exiting 18:31:08 (6140): No heartbeat from core client for 30 sec - exiting 18:31:09 (6140): No heartbeat from core client for 30 sec - exiting 18:31:10 (6140): No heartbeat from core client for 30 sec - exiting 18:31:11 (6140): No heartbeat from core client for 30 sec - exiting 18:31:12 (6140): No heartbeat from core client for 30 sec - exiting 18:31:13 (6140): No heartbeat from core client for 30 sec - exiting 18:31:14 (6140): No heartbeat from core client for 30 sec - exiting 18:31:15 (6140): No heartbeat from core client for 30 sec - exiting 18:31:16 (6140): No heartbeat from core client for 30 sec - exiting 18:31:17 (6140): No heartbeat from core client for 30 sec - exiting 18:31:18 (6140): No heartbeat from core client for 30 sec - exiting 18:31:19 (6140): No heartbeat from core client for 30 sec - exiting 18:31:21 (6140): No heartbeat from core client for 30 sec - exiting 18:31:22 (6140): No heartbeat from core client for 30 sec - exiting 18:31:23 (6140): No heartbeat from core client for 30 sec - exiting 18:31:24 (6140): No heartbeat from core client for 30 sec - exiting 18:31:25 (6140): No heartbeat from core client for 30 sec - exiting 18:31:26 (6140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:18:45 (3156): No heartbeat from core client for 30 sec - exiting 19:18:46 (3156): No heartbeat from core client for 30 sec - exiting 19:18:47 (3156): No heartbeat from core client for 30 sec - exiting 19:18:48 (3156): No heartbeat from core client for 30 sec - exiting 19:18:49 (3156): No heartbeat from core client for 30 sec - exiting 19:18:50 (3156): No heartbeat from core client for 30 sec - exiting 19:18:52 (3156): No heartbeat from core client for 30 sec - exiting 19:18:53 (3156): No heartbeat from core client for 30 sec - exiting 19:18:54 (3156): No heartbeat from core client for 30 sec - exiting 19:18:55 (3156): No heartbeat from core client for 30 sec - exiting 19:18:56 (3156): No heartbeat from core client for 30 sec - exiting 19:18:57 (3156): No heartbeat from core client for 30 sec - exiting 19:18:58 (3156): No heartbeat from core client for 30 sec - exiting 19:18:59 (3156): No heartbeat from core client for 30 sec - exiting 19:19:00 (3156): No heartbeat from core client for 30 sec - exiting 19:19:01 (3156): No heartbeat from core client for 30 sec - exiting 19:19:02 (3156): No heartbeat from core client for 30 sec - exiting 19:19:04 (3156): No heartbeat from core client for 30 sec - exiting 19:19:05 (3156): No heartbeat from core client for 30 sec - exiting 19:19:06 (3156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4504, iMonCtr=1 Model crash detected, will try to restart... 17:09:52 (4812): No heartbeat from core client for 30 sec - exiting 17:09:54 (4812): No heartbeat from core client for 30 sec - exiting 17:09:55 (4812): No heartbeat from core client for 30 sec - exiting 17:09:56 (4812): No heartbeat from core client for 30 sec - exiting 17:09:57 (4812): No heartbeat from core client for 30 sec - exiting 17:09:58 (4812): No heartbeat from core client for 30 sec - exiting 17:09:59 (4812): No heartbeat from core client for 30 sec - exiting 17:10:00 (4812): No heartbeat from core client for 30 sec - exiting 17:10:01 (4812): No heartbeat from core client for 30 sec - exiting 17:10:02 (4812): No heartbeat from core client for 30 sec - exiting 17:10:03 (4812): No heartbeat from core client for 30 sec - exiting 17:10:05 (4812): No heartbeat from core client for 30 sec - exiting 17:10:06 (4812): No heartbeat from core client for 30 sec - exiting 17:10:07 (4812): No heartbeat from core client for 30 sec - exiting 17:10:08 (4812): No heartbeat from core client for 30 sec - exiting 17:10:09 (4812): No heartbeat from core client for 30 sec - exiting 17:10:10 (4812): No heartbeat from core client for 30 sec - exiting 17:10:11 (4812): No heartbeat from core client for 30 sec - exiting 17:10:12 (4812): No heartbeat from core client for 30 sec - exiting 17:10:13 (4812): No heartbeat from core client for 30 sec - exiting 17:10:15 (4812): No heartbeat from core client for 30 sec - exiting 17:10:16 (4812): No heartbeat from core client for 30 sec - exiting 17:10:17 (4812): No heartbeat from core client for 30 sec - exiting 17:10:18 (4812): No heartbeat from core client for 30 sec - exiting 17:10:19 (4812): No heartbeat from core client for 30 sec - exiting 17:10:20 (4812): No heartbeat from core client for 30 sec - exiting 17:10:21 (4812): No heartbeat from core client for 30 sec - exiting 17:10:22 (4812): No heartbeat from core client for 30 sec - exiting 17:10:23 (4812): No heartbeat from core client for 30 sec - exiting 17:10:24 (4812): No heartbeat from core client for 30 sec - exiting 17:10:25 (4812): No heartbeat from core client for 30 sec - exiting 17:10:27 (4812): No heartbeat from core client for 30 sec - exiting 17:10:28 (4812): No heartbeat from core client for 30 sec - exiting 17:10:29 (4812): No heartbeat from core client for 30 sec - exiting 17:10:30 (4812): No heartbeat from core client for 30 sec - exiting 17:10:31 (4812): No heartbeat from core client for 30 sec - exiting 17:10:32 (4812): No heartbeat from core client for 30 sec - exiting 17:10:33 (4812): No heartbeat from core client for 30 sec - exiting 17:10:34 (4812): No heartbeat from core client for 30 sec - exiting 17:10:35 (4812): No heartbeat from core client for 30 sec - exiting 17:10:36 (4812): No heartbeat from core client for 30 sec - exiting 17:10:37 (4812): No heartbeat from core client for 30 sec - exiting 17:10:39 (4812): No heartbeat from core client for 30 sec - exiting 17:10:40 (4812): No heartbeat from core client for 30 sec - exiting 17:10:41 (4812): No heartbeat from core client for 30 sec - exiting 17:10:42 (4812): No heartbeat from core client for 30 sec - exiting 17:10:43 (4812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6108, iMonCtr=1 Model crash detected, will try to restart... 02:03:19 (3604): No heartbeat from core client for 30 sec - exiting 02:03:20 (3604): No heartbeat from core client for 30 sec - exiting 02:03:21 (3604): No heartbeat from core client for 30 sec - exiting 02:03:22 (3604): No heartbeat from core client for 30 sec - exiting 02:03:23 (3604): No heartbeat from core client for 30 sec - exiting 02:03:25 (3604): No heartbeat from core client for 30 sec - exiting 02:03:26 (3604): No heartbeat from core client for 30 sec - exiting 02:03:27 (3604): No heartbeat from core client for 30 sec - exiting 02:03:28 (3604): No heartbeat from core client for 30 sec - exiting 02:03:29 (3604): No heartbeat from core client for 30 sec - exiting 02:03:30 (3604): No heartbeat from core client for 30 sec - exiting 02:03:31 (3604): No heartbeat from core client for 30 sec - exiting 02:03:32 (3604): No heartbeat from core client for 30 sec - exiting 02:03:33 (3604): No heartbeat from core client for 30 sec - exiting 02:03:34 (3604): No heartbeat from core client for 30 sec - exiting 02:03:35 (3604): No heartbeat from core client for 30 sec - exiting 02:03:37 (3604): No heartbeat from core client for 30 sec - exiting 02:03:38 (3604): No heartbeat from core client for 30 sec - exiting 02:03:39 (3604): No heartbeat from core client for 30 sec - exiting 02:03:40 (3604): No heartbeat from core client for 30 sec - exiting 02:03:41 (3604): No heartbeat from core client for 30 sec - exiting 02:03:42 (3604): No heartbeat from core client for 30 sec - exiting 02:03:43 (3604): No heartbeat from core client for 30 sec - exiting 02:03:44 (3604): No heartbeat from core client for 30 sec - exiting 02:03:45 (3604): No heartbeat from core client for 30 sec - exiting 02:03:46 (3604): No heartbeat from core client for 30 sec - exiting 02:03:48 (3604): No heartbeat from core client for 30 sec - exiting 02:03:49 (3604): No heartbeat from core client for 30 sec - exiting 02:03:50 (3604): No heartbeat from core client for 30 sec - exiting 02:03:51 (3604): No heartbeat from core client for 30 sec - exiting 02:03:52 (3604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:03:53 (3604): No heartbeat from core client for 30 sec - exiting 02:03:54 (3604): No heartbeat from core client for 30 sec - exiting 02:03:55 (3604): No heartbeat from core client for 30 sec - exiting 02:03:56 (3604): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 18:13:01 (4416): No heartbeat from core client for 30 sec - exiting 18:13:02 (4416): No heartbeat from core client for 30 sec - exiting 18:13:03 (4416): No heartbeat from core client for 30 sec - exiting 18:13:04 (4416): No heartbeat from core client for 30 sec - exiting 18:13:05 (4416): No heartbeat from core client for 30 sec - exiting 18:13:06 (4416): No heartbeat from core client for 30 sec - exiting 18:13:07 (4416): No heartbeat from core client for 30 sec - exiting 18:13:08 (4416): No heartbeat from core client for 30 sec - exiting 18:13:09 (4416): No heartbeat from core client for 30 sec - exiting 18:13:11 (4416): No heartbeat from core client for 30 sec - exiting 18:13:12 (4416): No heartbeat from core client for 30 sec - exiting 18:13:13 (4416): No heartbeat from core client for 30 sec - exiting 18:13:14 (4416): No heartbeat from core client for 30 sec - exiting 18:13:15 (4416): No heartbeat from core client for 30 sec - exiting 18:13:16 (4416): No heartbeat from core client for 30 sec - exiting 18:13:17 (4416): No heartbeat from core client for 30 sec - exiting 18:13:18 (4416): No heartbeat from core client for 30 sec - exiting 18:13:19 (4416): No heartbeat from core client for 30 sec - exiting 18:13:20 (4416): No heartbeat from core client for 30 sec - exiting 18:13:21 (4416): No heartbeat from core client for 30 sec - exiting 18:13:23 (4416): No heartbeat from core client for 30 sec - exiting 18:13:24 (4416): No heartbeat from core client for 30 sec - exiting 18:13:25 (4416): No heartbeat from core client for 30 sec - exiting 18:13:26 (4416): No heartbeat from core client for 30 sec - exiting 18:13:27 (4416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Model crashed: SETPOS: Unit 63 to Word Address -198 Failed with Error Code -1 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Nov 2012 21:42:26	1244719	15444455	hadcm3n_ze0q_1880_40_008247345_0	285,120	363,521	1.2750
25 Nov 2012 11:15:52	1244719	15444455	hadcm3n_ze0q_1880_40_008247345_0	259,200	330,606	1.2755
25 Nov 2012 00:28:31	1244719	15444455	hadcm3n_ze0q_1880_40_008247345_0	233,280	297,580	1.2756
24 Nov 2012 15:04:44	1244719	15444455	hadcm3n_ze0q_1880_40_008247345_0	207,360	264,940	1.2777
24 Nov 2012 05:57:48	1244719	15444455	hadcm3n_ze0q_1880_40_008247345_0	181,440	232,144	1.2795
23 Nov 2012 20:08:15	1244719	15444455	hadcm3n_ze0q_1880_40_008247345_0	155,520	199,274	1.2813
23 Nov 2012 10:36:12	1244719	15444455	hadcm3n_ze0q_1880_40_008247345_0	129,600	166,573	1.2853
23 Nov 2012 00:48:47	1244719	15444455	hadcm3n_ze0q_1880_40_008247345_0	103,680	133,455	1.2872
22 Nov 2012 13:53:32	1244719	15444455	hadcm3n_ze0q_1880_40_008247345_0	77,760	100,194	1.2885
22 Nov 2012 04:21:46	1244719	15444455	hadcm3n_ze0q_1880_40_008247345_0	51,840	67,007	1.2926
21 Nov 2012 18:05:16	1244719	15444455	hadcm3n_ze0q_1880_40_008247345_0	25,920	33,673	1.2991