Task 13158908

Name	hadcm3n_p7b7_1900_40_007227059_2
Workunit	7425299
Created	25 Jul 2011, 16:33:13 UTC
Sent	25 Jul 2011, 16:33:17 UTC
Report deadline	25 Oct 2011, 0:00:28 UTC
Received	5 Aug 2011, 10:21:04 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1135626
Run time	8 days 13 hours 4 min 2 sec
CPU time	7 days 11 hours 36 min 19 sec
Validate state	Invalid
Credit	5,287.68
Device peak FLOPS	3.63 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 09:59:41 (3268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:34:13 (3468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:36:02 (3888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:15:33 (928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:17:40 (708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:18:04 (2784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:19:41 (920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:33:54 (3400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:35:53 (972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:41:18 (2840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:04:08 (3228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:15:14 (2392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:43:50 (3920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:45:28 (3892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:19:39 (2940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:03:33 (1228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:00:28 (1384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:05:09 (1532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:24:34 (2940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:51:30 (3132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:51:31 (3132): No heartbeat from core client for 30 sec - exiting 04:51:32 (3132): No heartbeat from core client for 30 sec - exiting 04:51:33 (3132): No heartbeat from core client for 30 sec - exiting 04:51:34 (3132): No heartbeat from core client for 30 sec - exiting 04:51:35 (3132): No heartbeat from core client for 30 sec - exiting 04:54:48 (324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:54:49 (324): No heartbeat from core client for 30 sec - exiting 04:54:50 (324): No heartbeat from core client for 30 sec - exiting 04:54:52 (324): No heartbeat from core client for 30 sec - exiting 04:57:16 (3112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:57:17 (3112): No heartbeat from core client for 30 sec - exiting 05:03:19 (2768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:05:50 (2676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:05:51 (2676): No heartbeat from core client for 30 sec - exiting 05:09:06 (3444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:14:56 (3376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:16:50 (2316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:16:51 (2316): No heartbeat from core client for 30 sec - exiting 05:16:52 (2316): No heartbeat from core client for 30 sec - exiting 05:16:53 (2316): No heartbeat from core client for 30 sec - exiting 05:16:54 (2316): No heartbeat from core client for 30 sec - exiting 05:19:53 (3660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:23:17 (3252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:23:18 (3252): No heartbeat from core client for 30 sec - exiting 05:23:19 (3252): No heartbeat from core client for 30 sec - exiting 05:25:49 (3720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:25:50 (3720): No heartbeat from core client for 30 sec - exiting 05:25:51 (3720): No heartbeat from core client for 30 sec - exiting 05:25:52 (3720): No heartbeat from core client for 30 sec - exiting 05:25:53 (3720): No heartbeat from core client for 30 sec - exiting 05:25:54 (3720): No heartbeat from core client for 30 sec - exiting 05:25:55 (3720): No heartbeat from core client for 30 sec - exiting 05:25:57 (3720): No heartbeat from core client for 30 sec - exiting 05:25:58 (3720): No heartbeat from core client for 30 sec - exiting 05:29:34 (996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:29:35 (996): No heartbeat from core client for 30 sec - exiting 05:29:36 (996): No heartbeat from core client for 30 sec - exiting 05:33:19 (3840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:33:20 (3840): No heartbeat from core client for 30 sec - exiting 05:33:21 (3840): No heartbeat from core client for 30 sec - exiting 05:33:22 (3840): No heartbeat from core client for 30 sec - exiting 05:35:28 (3656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:35:30 (3656): No heartbeat from core client for 30 sec - exiting 05:35:31 (3656): No heartbeat from core client for 30 sec - exiting 05:35:32 (3656): No heartbeat from core client for 30 sec - exiting 05:35:33 (3656): No heartbeat from core client for 30 sec - exiting 05:35:34 (3656): No heartbeat from core client for 30 sec - exiting 05:39:43 (3992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:41:41 (2740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:41:43 (2740): No heartbeat from core client for 30 sec - exiting 05:41:44 (2740): No heartbeat from core client for 30 sec - exiting 05:44:23 (1112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:44:24 (1112): No heartbeat from core client for 30 sec - exiting 05:46:18 (3528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:46:19 (3528): No heartbeat from core client for 30 sec - exiting 05:46:21 (3528): No heartbeat from core client for 30 sec - exiting 05:51:06 (3128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:02:25 (2720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3328, iMonCtr=1 Model crash detected, will try to restart... 06:05:37 (3328): No heartbeat from core client for 30 sec - exiting 06:05:38 (3328): No heartbeat from core client for 30 sec - exiting 06:05:39 (3328): No heartbeat from core client for 30 sec - exiting 06:05:40 (3328): No heartbeat from core client for 30 sec - exiting 06:05:41 (3328): No heartbeat from core client for 30 sec - exiting 06:05:42 (3328): No heartbeat from core client for 30 sec - exiting 06:05:43 (3328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:06:16 (3328): No heartbeat from core client for 30 sec - exiting 06:06:17 (3328): No heartbeat from core client for 30 sec - exiting 06:06:18 (3328): No heartbeat from core client for 30 sec - exiting 06:06:19 (3328): No heartbeat from core client for 30 sec - exiting 06:06:20 (3328): No heartbeat from core client for 30 sec - exiting 06:06:21 (3328): No heartbeat from core client for 30 sec - exiting 06:06:22 (3328): No heartbeat from core client for 30 sec - exiting 06:08:09 (3656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:08:10 (3656): No heartbeat from core client for 30 sec - exiting 06:08:11 (3656): No heartbeat from core client for 30 sec - exiting 06:08:12 (3656): No heartbeat from core client for 30 sec - exiting 06:08:13 (3656): No heartbeat from core client for 30 sec - exiting 06:08:15 (3656): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3312, iMonCtr=1 Model crash detected, will try to restart... 06:10:42 (3312): No heartbeat from core client for 30 sec - exiting 06:10:43 (3312): No heartbeat from core client for 30 sec - exiting 06:10:44 (3312): No heartbeat from core client for 30 sec - exiting 06:10:45 (3312): No heartbeat from core client for 30 sec - exiting 06:10:46 (3312): No heartbeat from core client for 30 sec - exiting 06:10:47 (3312): No heartbeat from core client for 30 sec - exiting 06:10:48 (3312): No heartbeat from core client for 30 sec - exiting 06:10:50 (3312): No heartbeat from core client for 30 sec - exiting 06:10:51 (3312): No heartbeat from core client for 30 sec - exiting 06:10:52 (3312): No heartbeat from core client for 30 sec - exiting 06:10:53 (3312): No heartbeat from core client for 30 sec - exiting 06:10:54 (3312): No heartbeat from core client for 30 sec - exiting 06:10:55 (3312): No heartbeat from core client for 30 sec - exiting 06:10:56 (3312): No heartbeat from core client for 30 sec - exiting 06:10:57 (3312): No heartbeat from core client for 30 sec - exiting 06:10:58 (3312): No heartbeat from core client for 30 sec - exiting 06:10:59 (3312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:11:00 (3312): No heartbeat from core client for 30 sec - exiting 06:11:01 (3312): No heartbeat from core client for 30 sec - exiting 06:11:02 (3312): No heartbeat from core client for 30 sec - exiting 06:11:03 (3312): No heartbeat from core client for 30 sec - exiting 06:11:04 (3312): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3016, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3408, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3408, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3408, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
03 Aug 2011 16:36:21	1135626	13158908	hadcm3n_p7b7_1900_40_007227059_2	440,640	633,206	1.4370
02 Aug 2011 22:02:27	1135626	13158908	hadcm3n_p7b7_1900_40_007227059_2	414,720	598,046	1.4420
02 Aug 2011 10:39:41	1135626	13158908	hadcm3n_p7b7_1900_40_007227059_2	388,800	563,775	1.4500
01 Aug 2011 17:57:03	1135626	13158908	hadcm3n_p7b7_1900_40_007227059_2	362,880	527,998	1.4550
01 Aug 2011 03:50:34	1135626	13158908	hadcm3n_p7b7_1900_40_007227059_2	336,960	490,817	1.4566
31 Jul 2011 16:16:10	1135626	13158908	hadcm3n_p7b7_1900_40_007227059_2	311,040	453,086	1.4567
31 Jul 2011 03:53:30	1135626	13158908	hadcm3n_p7b7_1900_40_007227059_2	285,120	414,519	1.4538
30 Jul 2011 15:14:44	1135626	13158908	hadcm3n_p7b7_1900_40_007227059_2	259,200	376,227	1.4515
30 Jul 2011 02:03:19	1135626	13158908	hadcm3n_p7b7_1900_40_007227059_2	233,280	338,422	1.4507
29 Jul 2011 14:47:59	1135626	13158908	hadcm3n_p7b7_1900_40_007227059_2	207,360	301,176	1.4524
29 Jul 2011 02:44:34	1135626	13158908	hadcm3n_p7b7_1900_40_007227059_2	181,440	263,633	1.4530
28 Jul 2011 14:58:02	1135626	13158908	hadcm3n_p7b7_1900_40_007227059_2	155,520	225,388	1.4493
28 Jul 2011 02:59:17	1135626	13158908	hadcm3n_p7b7_1900_40_007227059_2	129,600	188,398	1.4537
27 Jul 2011 15:03:38	1135626	13158908	hadcm3n_p7b7_1900_40_007227059_2	103,680	150,888	1.4553
27 Jul 2011 03:23:19	1135626	13158908	hadcm3n_p7b7_1900_40_007227059_2	77,760	113,030	1.4536
26 Jul 2011 17:17:21	1135626	13158908	hadcm3n_p7b7_1900_40_007227059_2	51,840	75,822	1.4626
26 Jul 2011 05:04:43	1135626	13158908	hadcm3n_p7b7_1900_40_007227059_2	25,920	37,649	1.4525