Task 15936808

Name	hadcm3n_n2kc_1960_40_008371836_3
Workunit	8522695
Created	23 Aug 2013, 10:50:30 UTC
Sent	23 Aug 2013, 10:58:17 UTC
Report deadline	22 Nov 2013, 18:25:28 UTC
Received	4 Sep 2013, 12:13:54 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1230461
Run time	1 days 23 hours 47 min 22 sec
CPU time	1 days 16 hours 57 min 22 sec
Validate state	Invalid
Credit	1,244.16
Device peak FLOPS	2.93 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 21:54:50 (133876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:54:59 (133876): No heartbeat from core client for 30 sec - exiting 21:55:00 (133876): No heartbeat from core client for 30 sec - exiting 22:07:07 (139028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 23:14:20 (139180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:14:21 (139180): No heartbeat from core client for 30 sec - exiting 23:14:22 (139180): No heartbeat from core client for 30 sec - exiting 23:14:23 (139180): No heartbeat from core client for 30 sec - exiting 23:14:24 (139180): No heartbeat from core client for 30 sec - exiting 23:14:25 (139180): No heartbeat from core client for 30 sec - exiting 23:14:26 (139180): No heartbeat from core client for 30 sec - exiting 23:14:27 (139180): No heartbeat from core client for 30 sec - exiting 23:14:28 (139180): No heartbeat from core client for 30 sec - exiting 23:14:29 (139180): No heartbeat from core client for 30 sec - exiting 23:14:30 (139180): No heartbeat from core client for 30 sec - exiting 23:28:26 (105432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:10:03 (34856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:10:07 (34856): No heartbeat from core client for 30 sec - exiting 03:10:08 (34856): No heartbeat from core client for 30 sec - exiting 03:10:09 (34856): No heartbeat from core client for 30 sec - exiting 04:06:03 (5108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:11:06 (70488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:24:28 (139276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:38:28 (127252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:44:11 (28828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:06:44 (105948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:06:49 (105948): No heartbeat from core client for 30 sec - exiting 05:06:50 (105948): No heartbeat from core client for 30 sec - exiting 05:06:51 (105948): No heartbeat from core client for 30 sec - exiting 05:06:52 (105948): No heartbeat from core client for 30 sec - exiting 05:06:53 (105948): No heartbeat from core client for 30 sec - exiting 05:06:54 (105948): No heartbeat from core client for 30 sec - exiting 05:06:55 (105948): No heartbeat from core client for 30 sec - exiting 05:06:56 (105948): No heartbeat from core client for 30 sec - exiting 05:06:57 (105948): No heartbeat from core client for 30 sec - exiting 05:20:00 (85424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:24:02 (139592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:41:03 (140140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:41:23 (140140): No heartbeat from core client for 30 sec - exiting 08:41:24 (140140): No heartbeat from core client for 30 sec - exiting 08:41:25 (140140): No heartbeat from core client for 30 sec - exiting 08:41:26 (140140): No heartbeat from core client for 30 sec - exiting 08:41:27 (140140): No heartbeat from core client for 30 sec - exiting 08:41:28 (140140): No heartbeat from core client for 30 sec - exiting 12:00:53 (105416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 09:45:23 (15692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:17:47 (12864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:18:08 (12864): No heartbeat from core client for 30 sec - exiting 11:18:09 (12864): No heartbeat from core client for 30 sec - exiting 11:18:11 (12864): No heartbeat from core client for 30 sec - exiting 11:18:12 (12864): No heartbeat from core client for 30 sec - exiting 11:18:13 (12864): No heartbeat from core client for 30 sec - exiting 11:18:14 (12864): No heartbeat from core client for 30 sec - exiting 11:18:15 (12864): No heartbeat from core client for 30 sec - exiting 11:18:16 (12864): No heartbeat from core client for 30 sec - exiting 11:18:17 (12864): No heartbeat from core client for 30 sec - exiting 12:02:26 (16648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:02:27 (16648): No heartbeat from core client for 30 sec - exiting 12:02:28 (16648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 23:39:04 (16992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:40:47 (18212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:57:56 (16300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:01:21 (15812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:40:05 (17644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:39:00 (14768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:18:22 (17648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:00:19 (14828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:00:22 (14828): No heartbeat from core client for 30 sec - exiting 07:00:24 (14828): No heartbeat from core client for 30 sec - exiting 07:00:25 (14828): No heartbeat from core client for 30 sec - exiting 07:00:26 (14828): No heartbeat from core client for 30 sec - exiting 07:00:27 (14828): No heartbeat from core client for 30 sec - exiting 07:00:28 (14828): No heartbeat from core client for 30 sec - exiting 07:00:29 (14828): No heartbeat from core client for 30 sec - exiting 07:34:08 (18104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:22:33 (18240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 11:34:16 (5028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:34:17 (5028): No heartbeat from core client for 30 sec - exiting 11:34:19 (5028): No heartbeat from core client for 30 sec - exiting 11:34:20 (5028): No heartbeat from core client for 30 sec - exiting 11:34:21 (5028): No heartbeat from core client for 30 sec - exiting 11:34:22 (5028): No heartbeat from core client for 30 sec - exiting 11:34:23 (5028): No heartbeat from core client for 30 sec - exiting 11:34:24 (5028): No heartbeat from core client for 30 sec - exiting 11:34:25 (5028): No heartbeat from core client for 30 sec - exiting 11:34:26 (5028): No heartbeat from core client for 30 sec - exiting 11:34:27 (5028): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 13:06:54 (16300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:40:25 (16664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:40:27 (16664): No heartbeat from core client for 30 sec - exiting 13:40:28 (16664): No heartbeat from core client for 30 sec - exiting 13:40:29 (16664): No heartbeat from core client for 30 sec - exiting 13:40:30 (16664): No heartbeat from core client for 30 sec - exiting 13:40:31 (16664): No heartbeat from core client for 30 sec - exiting 13:40:32 (16664): No heartbeat from core client for 30 sec - exiting 13:40:33 (16664): No heartbeat from core client for 30 sec - exiting 13:40:34 (16664): No heartbeat from core client for 30 sec - exiting 13:40:35 (16664): No heartbeat from core client for 30 sec - exiting 13:40:36 (16664): No heartbeat from core client for 30 sec - exiting 14:31:16 (18856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:34:09 (17632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:40:05 (18152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:40:13 (18152): No heartbeat from core client for 30 sec - exiting 17:40:14 (18152): No heartbeat from core client for 30 sec - exiting 17:40:15 (18152): No heartbeat from core client for 30 sec - exiting 17:40:16 (18152): No heartbeat from core client for 30 sec - exiting 17:40:17 (18152): No heartbeat from core client for 30 sec - exiting 17:40:19 (18152): No heartbeat from core client for 30 sec - exiting 17:40:20 (18152): No heartbeat from core client for 30 sec - exiting 17:40:21 (18152): No heartbeat from core client for 30 sec - exiting 17:40:22 (18152): No heartbeat from core client for 30 sec - exiting 17:40:23 (18152): No heartbeat from core client for 30 sec - exiting 18:05:33 (17952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:40:21 (16116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:43:20 (11424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:35:17 (18756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:18:06 (19184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:54:10 (18024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:19:14 (18252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:30:09 (14060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:21:17 (19436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:03:45 (15380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:01:42 (732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:07:01 (16108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:12:09 (3792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:28:31 (20324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:35:23 (19688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:29:11 (19484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:17:24 (18440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:51:50 (19536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:22:18 (20056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=20532, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=20532, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=20532, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=20532, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=20532, iMonCtr=1 Model crash detected, will try to restart... 08:49:38 (20532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=38220, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Sep 2013 04:42:50	1230461	15936808	hadcm3n_n2kc_1960_40_008371836_3	103,680	129,469	1.2487
31 Aug 2013 20:20:03	1230461	15936808	hadcm3n_n2kc_1960_40_008371836_3	77,760	103,826	1.3352
31 Aug 2013 11:15:56	1230461	15936808	hadcm3n_n2kc_1960_40_008371836_3	51,840	75,589	1.4581
29 Aug 2013 16:43:39	1230461	15936808	hadcm3n_n2kc_1960_40_008371836_3	25,920	40,158	1.5493