Task 12990123

Name	hadcm3n_s4c6_1940_40_007299785_0
Workunit	7497209
Created	20 Jun 2011, 20:23:19 UTC
Sent	20 Jun 2011, 20:23:42 UTC
Report deadline	20 Sep 2011, 3:50:53 UTC
Received	15 Jul 2011, 16:18:54 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-2 (0xFFFFFFFE) Unknown error code
Computer ID	835467
Run time	3 days 23 hours 49 min 14 sec
CPU time	3 days 9 hours 49 min 35 sec
Validate state	Invalid
Credit	2,488.32
Device peak FLOPS	2.97 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.13.0</core_client_version> <![CDATA[ <message> - exit code -2 (0xfffffffe) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:51:55 (2588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:52:30 (652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:16:18 (3040): No heartbeat from core client for 30 sec - exiting 21:16:19 (3040): No heartbeat from core client for 30 sec - exiting 21:16:20 (3040): No heartbeat from core client for 30 sec - exiting 21:16:21 (3040): No heartbeat from core client for 30 sec - exiting 21:16:22 (3040): No heartbeat from core client for 30 sec - exiting 21:16:23 (3040): No heartbeat from core client for 30 sec - exiting 21:16:24 (3040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:44:37 (1596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:09:53 (2660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:34:15 (2500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:57:43 (2308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:31:54 (1444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:18:08 (2968): No heartbeat from core client for 30 sec - exiting 17:18:09 (2968): No heartbeat from core client for 30 sec - exiting 17:18:10 (2968): No heartbeat from core client for 30 sec - exiting 17:18:11 (2968): No heartbeat from core client for 30 sec - exiting 17:18:12 (2968): No heartbeat from core client for 30 sec - exiting 17:18:13 (2968): No heartbeat from core client for 30 sec - exiting 17:18:14 (2968): No heartbeat from core client for 30 sec - exiting 17:18:15 (2968): No heartbeat from core client for 30 sec - exiting 17:18:16 (2968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:29:31 (1560): No heartbeat from core client for 30 sec - exiting 18:29:32 (1560): No heartbeat from core client for 30 sec - exiting 18:29:33 (1560): No heartbeat from core client for 30 sec - exiting 18:29:34 (1560): No heartbeat from core client for 30 sec - exiting 18:29:35 (1560): No heartbeat from core client for 30 sec - exiting 18:29:36 (1560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:31:43 (1668): No heartbeat from core client for 30 sec - exiting 06:31:44 (1668): No heartbeat from core client for 30 sec - exiting 06:31:45 (1668): No heartbeat from core client for 30 sec - exiting 06:31:46 (1668): No heartbeat from core client for 30 sec - exiting 06:31:47 (1668): No heartbeat from core client for 30 sec - exiting 06:31:48 (1668): No heartbeat from core client for 30 sec - exiting 06:31:49 (1668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/s4c6ko.pje3c10 Error converting file to netcdf: dataout/s4c6ko.pie3c10 Error converting file to netcdf: dataout/s4c6ko.pfe3c10 Error converting file to netcdf: dataout/s4c6ka.phe3c10 Error converting file to netcdf: dataout/s4c6ka.pge3c10 Error converting file to netcdf: dataout/s4c6ka.pee3c10 Error converting file to netcdf: dataout/s4c6ka.pde3c10 Suspended CPDN Monitor - Suspend request from BOINC... 19:15:02 (1644): No heartbeat from core client for 30 sec - exiting 19:15:03 (1644): No heartbeat from core client for 30 sec - exiting 19:15:04 (1644): No heartbeat from core client for 30 sec - exiting 19:15:05 (1644): No heartbeat from core client for 30 sec - exiting 19:15:06 (1644): No heartbeat from core client for 30 sec - exiting 19:15:07 (1644): No heartbeat from core client for 30 sec - exiting 19:15:08 (1644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:32:36 (2556): No heartbeat from core client for 30 sec - exiting 06:32:37 (2556): No heartbeat from core client for 30 sec - exiting 06:32:38 (2556): No heartbeat from core client for 30 sec - exiting 06:32:39 (2556): No heartbeat from core client for 30 sec - exiting 06:32:40 (2556): No heartbeat from core client for 30 sec - exiting 06:32:41 (2556): No heartbeat from core client for 30 sec - exiting 06:32:42 (2556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:14:20 (2852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:51:11 (2448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:23:05 (1424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:34:57 (1676): No heartbeat from core client for 30 sec - exiting 21:34:58 (1676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:34:32 (3708): No heartbeat from core client for 30 sec - exiting 06:34:33 (3708): No heartbeat from core client for 30 sec - exiting 06:34:34 (3708): No heartbeat from core client for 30 sec - exiting 06:34:35 (3708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:33:58 (2364): No heartbeat from core client for 30 sec - exiting 06:33:59 (2364): No heartbeat from core client for 30 sec - exiting 06:34:00 (2364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:25:21 (1552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:03:10 (1112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:07:58 (3184): No heartbeat from core client for 30 sec - exiting 17:07:59 (3184): No heartbeat from core client for 30 sec - exiting 17:08:00 (3184): No heartbeat from core client for 30 sec - exiting 17:08:01 (3184): No heartbeat from core client for 30 sec - exiting 17:08:02 (3184): No heartbeat from core client for 30 sec - exiting 17:08:03 (3184): No heartbeat from core client for 30 sec - exiting 17:08:04 (3184): No heartbeat from core client for 30 sec - exiting 17:08:05 (3184): No heartbeat from core client for 30 sec - exiting 17:08:06 (3184): No heartbeat from core client for 30 sec - exiting 17:08:07 (3184): No heartbeat from core client for 30 sec - exiting 17:08:08 (3184): No heartbeat from core client for 30 sec - exiting 17:08:09 (3184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:06:51 (3284): No heartbeat from core client for 30 sec - exiting 23:06:52 (3284): No heartbeat from core client for 30 sec - exiting 23:06:53 (3284): No heartbeat from core client for 30 sec - exiting 23:06:54 (3284): No heartbeat from core client for 30 sec - exiting 23:06:55 (3284): No heartbeat from core client for 30 sec - exiting 23:06:56 (3284): No heartbeat from core client for 30 sec - exiting 23:06:57 (3284): No heartbeat from core client for 30 sec - exiting 23:06:58 (3284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:16:59 (2820): No heartbeat from core client for 30 sec - exiting 08:17:00 (2820): No heartbeat from core client for 30 sec - exiting 08:17:01 (2820): No heartbeat from core client for 30 sec - exiting 08:17:02 (2820): No heartbeat from core client for 30 sec - exiting 08:17:03 (2820): No heartbeat from core client for 30 sec - exiting 08:17:04 (2820): No heartbeat from core client for 30 sec - exiting 08:17:05 (2820): No heartbeat from core client for 30 sec - exiting 08:17:06 (2820): No heartbeat from core client for 30 sec - exiting 08:17:07 (2820): No heartbeat from core client for 30 sec - exiting 08:17:08 (2820): No heartbeat from core client for 30 sec - exiting 08:17:09 (2820): No heartbeat from core client for 30 sec - exiting 08:17:10 (2820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:34:57 (4040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:28:49 (2304): No heartbeat from core client for 30 sec - exiting 21:28:50 (2304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:37:24 (3716): No heartbeat from core client for 30 sec - exiting 15:37:25 (3716): No heartbeat from core client for 30 sec - exiting 15:37:26 (3716): No heartbeat from core client for 30 sec - exiting 15:37:27 (3716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:33:19 (3964): No heartbeat from core client for 30 sec - exiting 18:33:20 (3964): No heartbeat from core client for 30 sec - exiting 18:33:21 (3964): No heartbeat from core client for 30 sec - exiting 18:33:22 (3964): No heartbeat from core client for 30 sec - exiting 18:33:23 (3964): No heartbeat from core client for 30 sec - exiting 18:33:24 (3964): No heartbeat from core client for 30 sec - exiting 18:33:25 (3964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:22:00 (3272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:47:56 (3164): No heartbeat from core client for 30 sec - exiting 17:47:57 (3164): No heartbeat from core client for 30 sec - exiting 17:47:58 (3164): No heartbeat from core client for 30 sec - exiting 17:47:59 (3164): No heartbeat from core client for 30 sec - exiting 17:48:00 (3164): No heartbeat from core client for 30 sec - exiting 17:48:01 (3164): No heartbeat from core client for 30 sec - exiting 17:48:02 (3164): No heartbeat from core client for 30 sec - exiting 17:48:03 (3164): No heartbeat from core client for 30 sec - exiting 17:48:04 (3164): No heartbeat from core client for 30 sec - exiting 17:48:05 (3164): No heartbeat from core client for 30 sec - exiting 17:48:06 (3164): No heartbeat from core client for 30 sec - exiting 17:48:07 (3164): No heartbeat from core client for 30 sec - exiting 17:48:08 (3164): No heartbeat from core client for 30 sec - exiting 17:48:09 (3164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:54:16 (320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 19:46:43 (3208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3960, iMonCtr=1 Model crash detected, will try to restart... 20:50:48 (3960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:28:53 (3972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3368, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 17:59:57 (952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Could not launch model process. Last Error=5 Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Jul 2011 12:28:15	835467	12990123	hadcm3n_s4c6_1940_40_007299785_0	207,360	266,124	1.2834
09 Jul 2011 13:52:03	835467	12990123	hadcm3n_s4c6_1940_40_007299785_0	181,440	233,953	1.2894
07 Jul 2011 18:13:14	835467	12990123	hadcm3n_s4c6_1940_40_007299785_0	155,520	200,418	1.2887
03 Jul 2011 20:29:48	835467	12990123	hadcm3n_s4c6_1940_40_007299785_0	129,600	166,950	1.2882
02 Jul 2011 17:41:54	835467	12990123	hadcm3n_s4c6_1940_40_007299785_0	103,680	132,981	1.2826
30 Jun 2011 05:38:18	835467	12990123	hadcm3n_s4c6_1940_40_007299785_0	77,760	99,567	1.2804
28 Jun 2011 05:37:08	835467	12990123	hadcm3n_s4c6_1940_40_007299785_0	51,840	66,248	1.2779
26 Jun 2011 11:50:39	835467	12990123	hadcm3n_s4c6_1940_40_007299785_0	25,920	32,896	1.2691