Task 15486706

Name	hadcm3n_3hl6_1940_40_008259075_0
Workunit	8414199
Created	20 Dec 2012, 13:15:04 UTC
Sent	20 Dec 2012, 13:15:09 UTC
Report deadline	21 Mar 2013, 20:42:20 UTC
Received	4 Feb 2013, 16:19:55 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1143060
Run time	4 days 23 hours 2 min 46 sec
CPU time	3 days 19 hours 22 min 13 sec
Validate state	Invalid
Credit	1,244.16
Device peak FLOPS	2.52 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.60</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5144, iMonCtr=1 Model crash detected, will try to restart... 08:09:31 (2952): No heartbeat from core client for 30 sec - exiting 08:09:33 (2952): No heartbeat from core client for 30 sec - exiting 08:09:34 (2952): No heartbeat from core client for 30 sec - exiting 08:09:35 (2952): No heartbeat from core client for 30 sec - exiting 08:09:36 (2952): No heartbeat from core client for 30 sec - exiting 08:09:37 (2952): No heartbeat from core client for 30 sec - exiting 08:09:38 (2952): No heartbeat from core client for 30 sec - exiting 08:09:39 (2952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:57:19 (3976): No heartbeat from core client for 30 sec - exiting 07:57:20 (3976): No heartbeat from core client for 30 sec - exiting 07:57:21 (3976): No heartbeat from core client for 30 sec - exiting 07:57:23 (3976): No heartbeat from core client for 30 sec - exiting 07:57:24 (3976): No heartbeat from core client for 30 sec - exiting 07:57:25 (3976): No heartbeat from core client for 30 sec - exiting 07:57:26 (3976): No heartbeat from core client for 30 sec - exiting 07:57:27 (3976): No heartbeat from core client for 30 sec - exiting 07:57:28 (3976): No heartbeat from core client for 30 sec - exiting 07:57:29 (3976): No heartbeat from core client for 30 sec - exiting 07:57:30 (3976): No heartbeat from core client for 30 sec - exiting 07:57:31 (3976): No heartbeat from core client for 30 sec - exiting 07:57:32 (3976): No heartbeat from core client for 30 sec - exiting 07:57:34 (3976): No heartbeat from core client for 30 sec - exiting 07:57:35 (3976): No heartbeat from core client for 30 sec - exiting 07:57:36 (3976): No heartbeat from core client for 30 sec - exiting 07:57:37 (3976): No heartbeat from core client for 30 sec - exiting 07:57:38 (3976): No heartbeat from core client for 30 sec - exiting 07:57:39 (3976): No heartbeat from core client for 30 sec - exiting 07:57:40 (3976): No heartbeat from core client for 30 sec - exiting 07:57:41 (3976): No heartbeat from core client for 30 sec - exiting 07:57:42 (3976): No heartbeat from core client for 30 sec - exiting 07:57:44 (3976): No heartbeat from core client for 30 sec - exiting 07:57:45 (3976): No heartbeat from core client for 30 sec - exiting 07:57:46 (3976): No heartbeat from core client for 30 sec - exiting 07:57:47 (3976): No heartbeat from core client for 30 sec - exiting 07:57:48 (3976): No heartbeat from core client for 30 sec - exiting 07:57:49 (3976): No heartbeat from core client for 30 sec - exiting 07:57:50 (3976): No heartbeat from core client for 30 sec - exiting 07:57:51 (3976): No heartbeat from core client for 30 sec - exiting 07:57:52 (3976): No heartbeat from core client for 30 sec - exiting 07:57:54 (3976): No heartbeat from core client for 30 sec - exiting 07:57:55 (3976): No heartbeat from core client for 30 sec - exiting 07:57:56 (3976): No heartbeat from core client for 30 sec - exiting 07:57:57 (3976): No heartbeat from core client for 30 sec - exiting 07:57:58 (3976): No heartbeat from core client for 30 sec - exiting 07:57:59 (3976): No heartbeat from core client for 30 sec - exiting 07:58:00 (3976): No heartbeat from core client for 30 sec - exiting 07:58:01 (3976): No heartbeat from core client for 30 sec - exiting 07:58:02 (3976): No heartbeat from core client for 30 sec - exiting 07:58:03 (3976): No heartbeat from core client for 30 sec - exiting 07:58:05 (3976): No heartbeat from core client for 30 sec - exiting 07:58:06 (3976): No heartbeat from core client for 30 sec - exiting 07:58:07 (3976): No heartbeat from core client for 30 sec - exiting 07:58:08 (3976): No heartbeat from core client for 30 sec - exiting 07:58:09 (3976): No heartbeat from core client for 30 sec - exiting 07:58:10 (3976): No heartbeat from core client for 30 sec - exiting 07:58:11 (3976): No heartbeat from core client for 30 sec - exiting 07:58:12 (3976): No heartbeat from core client for 30 sec - exiting 07:58:13 (3976): No heartbeat from core client for 30 sec - exiting 07:58:15 (3976): No heartbeat from core client for 30 sec - exiting 07:58:16 (3976): No heartbeat from core client for 30 sec - exiting 07:58:17 (3976): No heartbeat from core client for 30 sec - exiting 07:58:18 (3976): No heartbeat from core client for 30 sec - exiting 07:58:19 (3976): No heartbeat from core client for 30 sec - exiting 07:58:20 (3976): No heartbeat from core client for 30 sec - exiting 07:58:21 (3976): No heartbeat from core client for 30 sec - exiting 07:58:22 (3976): No heartbeat from core client for 30 sec - exiting 07:58:23 (3976): No heartbeat from core client for 30 sec - exiting 07:58:25 (3976): No heartbeat from core client for 30 sec - exiting 07:58:26 (3976): No heartbeat from core client for 30 sec - exiting 07:58:27 (3976): No heartbeat from core client for 30 sec - exiting 07:58:28 (3976): No heartbeat from core client for 30 sec - exiting 07:58:29 (3976): No heartbeat from core client for 30 sec - exiting 07:58:30 (3976): No heartbeat from core client for 30 sec - exiting 07:58:31 (3976): No heartbeat from core client for 30 sec - exiting 07:58:32 (3976): No heartbeat from core client for 30 sec - exiting 07:58:33 (3976): No heartbeat from core client for 30 sec - exiting 07:58:34 (3976): No heartbeat from core client for 30 sec - exiting 07:58:35 (3976): No heartbeat from core client for 30 sec - exiting 07:58:36 (3976): No heartbeat from core client for 30 sec - exiting 07:58:37 (3976): No heartbeat from core client for 30 sec - exiting 07:58:38 (3976): No heartbeat from core client for 30 sec - exiting 07:58:39 (3976): No heartbeat from core client for 30 sec - exiting 07:58:40 (3976): No heartbeat from core client for 30 sec - exiting 07:58:41 (3976): No heartbeat from core client for 30 sec - exiting 07:58:42 (3976): No heartbeat from core client for 30 sec - exiting 07:58:43 (3976): No heartbeat from core client for 30 sec - exiting 07:58:44 (3976): No heartbeat from core client for 30 sec - exiting 07:58:45 (3976): No heartbeat from core client for 30 sec - exiting 07:58:46 (3976): No heartbeat from core client for 30 sec - exiting 07:58:47 (3976): No heartbeat from core client for 30 sec - exiting 07:58:48 (3976): No heartbeat from core client for 30 sec - exiting 07:58:49 (3976): No heartbeat from core client for 30 sec - exiting 07:58:50 (3976): No heartbeat from core client for 30 sec - exiting 07:58:51 (3976): No heartbeat from core client for 30 sec - exiting 07:58:52 (3976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=636, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=636, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3612, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3612, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3612, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3612, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2460, iMonCtr=1 Model crash detected, will try to restart... 13:07:09 (3116): No heartbeat from core client for 30 sec - exiting 13:07:11 (3116): No heartbeat from core client for 30 sec - exiting 13:07:12 (3116): No heartbeat from core client for 30 sec - exiting 13:07:13 (3116): No heartbeat from core client for 30 sec - exiting 13:07:14 (3116): No heartbeat from core client for 30 sec - exiting 13:07:15 (3116): No heartbeat from core client for 30 sec - exiting 13:07:16 (3116): No heartbeat from core client for 30 sec - exiting 13:07:17 (3116): No heartbeat from core client for 30 sec - exiting 13:07:18 (3116): No heartbeat from core client for 30 sec - exiting 13:07:19 (3116): No heartbeat from core client for 30 sec - exiting 13:07:21 (3116): No heartbeat from core client for 30 sec - exiting 13:07:22 (3116): No heartbeat from core client for 30 sec - exiting 13:07:23 (3116): No heartbeat from core client for 30 sec - exiting 13:07:24 (3116): No heartbeat from core client for 30 sec - exiting 13:07:25 (3116): No heartbeat from core client for 30 sec - exiting 13:07:26 (3116): No heartbeat from core client for 30 sec - exiting 13:07:27 (3116): No heartbeat from core client for 30 sec - exiting 13:07:28 (3116): No heartbeat from core client for 30 sec - exiting 13:07:29 (3116): No heartbeat from core client for 30 sec - exiting 13:07:30 (3116): No heartbeat from core client for 30 sec - exiting 13:07:31 (3116): No heartbeat from core client for 30 sec - exiting 13:07:33 (3116): No heartbeat from core client for 30 sec - exiting 13:07:34 (3116): No heartbeat from core client for 30 sec - exiting 13:07:35 (3116): No heartbeat from core client for 30 sec - exiting 13:07:36 (3116): No heartbeat from core client for 30 sec - exiting 13:07:37 (3116): No heartbeat from core client for 30 sec - exiting 13:07:38 (3116): No heartbeat from core client for 30 sec - exiting 13:07:39 (3116): No heartbeat from core client for 30 sec - exiting 13:07:40 (3116): No heartbeat from core client for 30 sec - exiting 13:07:41 (3116): No heartbeat from core client for 30 sec - exiting 13:07:42 (3116): No heartbeat from core client for 30 sec - exiting 13:07:43 (3116): No heartbeat from core client for 30 sec - exiting 13:07:45 (3116): No heartbeat from core client for 30 sec - exiting 13:07:46 (3116): No heartbeat from core client for 30 sec - exiting 13:07:47 (3116): No heartbeat from core client for 30 sec - exiting 13:07:48 (3116): No heartbeat from core client for 30 sec - exiting 13:07:49 (3116): No heartbeat from core client for 30 sec - exiting 13:07:50 (3116): No heartbeat from core client for 30 sec - exiting 13:07:51 (3116): No heartbeat from core client for 30 sec - exiting 13:07:52 (3116): No heartbeat from core client for 30 sec - exiting 13:07:53 (3116): No heartbeat from core client for 30 sec - exiting 13:07:54 (3116): No heartbeat from core client for 30 sec - exiting 13:07:55 (3116): No heartbeat from core client for 30 sec - exiting 13:07:56 (3116): No heartbeat from core client for 30 sec - exiting 13:07:58 (3116): No heartbeat from core client for 30 sec - exiting 13:07:59 (3116): No heartbeat from core client for 30 sec - exiting 13:08:00 (3116): No heartbeat from core client for 30 sec - exiting 13:08:01 (3116): No heartbeat from core client for 30 sec - exiting 13:08:02 (3116): No heartbeat from core client for 30 sec - exiting 13:08:03 (3116): No heartbeat from core client for 30 sec - exiting 13:08:04 (3116): No heartbeat from core client for 30 sec - exiting 13:08:05 (3116): No heartbeat from core client for 30 sec - exiting 13:08:06 (3116): No heartbeat from core client for 30 sec - exiting 13:08:07 (3116): No heartbeat from core client for 30 sec - exiting 13:08:09 (3116): No heartbeat from core client for 30 sec - exiting 13:08:10 (3116): No heartbeat from core client for 30 sec - exiting 13:08:11 (3116): No heartbeat from core client for 30 sec - exiting 13:08:12 (3116): No heartbeat from core client for 30 sec - exiting 13:08:13 (3116): No heartbeat from core client for 30 sec - exiting 13:08:14 (3116): No heartbeat from core client for 30 sec - exiting 13:08:15 (3116): No heartbeat from core client for 30 sec - exiting 13:08:16 (3116): No heartbeat from core client for 30 sec - exiting 13:08:17 (3116): No heartbeat from core client for 30 sec - exiting 13:08:18 (3116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:39:15 (3180): No heartbeat from core client for 30 sec - exiting 10:39:16 (3180): No heartbeat from core client for 30 sec - exiting 10:39:17 (3180): No heartbeat from core client for 30 sec - exiting 10:39:18 (3180): No heartbeat from core client for 30 sec - exiting 10:39:20 (3180): No heartbeat from core client for 30 sec - exiting 10:39:21 (3180): No heartbeat from core client for 30 sec - exiting 10:39:22 (3180): No heartbeat from core client for 30 sec - exiting 10:39:23 (3180): No heartbeat from core client for 30 sec - exiting 10:39:24 (3180): No heartbeat from core client for 30 sec - exiting 10:39:25 (3180): No heartbeat from core client for 30 sec - exiting 10:39:26 (3180): No heartbeat from core client for 30 sec - exiting 10:39:27 (3180): No heartbeat from core client for 30 sec - exiting 10:39:28 (3180): No heartbeat from core client for 30 sec - exiting 10:39:29 (3180): No heartbeat from core client for 30 sec - exiting 10:39:30 (3180): No heartbeat from core client for 30 sec - exiting 10:39:32 (3180): No heartbeat from core client for 30 sec - exiting 10:39:33 (3180): No heartbeat from core client for 30 sec - exiting 10:39:34 (3180): No heartbeat from core client for 30 sec - exiting 10:39:35 (3180): No heartbeat from core client for 30 sec - exiting 10:39:36 (3180): No heartbeat from core client for 30 sec - exiting 10:39:37 (3180): No heartbeat from core client for 30 sec - exiting 10:39:38 (3180): No heartbeat from core client for 30 sec - exiting 10:39:39 (3180): No heartbeat from core client for 30 sec - exiting 10:39:40 (3180): No heartbeat from core client for 30 sec - exiting 10:39:41 (3180): No heartbeat from core client for 30 sec - exiting 10:39:42 (3180): No heartbeat from core client for 30 sec - exiting 10:39:44 (3180): No heartbeat from core client for 30 sec - exiting 10:39:45 (3180): No heartbeat from core client for 30 sec - exiting 10:39:46 (3180): No heartbeat from core client for 30 sec - exiting 10:39:47 (3180): No heartbeat from core client for 30 sec - exiting 10:39:48 (3180): No heartbeat from core client for 30 sec - exiting 10:39:49 (3180): No heartbeat from core client for 30 sec - exiting 10:39:50 (3180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6036, iMonCtr=1 Model crash detected, will try to restart... 09:46:45 (3984): No heartbeat from core client for 30 sec - exiting 09:46:46 (3984): No heartbeat from core client for 30 sec - exiting 09:46:47 (3984): No heartbeat from core client for 30 sec - exiting 09:46:49 (3984): No heartbeat from core client for 30 sec - exiting 09:46:50 (3984): No heartbeat from core client for 30 sec - exiting 09:46:51 (3984): No heartbeat from core client for 30 sec - exiting 09:46:52 (3984): No heartbeat from core client for 30 sec - exiting 09:46:53 (3984): No heartbeat from core client for 30 sec - exiting 09:46:54 (3984): No heartbeat from core client for 30 sec - exiting 09:46:55 (3984): No heartbeat from core client for 30 sec - exiting 09:46:56 (3984): No heartbeat from core client for 30 sec - exiting 09:46:57 (3984): No heartbeat from core client for 30 sec - exiting 09:46:58 (3984): No heartbeat from core client for 30 sec - exiting 09:46:59 (3984): No heartbeat from core client for 30 sec - exiting 09:47:00 (3984): No heartbeat from core client for 30 sec - exiting 09:47:01 (3984): No heartbeat from core client for 30 sec - exiting 09:47:02 (3984): No heartbeat from core client for 30 sec - exiting 09:47:03 (3984): No heartbeat from core client for 30 sec - exiting 09:47:05 (3984): No heartbeat from core client for 30 sec - exiting 09:47:06 (3984): No heartbeat from core client for 30 sec - exiting 09:47:07 (3984): No heartbeat from core client for 30 sec - exiting 09:47:08 (3984): No heartbeat from core client for 30 sec - exiting 09:47:09 (3984): No heartbeat from core client for 30 sec - exiting 09:47:10 (3984): No heartbeat from core client for 30 sec - exiting 09:47:11 (3984): No heartbeat from core client for 30 sec - exiting 09:47:12 (3984): No heartbeat from core client for 30 sec - exiting 09:47:13 (3984): No heartbeat from core client for 30 sec - exiting 09:47:14 (3984): No heartbeat from core client for 30 sec - exiting 09:47:15 (3984): No heartbeat from core client for 30 sec - exiting 09:47:16 (3984): No heartbeat from core client for 30 sec - exiting 09:47:17 (3984): No heartbeat from core client for 30 sec - exiting 09:47:18 (3984): No heartbeat from core client for 30 sec - exiting 09:47:20 (3984): No heartbeat from core client for 30 sec - exiting 09:47:21 (3984): No heartbeat from core client for 30 sec - exiting 09:47:22 (3984): No heartbeat from core client for 30 sec - exiting 09:47:23 (3984): No heartbeat from core client for 30 sec - exiting 09:47:24 (3984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4056, iMonCtr=1 Model crash detected, will try to restart... 08:11:03 (1332): No heartbeat from core client for 30 sec - exiting 08:11:04 (1332): No heartbeat from core client for 30 sec - exiting 08:11:05 (1332): No heartbeat from core client for 30 sec - exiting 08:11:06 (1332): No heartbeat from core client for 30 sec - exiting 08:11:07 (1332): No heartbeat from core client for 30 sec - exiting 08:11:08 (1332): No heartbeat from core client for 30 sec - exiting 08:11:09 (1332): No heartbeat from core client for 30 sec - exiting 08:11:10 (1332): No heartbeat from core client for 30 sec - exiting 08:11:11 (1332): No heartbeat from core client for 30 sec - exiting 08:11:12 (1332): No heartbeat from core client for 30 sec - exiting 08:11:13 (1332): No heartbeat from core client for 30 sec - exiting 08:11:14 (1332): No heartbeat from core client for 30 sec - exiting 08:11:15 (1332): No heartbeat from core client for 30 sec - exiting 08:11:16 (1332): No heartbeat from core client for 30 sec - exiting 08:11:17 (1332): No heartbeat from core client for 30 sec - exiting 08:11:18 (1332): No heartbeat from core client for 30 sec - exiting 08:11:19 (1332): No heartbeat from core client for 30 sec - exiting 08:11:20 (1332): No heartbeat from core client for 30 sec - exiting 08:11:21 (1332): No heartbeat from core client for 30 sec - exiting 08:11:22 (1332): No heartbeat from core client for 30 sec - exiting 08:11:23 (1332): No heartbeat from core client for 30 sec - exiting 08:11:24 (1332): No heartbeat from core client for 30 sec - exiting 08:11:25 (1332): No heartbeat from core client for 30 sec - exiting 08:11:26 (1332): No heartbeat from core client for 30 sec - exiting 08:11:27 (1332): No heartbeat from core client for 30 sec - exiting 08:11:28 (1332): No heartbeat from core client for 30 sec - exiting 08:11:29 (1332): No heartbeat from core client for 30 sec - exiting 08:11:30 (1332): No heartbeat from core client for 30 sec - exiting 08:11:31 (1332): No heartbeat from core client for 30 sec - exiting 08:11:32 (1332): No heartbeat from core client for 30 sec - exiting 08:11:33 (1332): No heartbeat from core client for 30 sec - exiting 08:11:34 (1332): No heartbeat from core client for 30 sec - exiting 08:11:35 (1332): No heartbeat from core client for 30 sec - exiting 08:11:36 (1332): No heartbeat from core client for 30 sec - exiting 08:11:37 (1332): No heartbeat from core client for 30 sec - exiting 08:11:38 (1332): No heartbeat from core client for 30 sec - exiting 08:11:39 (1332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:11:40 (1332): No heartbeat from core client for 30 sec - exiting 12:40:09 (5524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6484, iMonCtr=1 Model crash detected, will try to restart... 08:10:35 (1804): No heartbeat from core client for 30 sec - exiting 08:10:36 (1804): No heartbeat from core client for 30 sec - exiting 08:10:38 (1804): No heartbeat from core client for 30 sec - exiting 08:10:39 (1804): No heartbeat from core client for 30 sec - exiting 08:10:40 (1804): No heartbeat from core client for 30 sec - exiting 08:10:41 (1804): No heartbeat from core client for 30 sec - exiting 08:10:42 (1804): No heartbeat from core client for 30 sec - exiting 08:10:43 (1804): No heartbeat from core client for 30 sec - exiting 08:10:44 (1804): No heartbeat from core client for 30 sec - exiting 08:10:45 (1804): No heartbeat from core client for 30 sec - exiting 08:10:46 (1804): No heartbeat from core client for 30 sec - exiting 08:10:47 (1804): No heartbeat from core client for 30 sec - exiting 08:10:49 (1804): No heartbeat from core client for 30 sec - exiting 08:10:50 (1804): No heartbeat from core client for 30 sec - exiting 08:10:51 (1804): No heartbeat from core client for 30 sec - exiting 08:10:52 (1804): No heartbeat from core client for 30 sec - exiting 08:10:53 (1804): No heartbeat from core client for 30 sec - exiting 08:10:54 (1804): No heartbeat from core client for 30 sec - exiting 08:10:55 (1804): No heartbeat from core client for 30 sec - exiting 08:10:56 (1804): No heartbeat from core client for 30 sec - exiting 08:10:57 (1804): No heartbeat from core client for 30 sec - exiting 08:10:58 (1804): No heartbeat from core client for 30 sec - exiting 08:10:59 (1804): No heartbeat from core client for 30 sec - exiting 08:11:01 (1804): No heartbeat from core client for 30 sec - exiting 08:11:02 (1804): No heartbeat from core client for 30 sec - exiting 08:11:03 (1804): No heartbeat from core client for 30 sec - exiting 08:11:04 (1804): No heartbeat from core client for 30 sec - exiting 08:11:05 (1804): No heartbeat from core client for 30 sec - exiting 08:11:06 (1804): No heartbeat from core client for 30 sec - exiting 08:11:07 (1804): No heartbeat from core client for 30 sec - exiting 08:11:08 (1804): No heartbeat from core client for 30 sec - exiting 08:11:09 (1804): No heartbeat from core client for 30 sec - exiting 08:11:10 (1804): No heartbeat from core client for 30 sec - exiting 08:11:11 (1804): No heartbeat from core client for 30 sec - exiting 08:11:12 (1804): No heartbeat from core client for 30 sec - exiting 08:11:13 (1804): No heartbeat from core client for 30 sec - exiting 08:11:14 (1804): No heartbeat from core client for 30 sec - exiting 08:11:15 (1804): No heartbeat from core client for 30 sec - exiting 08:11:16 (1804): No heartbeat from core client for 30 sec - exiting 08:11:17 (1804): No heartbeat from core client for 30 sec - exiting 08:11:18 (1804): No heartbeat from core client for 30 sec - exiting 08:11:19 (1804): No heartbeat from core client for 30 sec - exiting 08:11:20 (1804): No heartbeat from core client for 30 sec - exiting 08:11:21 (1804): No heartbeat from core client for 30 sec - exiting 08:11:22 (1804): No heartbeat from core client for 30 sec - exiting 08:11:23 (1804): No heartbeat from core client for 30 sec - exiting 08:11:24 (1804): No heartbeat from core client for 30 sec - exiting 08:11:25 (1804): No heartbeat from core client for 30 sec - exiting 08:11:26 (1804): No heartbeat from core client for 30 sec - exiting 08:11:27 (1804): No heartbeat from core client for 30 sec - exiting 08:11:28 (1804): No heartbeat from core client for 30 sec - exiting 08:11:29 (1804): No heartbeat from core client for 30 sec - exiting 08:11:30 (1804): No heartbeat from core client for 30 sec - exiting 08:11:31 (1804): No heartbeat from core client for 30 sec - exiting 08:11:32 (1804): No heartbeat from core client for 30 sec - exiting 08:11:33 (1804): No heartbeat from core client for 30 sec - exiting 08:11:34 (1804): No heartbeat from core client for 30 sec - exiting 08:11:35 (1804): No heartbeat from core client for 30 sec - exiting 08:11:36 (1804): No heartbeat from core client for 30 sec - exiting 08:11:37 (1804): No heartbeat from core client for 30 sec - exiting 08:11:39 (1804): No heartbeat from core client for 30 sec - exiting 08:11:40 (1804): No heartbeat from core client for 30 sec - exiting 08:11:41 (1804): No heartbeat from core client for 30 sec - exiting 08:11:42 (1804): No heartbeat from core client for 30 sec - exiting 08:11:43 (1804): No heartbeat from core client for 30 sec - exiting 08:11:44 (1804): No heartbeat from core client for 30 sec - exiting 08:11:45 (1804): No heartbeat from core client for 30 sec - exiting 08:11:46 (1804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:11:47 (1804): No heartbeat from core client for 30 sec - exiting 08:11:48 (1804): No heartbeat from core client for 30 sec - exiting 08:14:47 (4736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2848, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2848, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2848, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2848, iMonCtr=1 Model crash detected, will try to restart... CCPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4564, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2668, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2668, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2668, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2668, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2668, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2668, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish 07:55:22 (2864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:57:48 (2888): No heartbeat from core client for 30 sec - exiting 07:57:49 (2888): No heartbeat from core client for 30 sec - exiting 07:57:50 (2888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 41 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 41 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 41 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 41 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 41 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 41 - Return code = 16 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
24 Jan 2013 22:53:16	1143060	15486706	hadcm3n_3hl6_1940_40_008259075_0	103,680	273,987	2.6426
21 Jan 2013 20:47:18	1143060	15486706	hadcm3n_3hl6_1940_40_008259075_0	77,760	205,426	2.6418
14 Jan 2013 17:21:33	1143060	15486706	hadcm3n_3hl6_1940_40_008259075_0	51,840	136,690	2.6368
07 Jan 2013 15:46:44	1143060	15486706	hadcm3n_3hl6_1940_40_008259075_0	25,920	68,901	2.6582