Task 12980353

Name	hadcm3n_d7q3_1940_40_007295475_0
Workunit	7492749
Created	15 Jun 2011, 10:39:05 UTC
Sent	15 Jun 2011, 10:39:15 UTC
Report deadline	14 Sep 2011, 18:06:26 UTC
Received	8 Jul 2011, 4:53:34 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1153245
Run time	7 days 0 hours 44 min 47 sec
CPU time	6 days 9 hours 10 min 46 sec
Validate state	Invalid
Credit	2,799.36
Device peak FLOPS	2.57 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:25:16 (6896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:25:19 (6896): No heartbeat from core client for 30 sec - exiting 18:25:20 (6896): No heartbeat from core client for 30 sec - exiting 18:25:21 (6896): No heartbeat from core client for 30 sec - exiting 18:25:22 (6896): No heartbeat from core client for 30 sec - exiting 18:25:23 (6896): No heartbeat from core client for 30 sec - exiting 18:25:24 (6896): No heartbeat from core client for 30 sec - exiting 18:25:25 (6896): No heartbeat from core client for 30 sec - exiting 18:25:26 (6896): No heartbeat from core client for 30 sec - exiting 18:25:27 (6896): No heartbeat from core client for 30 sec - exiting 18:25:28 (6896): No heartbeat from core client for 30 sec - exiting 18:25:29 (6896): No heartbeat from core client for 30 sec - exiting 18:25:30 (6896): No heartbeat from core client for 30 sec - exiting 18:25:31 (6896): No heartbeat from core client for 30 sec - exiting 18:25:32 (6896): No heartbeat from core client for 30 sec - exiting 18:25:33 (6896): No heartbeat from core client for 30 sec - exiting 18:25:34 (6896): No heartbeat from core client for 30 sec - exiting 18:25:35 (6896): No heartbeat from core client for 30 sec - exiting 18:25:36 (6896): No heartbeat from core client for 30 sec - exiting 18:25:37 (6896): No heartbeat from core client for 30 sec - exiting 18:25:38 (6896): No heartbeat from core client for 30 sec - exiting 18:25:39 (6896): No heartbeat from core client for 30 sec - exiting 18:25:40 (6896): No heartbeat from core client for 30 sec - exiting 18:25:41 (6896): No heartbeat from core client for 30 sec - exiting 18:25:42 (6896): No heartbeat from core client for 30 sec - exiting 18:25:43 (6896): No heartbeat from core client for 30 sec - exiting 18:25:44 (6896): No heartbeat from core client for 30 sec - exiting 18:25:45 (6896): No heartbeat from core client for 30 sec - exiting 18:25:46 (6896): No heartbeat from core client for 30 sec - exiting 18:25:47 (6896): No heartbeat from core client for 30 sec - exiting 18:25:48 (6896): No heartbeat from core client for 30 sec - exiting 18:25:49 (6896): No heartbeat from core client for 30 sec - exiting 18:25:50 (6896): No heartbeat from core client for 30 sec - exiting 18:25:51 (6896): No heartbeat from core client for 30 sec - exiting 18:25:52 (6896): No heartbeat from core client for 30 sec - exiting 18:25:53 (6896): No heartbeat from core client for 30 sec - exiting 18:25:54 (6896): No heartbeat from core client for 30 sec - exiting 18:25:55 (6896): No heartbeat from core client for 30 sec - exiting 18:25:56 (6896): No heartbeat from core client for 30 sec - exiting 18:25:57 (6896): No heartbeat from core client for 30 sec - exiting 18:25:58 (6896): No heartbeat from core client for 30 sec - exiting 18:25:59 (6896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6852, selfPID=6852, iMonCtr=1 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/d7q3ko.pje1c10 Error converting file to netcdf: dataout/d7q3ko.pie1c10 Error converting file to netcdf: dataout/d7q3ko.pfe1c10 Error converting file to netcdf: dataout/d7q3ka.phe1c10 Error converting file to netcdf: dataout/d7q3ka.pge1c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=0, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:14:47 (8644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:19:58 (8644): No heartbeat from core client for 30 sec - exiting 19:19:59 (8644): No heartbeat from core client for 30 sec - exiting 19:20:00 (8644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:00:13 (7832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... forrtl: Access is denied. CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7896, selfPID=7896, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:24:41 (9892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:24:42 (9892): No heartbeat from core client for 30 sec - exiting 19:24:43 (9892): No heartbeat from core client for 30 sec - exiting 19:24:44 (9892): No heartbeat from core client for 30 sec - exiting 19:24:45 (9892): No heartbeat from core client for 30 sec - exiting 19:24:46 (9892): No heartbeat from core client for 30 sec - exiting 19:24:47 (9892): No heartbeat from core client for 30 sec - exiting 19:24:48 (9892): No heartbeat from core client for 30 sec - exiting 19:24:49 (9892): No heartbeat from core client for 30 sec - exiting 19:24:50 (9892): No heartbeat from core client for 30 sec - exiting 19:24:51 (9892): No heartbeat from core client for 30 sec - exiting 19:24:52 (9892): No heartbeat from core client for 30 sec - exiting 19:24:53 (9892): No heartbeat from core client for 30 sec - exiting 19:24:54 (9892): No heartbeat from core client for 30 sec - exiting 19:24:55 (9892): No heartbeat from core client for 30 sec - exiting 19:24:56 (9892): No heartbeat from core client for 30 sec - exiting 19:24:57 (9892): No heartbeat from core client for 30 sec - exiting 19:24:58 (9892): No heartbeat from core client for 30 sec - exiting 19:24:59 (9892): No heartbeat from core client for 30 sec - exiting 19:25:00 (9892): No heartbeat from core client for 30 sec - exiting 19:25:01 (9892): No heartbeat from core client for 30 sec - exiting 19:25:02 (9892): No heartbeat from core client for 30 sec - exiting 19:25:03 (9892): No heartbeat from core client for 30 sec - exiting 19:25:04 (9892): No heartbeat from core client for 30 sec - exiting 19:25:05 (9892): No heartbeat from core client for 30 sec - exiting 19:25:06 (9892): No heartbeat from core client for 30 sec - exiting 19:25:07 (9892): No heartbeat from core client for 30 sec - exiting 19:25:08 (9892): No heartbeat from core client for 30 sec - exiting 19:25:09 (9892): No heartbeat from core client for 30 sec - exiting 19:39:30 (8476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:39:31 (8476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:08:03 (9720): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4820, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4820, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7220, selfPID=7220, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/d7q3ko.pje5c10 CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/d7q3ko.pje5c10 Error converting file to netcdf: dataout/d7q3ko.pie5c10 Error converting file to netcdf: dataout/d7q3ko.pfe5c10 Error converting file to netcdf: dataout/d7q3ka.phe5c10 Error converting file to netcdf: dataout/d7q3ka.pge5c10 Error converting file to netcdf: dataout/d7q3ka.pee5c10 Error converting file to netcdf: dataout/d7q3ka.pde5c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:01:37 (7320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:15:09 (460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:06:30 (6072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:45:11 (10092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:08:48 (9592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:05:49 (7804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:15:13 (6664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:15:15 (6664): No heartbeat from core client for 30 sec - exiting 19:15:31 (6664): No heartbeat from core client for 30 sec - exiting 19:15:32 (6664): No heartbeat from core client for 30 sec - exiting 19:15:33 (6664): No heartbeat from core client for 30 sec - exiting 19:15:34 (6664): No heartbeat from core client for 30 sec - exiting 19:15:35 (6664): No heartbeat from core client for 30 sec - exiting 19:15:36 (6664): No heartbeat from core client for 30 sec - exiting 19:15:37 (6664): No heartbeat from core client for 30 sec - exiting 19:15:38 (6664): No heartbeat from core client for 30 sec - exiting 19:15:39 (6664): No heartbeat from core client for 30 sec - exiting 19:15:40 (6664): No heartbeat from core client for 30 sec - exiting 19:15:41 (6664): No heartbeat from core client for 30 sec - exiting 19:15:42 (6664): No heartbeat from core client for 30 sec - exiting 19:15:43 (6664): No heartbeat from core client for 30 sec - exiting 19:15:44 (6664): No heartbeat from core client for 30 sec - exiting 19:15:45 (6664): No heartbeat from core client for 30 sec - exiting 19:15:46 (6664): No heartbeat from core client for 30 sec - exiting 19:15:47 (6664): No heartbeat from core client for 30 sec - exiting 19:15:48 (6664): No heartbeat from core client for 30 sec - exiting 19:15:49 (6664): No heartbeat from core client for 30 sec - exiting 19:15:50 (6664): No heartbeat from core client for 30 sec - exiting 19:15:51 (6664): No heartbeat from core client for 30 sec - exiting 19:15:52 (6664): No heartbeat from core client for 30 sec - exiting Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
08 Jul 2011 04:57:17	1153245	12980353	hadcm3n_d7q3_1940_40_007295475_0	233,280	546,698	2.3435
05 Jul 2011 06:58:08	1153245	12980353	hadcm3n_d7q3_1940_40_007295475_0	207,360	478,829	2.3092
30 Jun 2011 09:50:09	1153245	12980353	hadcm3n_d7q3_1940_40_007295475_0	181,440	417,743	2.3024
29 Jun 2011 04:22:15	1153245	12980353	hadcm3n_d7q3_1940_40_007295475_0	155,520	358,300	2.3039
28 Jun 2011 02:59:52	1153245	12980353	hadcm3n_d7q3_1940_40_007295475_0	129,600	299,463	2.3107
25 Jun 2011 17:12:17	1153245	12980353	hadcm3n_d7q3_1940_40_007295475_0	103,680	244,119	2.3545
23 Jun 2011 12:01:47	1153245	12980353	hadcm3n_d7q3_1940_40_007295475_0	77,760	182,852	2.3515
21 Jun 2011 13:10:48	1153245	12980353	hadcm3n_d7q3_1940_40_007295475_0	51,840	123,206	2.3767
17 Jun 2011 10:51:17	1153245	12980353	hadcm3n_d7q3_1940_40_007295475_0	25,920	62,851	2.4248