Task 15450330

Name	hadcm3n_zg8s_1880_40_008249864_1
Workunit	8404988
Created	22 Nov 2012, 1:50:12 UTC
Sent	22 Nov 2012, 1:50:21 UTC
Report deadline	21 Feb 2013, 9:17:32 UTC
Received	18 Feb 2013, 18:48:12 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1133571
Run time	16 days 1 hours 13 min 38 sec
CPU time	9 days 11 hours 52 min 14 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.11 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 09:45:49 (5660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:46:05 (6040): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5768, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4424, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4872, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:27:06 (2784): No heartbeat from core client for 30 sec - exiting 18:27:07 (2784): No heartbeat from core client for 30 sec - exiting 18:27:08 (2784): No heartbeat from core client for 30 sec - exiting 18:27:09 (2784): No heartbeat from core client for 30 sec - exiting 18:27:11 (2784): No heartbeat from core client for 30 sec - exiting 18:27:12 (2784): No heartbeat from core client for 30 sec - exiting 18:27:13 (2784): No heartbeat from core client for 30 sec - exiting 18:27:14 (2784): No heartbeat from core client for 30 sec - exiting 18:27:15 (2784): No heartbeat from core client for 30 sec - exiting 18:27:16 (2784): No heartbeat from core client for 30 sec - exiting 18:27:17 (2784): No heartbeat from core client for 30 sec - exiting 18:27:18 (2784): No heartbeat from core client for 30 sec - exiting 18:27:19 (2784): No heartbeat from core client for 30 sec - exiting 18:27:21 (2784): No heartbeat from core client for 30 sec - exiting 18:27:22 (2784): No heartbeat from core client for 30 sec - exiting 18:27:23 (2784): No heartbeat from core client for 30 sec - exiting 18:27:24 (2784): No heartbeat from core client for 30 sec - exiting 18:27:25 (2784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:38:08 (1344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:38:09 (1344): No heartbeat from core client for 30 sec - exiting 18:38:11 (1344): No heartbeat from core client for 30 sec - exiting 18:38:12 (1344): No heartbeat from core client for 30 sec - exiting 18:38:13 (1344): No heartbeat from core client for 30 sec - exiting 18:38:14 (1344): No heartbeat from core client for 30 sec - exiting 18:44:27 (2604): No heartbeat from core client for 30 sec - exiting 18:44:28 (2604): No heartbeat from core client for 30 sec - exiting 18:44:29 (2604): No heartbeat from core client for 30 sec - exiting 18:44:30 (2604): No heartbeat from core client for 30 sec - exiting 18:44:31 (2604): No heartbeat from core client for 30 sec - exiting 18:44:32 (2604): No heartbeat from core client for 30 sec - exiting 18:44:33 (2604): No heartbeat from core client for 30 sec - exiting 18:44:34 (2604): No heartbeat from core client for 30 sec - exiting 18:44:35 (2604): No heartbeat from core client for 30 sec - exiting 18:44:36 (2604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:47:45 (4252): No heartbeat from core client for 30 sec - exiting 18:51:58 (4252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:56:55 (1152): No heartbeat from core client for 30 sec - exiting 18:56:56 (1152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=660, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5984, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4976, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5096, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5036, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5276, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4320, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/zg8sko.pj84c10 Error converting file to netcdf: dataout/zg8sko.pi84c10 Error converting file to netcdf: dataout/zg8sko.pf84c10 Error converting file to netcdf: dataout/zg8ska.ph84c10 Error converting file to netcdf: dataout/zg8ska.pg84c10 Error converting file to netcdf: dataout/zg8ska.pe84c10 Error converting file to netcdf: dataout/zg8ska.pd84c10 CPDN Monitor - Quit request from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5120, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4864, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4528, iMonCtr=1 Model crash detected, will try to restart... Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4224, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5196, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5088, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4348, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5704, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5324, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5324, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5492, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5340, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5136, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:03:05 (4732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:03:06 (4732): No heartbeat from core client for 30 sec - exiting 18:03:07 (4732): No heartbeat from core client for 30 sec - exiting 18:03:08 (4732): No heartbeat from core client for 30 sec - exiting 18:04:43 (3512): No heartbeat from core client for 30 sec - exiting 18:04:44 (3512): No heartbeat from core client for 30 sec - exiting 18:04:45 (3512): No heartbeat from core client for 30 sec - exiting 18:04:46 (3512): No heartbeat from core client for 30 sec - exiting 18:04:47 (3512): No heartbeat from core client for 30 sec - exiting 18:04:48 (3512): No heartbeat from core client for 30 sec - exiting 18:04:49 (3512): No heartbeat from core client for 30 sec - exiting 18:04:50 (3512): No heartbeat from core client for 30 sec - exiting 18:04:51 (3512): No heartbeat from core client for 30 sec - exiting 18:04:52 (3512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:06:26 (3136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:07:07 (3140): No heartbeat from core client for 30 sec - exiting 18:07:08 (3140): No heartbeat from core client for 30 sec - exiting 18:07:09 (3140): No heartbeat from core client for 30 sec - exiting 18:07:10 (3140): No heartbeat from core client for 30 sec - exiting 18:07:11 (3140): No heartbeat from core client for 30 sec - exiting 18:07:12 (3140): No heartbeat from core client for 30 sec - exiting 18:07:13 (3140): No heartbeat from core client for 30 sec - exiting 18:07:14 (3140): No heartbeat from core client for 30 sec - exiting 18:07:15 (3140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:09:34 (1300): No heartbeat from core client for 30 sec - exiting 18:09:35 (1300): No heartbeat from core client for 30 sec - exiting 18:09:36 (1300): No heartbeat from core client for 30 sec - exiting 18:09:37 (1300): No heartbeat from core client for 30 sec - exiting 18:09:38 (1300): No heartbeat from core client for 30 sec - exiting 18:09:39 (1300): No heartbeat from core client for 30 sec - exiting 18:09:40 (1300): No heartbeat from core client for 30 sec - exiting 18:09:41 (1300): No heartbeat from core client for 30 sec - exiting 18:09:42 (1300): No heartbeat from core client for 30 sec - exiting 18:09:43 (1300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:13:34 (1088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:13:35 (1088): No heartbeat from core client for 30 sec - exiting 18:13:36 (1088): No heartbeat from core client for 30 sec - exiting 18:15:05 (3088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:17:05 (1068): No heartbeat from core client for 30 sec - exiting 18:17:06 (1068): No heartbeat from core client for 30 sec - exiting 18:17:07 (1068): No heartbeat from core client for 30 sec - exiting 18:17:08 (1068): No heartbeat from core client for 30 sec - exiting 18:17:09 (1068): No heartbeat from core client for 30 sec - exiting 18:17:10 (1068): No heartbeat from core client for 30 sec - exiting 18:17:11 (1068): No heartbeat from core client for 30 sec - exiting 18:17:12 (1068): No heartbeat from core client for 30 sec - exiting 18:17:13 (1068): No heartbeat from core client for 30 sec - exiting 18:17:14 (1068): No heartbeat from core client for 30 sec - exiting 18:17:15 (1068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:20:07 (2564): No heartbeat from core client for 30 sec - exiting 18:20:08 (2564): No heartbeat from core client for 30 sec - exiting 18:20:14 (2564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:20:15 (2564): No heartbeat from core client for 30 sec - exiting 18:20:16 (2564): No heartbeat from core client for 30 sec - exiting 18:20:17 (2564): No heartbeat from core client for 30 sec - exiting 18:20:18 (2564): No heartbeat from core client for 30 sec - exiting 18:21:22 (4288): No heartbeat from core client for 30 sec - exiting 18:21:23 (4288): No heartbeat from core client for 30 sec - exiting 18:21:24 (4288): No heartbeat from core client for 30 sec - exiting 18:21:25 (4288): No heartbeat from core client for 30 sec - exiting 18:21:26 (4288): No heartbeat from core client for 30 sec - exiting 18:21:27 (4288): No heartbeat from core client for 30 sec - exiting 18:21:28 (4288): No heartbeat from core client for 30 sec - exiting 18:21:29 (4288): No heartbeat from core client for 30 sec - exiting 18:21:30 (4288): No heartbeat from core client for 30 sec - exiting 18:21:31 (4288): No heartbeat from core client for 30 sec - exiting 18:21:32 (4288): No heartbeat from core client for 30 sec - exiting 18:21:33 (4288): No heartbeat from core client for 30 sec - exiting 18:21:34 (4288): No heartbeat from core client for 30 sec - exiting 18:21:35 (4288): No heartbeat from core client for 30 sec - exiting 18:21:36 (4288): No heartbeat from core client for 30 sec - exiting 18:21:37 (4288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:23:09 (3940): No heartbeat from core client for 30 sec - exiting 18:23:18 (3940): No heartbeat from core client for 30 sec - exiting 18:23:19 (3940): No heartbeat from core client for 30 sec - exiting 18:23:20 (3940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:26:30 (2144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:26:31 (2144): No heartbeat from core client for 30 sec - exiting 18:26:32 (2144): No heartbeat from core client for 30 sec - exiting 18:28:51 (2140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:28:52 (2140): No heartbeat from core client for 30 sec - exiting 18:31:32 (5784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:34:49 (5132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:34:50 (5132): No heartbeat from core client for 30 sec - exiting 18:34:51 (5132): No heartbeat from core client for 30 sec - exiting 18:34:52 (5132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5064, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4244, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5320, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5080, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:53:20 (5376): No heartbeat from core client for 30 sec - exiting 10:53:22 (5376): No heartbeat from core client for 30 sec - exiting 10:53:23 (5376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:53:24 (5376): No heartbeat from core client for 30 sec - exiting 10:53:25 (5376): No heartbeat from core client for 30 sec - exiting 10:53:26 (5376): No heartbeat from core client for 30 sec - exiting 10:53:27 (5376): No heartbeat from core client for 30 sec - exiting 10:53:28 (5376): No heartbeat from core client for 30 sec - exiting 11:04:25 (4016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:04:26 (4016): No heartbeat from core client for 30 sec - exiting 11:04:27 (4016): No heartbeat from core client for 30 sec - exiting 11:04:28 (4016): No heartbeat from core client for 30 sec - exiting 11:06:11 (5932): No heartbeat from core client for 30 sec - exiting 11:06:12 (5932): No heartbeat from core client for 30 sec - exiting 11:06:13 (5932): No heartbeat from core client for 30 sec - exiting 11:06:14 (5932): No heartbeat from core client for 30 sec - exiting 11:06:15 (5932): No heartbeat from core client for 30 sec - exiting 11:06:16 (5932): No heartbeat from core client for 30 sec - exiting 11:06:17 (5932): No heartbeat from core client for 30 sec - exiting 11:06:18 (5932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:18:36 (4460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:18:37 (4460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:06:26 (1512): No heartbeat from core client for 30 sec - exiting 20:06:27 (1512): No heartbeat from core client for 30 sec - exiting 20:06:28 (1512): No heartbeat from core client for 30 sec - exiting 20:06:29 (1512): No heartbeat from core client for 30 sec - exiting 20:06:30 (1512): No heartbeat from core client for 30 sec - exiting 20:06:31 (1512): No heartbeat from core client for 30 sec - exiting 20:06:32 (1512): No heartbeat from core client for 30 sec - exiting 20:06:33 (1512): No heartbeat from core client for 30 sec - exiting 20:06:34 (1512): No heartbeat from core client for 30 sec - exiting 20:06:35 (1512): No heartbeat from core client for 30 sec - exiting 20:06:36 (1512): No heartbeat from core client for 30 sec - exiting 20:06:37 (1512): No heartbeat from core client for 30 sec - exiting 20:06:38 (1512): No heartbeat from core client for 30 sec - exiting 20:06:39 (1512): No heartbeat from core client for 30 sec - exiting 20:06:40 (1512): No heartbeat from core client for 30 sec - exiting 20:06:41 (1512): No heartbeat from core client for 30 sec - exiting 20:06:42 (1512): No heartbeat from core client for 30 sec - exiting 20:06:43 (1512): No heartbeat from core client for 30 sec - exiting 20:06:44 (1512): No heartbeat from core client for 30 sec - exiting 20:06:45 (1512): No heartbeat from core client for 30 sec - exiting 20:06:46 (1512): No heartbeat from core client for 30 sec - exiting 20:06:47 (1512): No heartbeat from core client for 30 sec - exiting 20:06:48 (1512): No heartbeat from core client for 30 sec - exiting 20:06:54 (1512): No heartbeat from core client for 30 sec - exiting 20:06:55 (1512): No heartbeat from core client for 30 sec - exiting 20:06:56 (1512): No heartbeat from core client for 30 sec - exiting 20:06:57 (1512): No heartbeat from core client for 30 sec - exiting 20:06:58 (1512): No heartbeat from core client for 30 sec - exiting 20:06:59 (1512): No heartbeat from core client for 30 sec - exiting 20:07:00 (1512): No heartbeat from core client for 30 sec - exiting 20:07:01 (1512): No heartbeat from core client for 30 sec - exiting 20:07:02 (1512): No heartbeat from core client for 30 sec - exiting 20:07:03 (1512): No heartbeat from core client for 30 sec - exiting 20:07:04 (1512): No heartbeat from core client for 30 sec - exiting 20:07:05 (1512): No heartbeat from core client for 30 sec - exiting 20:07:07 (1512): No heartbeat from core client for 30 sec - exiting 20:07:08 (1512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4032, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=984, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4180, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76EBEB9D read attempt to address 0x40990A35 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_zg8s_1880_40_008249864/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Feb 2013 16:50:53	1133571	15450330	hadcm3n_zg8s_1880_40_008249864_1	259,200	820,326	3.1648
10 Feb 2013 00:45:02	1133571	15450330	hadcm3n_zg8s_1880_40_008249864_1	233,280	740,745	3.1753
31 Jan 2013 02:14:33	1133571	15450330	hadcm3n_zg8s_1880_40_008249864_1	207,360	662,297	3.1939
19 Jan 2013 21:42:02	1133571	15450330	hadcm3n_zg8s_1880_40_008249864_1	181,440	582,050	3.2079
16 Jan 2013 03:46:27	1133571	15450330	hadcm3n_zg8s_1880_40_008249864_1	155,520	502,714	3.2325
09 Jan 2013 01:18:27	1133571	15450330	hadcm3n_zg8s_1880_40_008249864_1	129,600	420,343	3.2434
29 Dec 2012 17:02:40	1133571	15450330	hadcm3n_zg8s_1880_40_008249864_1	103,680	333,571	3.2173
17 Dec 2012 22:49:53	1133571	15450330	hadcm3n_zg8s_1880_40_008249864_1	77,760	241,697	3.1082
14 Dec 2012 18:22:46	1133571	15450330	hadcm3n_zg8s_1880_40_008249864_1	51,840	162,597	3.1365
04 Dec 2012 23:22:24	1133571	15450330	hadcm3n_zg8s_1880_40_008249864_1	25,920	83,794	3.2328