Task 13560589

Name	hadcm3n_y9si_1900_40_007527406_1
Workunit	7724881
Created	28 Oct 2011, 13:50:43 UTC
Sent	28 Oct 2011, 22:27:53 UTC
Report deadline	28 Jan 2012, 5:55:04 UTC
Received	21 Jan 2012, 0:09:04 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1160009
Run time	15 days 13 hours 37 min 32 sec
CPU time	14 days 5 hours 28 min 19 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.46 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4136, iMonCtr=1 Model crash detected, will try to restart... 17:48:09 (4712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4648, iMonCtr=1 Model crash detected, will try to restart... 18:02:47 (5480): No heartbeat from core client for 30 sec - exiting 18:02:48 (5480): No heartbeat from core client for 30 sec - exiting 18:02:50 (5480): No heartbeat from core client for 30 sec - exiting 18:02:51 (5480): No heartbeat from core client for 30 sec - exiting 18:02:52 (5480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:15:09 (5400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:56:38 (4496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4576, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2416, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5732, iMonCtr=1 Model crash detected, will try to restart... 18:56:24 (1760): No heartbeat from core client for 30 sec - exiting 18:56:25 (1760): No heartbeat from core client for 30 sec - exiting 18:56:26 (1760): No heartbeat from core client for 30 sec - exiting 18:56:27 (1760): No heartbeat from core client for 30 sec - exiting 18:56:28 (1760): No heartbeat from core client for 30 sec - exiting 18:56:29 (1760): No heartbeat from core client for 30 sec - exiting 18:56:30 (1760): No heartbeat from core client for 30 sec - exiting 18:56:32 (1760): No heartbeat from core client for 30 sec - exiting 18:56:33 (1760): No heartbeat from core client for 30 sec - exiting 18:56:34 (1760): No heartbeat from core client for 30 sec - exiting 18:56:35 (1760): No heartbeat from core client for 30 sec - exiting 18:56:36 (1760): No heartbeat from core client for 30 sec - exiting 18:56:37 (1760): No heartbeat from core client for 30 sec - exiting 18:56:38 (1760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4028, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4028, iMonCtr=1 Model crash detected, will try to restart... 19:45:50 (5040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6916, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6916, iMonCtr=1 Model crash detected, will try to restart... 08:17:35 (3264): No heartbeat from core client for 30 sec - exiting 08:17:36 (3264): No heartbeat from core client for 30 sec - exiting 08:17:37 (3264): No heartbeat from core client for 30 sec - exiting 08:17:38 (3264): No heartbeat from core client for 30 sec - exiting 08:17:40 (3264): No heartbeat from core client for 30 sec - exiting 08:17:41 (3264): No heartbeat from core client for 30 sec - exiting 08:17:42 (3264): No heartbeat from core client for 30 sec - exiting 08:17:43 (3264): No heartbeat from core client for 30 sec - exiting 08:17:44 (3264): No heartbeat from core client for 30 sec - exiting 08:17:45 (3264): No heartbeat from core client for 30 sec - exiting 08:17:46 (3264): No heartbeat from core client for 30 sec - exiting 08:17:47 (3264): No heartbeat from core client for 30 sec - exiting 08:17:48 (3264): No heartbeat from core client for 30 sec - exiting 08:17:49 (3264): No heartbeat from core client for 30 sec - exiting 08:17:51 (3264): No heartbeat from core client for 30 sec - exiting 08:17:52 (3264): No heartbeat from core client for 30 sec - exiting 08:17:53 (3264): No heartbeat from core client for 30 sec - exiting 08:17:54 (3264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4528, iMonCtr=1 Model crash detected, will try to restart... 15:57:48 (5052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:49:51 (1396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5840, iMonCtr=1 Model crash detected, will try to restart... 17:18:32 (4872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5724, iMonCtr=1 Model crash detected, will try to restart... 19:27:55 (5076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:17:05 (5032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:25:47 (4856): No heartbeat from core client for 30 sec - exiting 19:25:48 (4856): No heartbeat from core client for 30 sec - exiting 19:25:49 (4856): No heartbeat from core client for 30 sec - exiting 19:25:51 (4856): No heartbeat from core client for 30 sec - exiting 19:25:52 (4856): No heartbeat from core client for 30 sec - exiting 19:25:53 (4856): No heartbeat from core client for 30 sec - exiting 19:25:54 (4856): No heartbeat from core client for 30 sec - exiting 19:25:55 (4856): No heartbeat from core client for 30 sec - exiting 19:25:56 (4856): No heartbeat from core client for 30 sec - exiting 19:25:57 (4856): No heartbeat from core client for 30 sec - exiting 19:25:58 (4856): No heartbeat from core client for 30 sec - exiting 19:25:59 (4856): No heartbeat from core client for 30 sec - exiting 19:26:00 (4856): No heartbeat from core client for 30 sec - exiting 19:26:01 (4856): No heartbeat from core client for 30 sec - exiting 19:26:03 (4856): No heartbeat from core client for 30 sec - exiting 19:26:04 (4856): No heartbeat from core client for 30 sec - exiting 19:26:05 (4856): No heartbeat from core client for 30 sec - exiting 19:26:06 (4856): No heartbeat from core client for 30 sec - exiting 19:26:07 (4856): No heartbeat from core client for 30 sec - exiting 19:26:08 (4856): No heartbeat from core client for 30 sec - exiting 19:26:09 (4856): No heartbeat from core client for 30 sec - exiting 19:26:10 (4856): No heartbeat from core client for 30 sec - exiting 19:26:11 (4856): No heartbeat from core client for 30 sec - exiting 19:26:12 (4856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:42:29 (5484): No heartbeat from core client for 30 sec - exiting 19:42:30 (5484): No heartbeat from core client for 30 sec - exiting 19:42:31 (5484): No heartbeat from core client for 30 sec - exiting 19:42:32 (5484): No heartbeat from core client for 30 sec - exiting 19:42:34 (5484): No heartbeat from core client for 30 sec - exiting 19:42:35 (5484): No heartbeat from core client for 30 sec - exiting 19:42:36 (5484): No heartbeat from core client for 30 sec - exiting 19:42:37 (5484): No heartbeat from core client for 30 sec - exiting 19:42:38 (5484): No heartbeat from core client for 30 sec - exiting 19:42:39 (5484): No heartbeat from core client for 30 sec - exiting 19:42:40 (5484): No heartbeat from core client for 30 sec - exiting 19:42:41 (5484): No heartbeat from core client for 30 sec - exiting 19:42:42 (5484): No heartbeat from core client for 30 sec - exiting 19:42:43 (5484): No heartbeat from core client for 30 sec - exiting 19:42:44 (5484): No heartbeat from core client for 30 sec - exiting 19:42:46 (5484): No heartbeat from core client for 30 sec - exiting 19:42:47 (5484): No heartbeat from core client for 30 sec - exiting 19:42:48 (5484): No heartbeat from core client for 30 sec - exiting 19:42:49 (5484): No heartbeat from core client for 30 sec - exiting 19:42:50 (5484): No heartbeat from core client for 30 sec - exiting 19:42:51 (5484): No heartbeat from core client for 30 sec - exiting 19:42:52 (5484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4268, iMonCtr=1 Model crash detected, will try to restart... 19:31:33 (5252): No heartbeat from core client for 30 sec - exiting 19:31:35 (5252): No heartbeat from core client for 30 sec - exiting 19:31:36 (5252): No heartbeat from core client for 30 sec - exiting 19:31:37 (5252): No heartbeat from core client for 30 sec - exiting 19:31:38 (5252): No heartbeat from core client for 30 sec - exiting 19:31:39 (5252): No heartbeat from core client for 30 sec - exiting 19:31:40 (5252): No heartbeat from core client for 30 sec - exiting 19:31:41 (5252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:03:43 (5136): No heartbeat from core client for 30 sec - exiting 08:03:44 (5136): No heartbeat from core client for 30 sec - exiting 08:03:45 (5136): No heartbeat from core client for 30 sec - exiting 08:03:46 (5136): No heartbeat from core client for 30 sec - exiting 08:03:47 (5136): No heartbeat from core client for 30 sec - exiting 08:03:48 (5136): No heartbeat from core client for 30 sec - exiting 08:03:49 (5136): No heartbeat from core client for 30 sec - exiting 08:03:51 (5136): No heartbeat from core client for 30 sec - exiting 08:03:52 (5136): No heartbeat from core client for 30 sec - exiting 08:03:53 (5136): No heartbeat from core client for 30 sec - exiting 08:03:54 (5136): No heartbeat from core client for 30 sec - exiting 08:03:55 (5136): No heartbeat from core client for 30 sec - exiting 08:03:56 (5136): No heartbeat from core client for 30 sec - exiting 08:03:57 (5136): No heartbeat from core client for 30 sec - exiting 08:03:58 (5136): No heartbeat from core client for 30 sec - exiting 08:03:59 (5136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:38:05 (4480): No heartbeat from core client for 30 sec - exiting 17:38:06 (4480): No heartbeat from core client for 30 sec - exiting 17:38:07 (4480): No heartbeat from core client for 30 sec - exiting 17:38:09 (4480): No heartbeat from core client for 30 sec - exiting 17:38:10 (4480): No heartbeat from core client for 30 sec - exiting 17:38:11 (4480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:38:17 (5348): Can't acquire lockfile (32) - waiting 35s Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5620, iMonCtr=1 Model crash detected, will try to restart... 07:44:12 (4180): No heartbeat from core client for 30 sec - exiting 07:44:14 (4180): No heartbeat from core client for 30 sec - exiting 07:44:15 (4180): No heartbeat from core client for 30 sec - exiting 07:44:16 (4180): No heartbeat from core client for 30 sec - exiting 07:44:17 (4180): No heartbeat from core client for 30 sec - exiting 07:44:18 (4180): No heartbeat from core client for 30 sec - exiting 07:44:19 (4180): No heartbeat from core client for 30 sec - exiting 07:44:20 (4180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:13:51 (4148): No heartbeat from core client for 30 sec - exiting 10:13:53 (4148): No heartbeat from core client for 30 sec - exiting 10:13:54 (4148): No heartbeat from core client for 30 sec - exiting 10:13:55 (4148): No heartbeat from core client for 30 sec - exiting 10:13:56 (4148): No heartbeat from core client for 30 sec - exiting 10:13:57 (4148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4904, iMonCtr=1 Model crash detected, will try to restart... 21:13:25 (4148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:39:42 (5248): No heartbeat from core client for 30 sec - exiting 18:39:43 (5248): No heartbeat from core client for 30 sec - exiting 18:39:44 (5248): No heartbeat from core client for 30 sec - exiting 18:39:45 (5248): No heartbeat from core client for 30 sec - exiting 18:39:46 (5248): No heartbeat from core client for 30 sec - exiting 18:39:47 (5248): No heartbeat from core client for 30 sec - exiting 18:39:48 (5248): No heartbeat from core client for 30 sec - exiting 18:39:49 (5248): No heartbeat from core client for 30 sec - exiting 18:39:51 (5248): No heartbeat from core client for 30 sec - exiting 18:39:52 (5248): No heartbeat from core client for 30 sec - exiting 18:39:53 (5248): No heartbeat from core client for 30 sec - exiting 18:39:54 (5248): No heartbeat from core client for 30 sec - exiting 18:39:55 (5248): No heartbeat from core client for 30 sec - exiting 18:39:56 (5248): No heartbeat from core client for 30 sec - exiting 18:39:57 (5248): No heartbeat from core client for 30 sec - exiting 18:39:58 (5248): No heartbeat from core client for 30 sec - exiting 18:39:59 (5248): No heartbeat from core client for 30 sec - exiting 18:40:00 (5248): No heartbeat from core client for 30 sec - exiting 18:40:02 (5248): No heartbeat from core client for 30 sec - exiting 18:40:03 (5248): No heartbeat from core client for 30 sec - exiting 18:40:04 (5248): No heartbeat from core client for 30 sec - exiting 18:40:05 (5248): No heartbeat from core client for 30 sec - exiting 18:40:06 (5248): No heartbeat from core client for 30 sec - exiting 18:40:07 (5248): No heartbeat from core client for 30 sec - exiting 18:40:08 (5248): No heartbeat from core client for 30 sec - exiting 18:40:09 (5248): No heartbeat from core client for 30 sec - exiting 18:40:10 (5248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:41:27 (6072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:40:45 (5204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5540, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5540, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4948, iMonCtr=1 Model crash detected, will try to restart... 16:11:34 (4988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3728, iMonCtr=1 Model crash detected, will try to restart... 10:45:41 (4952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5816, iMonCtr=1 Model crash detected, will try to restart... 19:31:59 (5136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4392, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5696, iMonCtr=1 Model crash detected, will try to restart... 20:49:08 (1776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4656, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1736, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5948, iMonCtr=1 Model crash detected, will try to restart... 11:41:02 (2216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:34:35 (1804): No heartbeat from core client for 30 sec - exiting 17:34:36 (1804): No heartbeat from core client for 30 sec - exiting 17:34:37 (1804): No heartbeat from core client for 30 sec - exiting 17:34:38 (1804): No heartbeat from core client for 30 sec - exiting 17:34:40 (1804): No heartbeat from core client for 30 sec - exiting 17:34:41 (1804): No heartbeat from core client for 30 sec - exiting 17:34:42 (1804): No heartbeat from core client for 30 sec - exiting 17:34:43 (1804): No heartbeat from core client for 30 sec - exiting 17:34:44 (1804): No heartbeat from core client for 30 sec - exiting 17:34:45 (1804): No heartbeat from core client for 30 sec - exiting 17:34:46 (1804): No heartbeat from core client for 30 sec - exiting 17:34:47 (1804): No heartbeat from core client for 30 sec - exiting 17:34:48 (1804): No heartbeat from core client for 30 sec - exiting 17:34:49 (1804): No heartbeat from core client for 30 sec - exiting 17:34:50 (1804): No heartbeat from core client for 30 sec - exiting 17:34:52 (1804): No heartbeat from core client for 30 sec - exiting 17:34:53 (1804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... zip error: Could not create output file (was replacing the original zip file) Suspended CPDN Monitor - Suspend request from BOINC... 18:54:55 (5272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:25:47 (5060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3488, iMonCtr=1 Model crash detected, will try to restart... 17:59:26 (5172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7224, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:47:25 (1780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2064, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4620, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3880, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3904, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2976, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6736, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6360, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6360, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6360, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6896, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4548, iMonCtr=1 Model crash detected, will try to restart... 09:34:09 (5592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1428, iMonCtr=1 Model crash detected, will try to restart... 09:31:45 (5724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:08:19 (5628): No heartbeat from core client for 30 sec - exiting 20:08:20 (5628): No heartbeat from core client for 30 sec - exiting 20:08:22 (5628): No heartbeat from core client for 30 sec - exiting 20:08:23 (5628): No heartbeat from core client for 30 sec - exiting 20:08:24 (5628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:14:36 (6128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6504, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6684, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6684, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6684, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6684, iMonCtr=1 Model crash detected, will try to restart... 12:47:43 (5200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x770F7373 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... 12:14:34 (5456): No heartbeat from core client for 30 sec - exiting 12:14:35 (5456): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76F771A8 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_y9si_1900_40_007527406/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish 12:14:36 (5456): No heartbeat from core client for 30 sec - exiting 12:14:37 (5456): No heartbeat from core client for 30 sec - exiting </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Jan 2012 06:39:58	1160009	13560589	hadcm3n_y9si_1900_40_007527406_1	518,400	1,220,319	2.3540
13 Jan 2012 20:42:29	1160009	13560589	hadcm3n_y9si_1900_40_007527406_1	492,480	1,161,738	2.3590
08 Jan 2012 01:38:26	1160009	13560589	hadcm3n_y9si_1900_40_007527406_1	466,560	1,095,281	2.3476
07 Jan 2012 07:16:55	1160009	13560589	hadcm3n_y9si_1900_40_007527406_1	440,640	1,034,104	2.3468
06 Jan 2012 00:36:06	1160009	13560589	hadcm3n_y9si_1900_40_007527406_1	414,720	969,943	2.3388
04 Jan 2012 16:58:12	1160009	13560589	hadcm3n_y9si_1900_40_007527406_1	388,800	905,974	2.3302
29 Dec 2011 01:16:45	1160009	13560589	hadcm3n_y9si_1900_40_007527406_1	362,880	846,518	2.3328
21 Dec 2011 01:56:08	1160009	13560589	hadcm3n_y9si_1900_40_007527406_1	336,960	789,465	2.3429
17 Dec 2011 01:32:04	1160009	13560589	hadcm3n_y9si_1900_40_007527406_1	311,040	727,326	2.3384
14 Dec 2011 18:44:39	1160009	13560589	hadcm3n_y9si_1900_40_007527406_1	285,120	660,551	2.3167
12 Dec 2011 04:37:20	1160009	13560589	hadcm3n_y9si_1900_40_007527406_1	259,200	598,132	2.3076
08 Dec 2011 06:12:25	1160009	13560589	hadcm3n_y9si_1900_40_007527406_1	233,280	538,382	2.3079
06 Dec 2011 04:53:37	1160009	13560589	hadcm3n_y9si_1900_40_007527406_1	207,360	479,285	2.3114
01 Dec 2011 08:55:51	1160009	13560589	hadcm3n_y9si_1900_40_007527406_1	181,440	416,997	2.2983
29 Nov 2011 00:28:17	1160009	13560589	hadcm3n_y9si_1900_40_007527406_1	155,520	360,137	2.3157
22 Nov 2011 08:38:47	1160009	13560589	hadcm3n_y9si_1900_40_007527406_1	129,600	304,882	2.3525
15 Nov 2011 19:06:26	1160009	13560589	hadcm3n_y9si_1900_40_007527406_1	103,680	242,978	2.3435
08 Nov 2011 05:45:51	1160009	13560589	hadcm3n_y9si_1900_40_007527406_1	77,760	178,972	2.3016
05 Nov 2011 00:49:17	1160009	13560589	hadcm3n_y9si_1900_40_007527406_1	51,840	121,511	2.3440
31 Oct 2011 19:00:45	1160009	13560589	hadcm3n_y9si_1900_40_007527406_1	25,920	58,837	2.2699