Task 12618321

Name	hadam3p_pnw_2z3d_1979_1_007178977_0
Workunit	7377259
Created	22 Feb 2011, 11:57:56 UTC
Sent	8 Mar 2011, 16:22:09 UTC
Report deadline	18 Feb 2012, 21:42:09 UTC
Received	25 Mar 2011, 10:53:12 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	729212
Run time	7 days 16 hours 34 min 27 sec
CPU time	3 days 2 hours 48 min 17 sec
Validate state	Invalid
Credit	1,003.37
Device peak FLOPS	1.50 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Pacific North West v6.08 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7652, selfPID=7652, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5524, selfPID=5524, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11220, selfPID=11220, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9720, selfPID=9720, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=10544, selfPID=0, iMonCtr=1 No Process Handle Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=10544, selfPID=10544, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=17732, selfPID=17732, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6928, selfPID=6928, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=19744, selfPID=19744, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7464, selfPID=7464, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8852, selfPID=8852, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12464, selfPID=12464, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=14024, selfPID=14024, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6248, selfPID=6248, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9820, selfPID=9820, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... RCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12728, selfPID=12728, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:16:20 (7532): No heartbeat from core client for 30 sec - exiting 08:16:21 (7532): No heartbeat from core client for 30 sec - exiting 08:16:22 (7532): No heartbeat from core client for 30 sec - exiting 08:16:23 (7532): No heartbeat from core client for 30 sec - exiting 08:16:24 (7532): No heartbeat from core client for 30 sec - exiting 08:16:25 (7532): No heartbeat from core client for 30 sec - exiting 08:16:26 (7532): No heartbeat from core client for 30 sec - exiting 08:16:27 (7532): No heartbeat from core client for 30 sec - exiting 08:16:28 (7532): No heartbeat from core client for 30 sec - exiting 08:16:30 (7532): No heartbeat from core client for 30 sec - exiting 08:16:31 (7532): No heartbeat from core client for 30 sec - exiting 08:16:32 (7532): No heartbeat from core client for 30 sec - exiting 08:16:33 (7532): No heartbeat from core client for 30 sec - exiting 08:16:34 (7532): No heartbeat from core client for 30 sec - exiting 08:16:35 (7532): No heartbeat from core client for 30 sec - exiting 08:16:36 (7532): No heartbeat from core client for 30 sec - exiting 08:16:37 (7532): No heartbeat from core client for 30 sec - exiting 08:16:38 (7532): No heartbeat from core client for 30 sec - exiting 08:16:39 (7532): No heartbeat from core client for 30 sec - exiting 08:16:40 (7532): No heartbeat from core client for 30 sec - exiting 08:16:41 (7532): No heartbeat from core client for 30 sec - exiting 08:16:42 (7532): No heartbeat from core client for 30 sec - exiting 08:16:43 (7532): No heartbeat from core client for 30 sec - exiting 08:16:44 (7532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=15164, selfPID=15164, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=15640, selfPID=15640, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=15732, selfPID=15732, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=14996, selfPID=14996, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=10544, selfPID=10544, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3416, selfPID=3416, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7160, selfPID=7160, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=16804, selfPID=16804, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9640, selfPID=9640, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3708, selfPID=3708, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=10796, selfPID=10796, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=13440, selfPID=13440, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8564, selfPID=8564, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=17512, selfPID=17512, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9484, selfPID=9484, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=10196, selfPID=10196, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=18296, selfPID=18296, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=16604, selfPID=16604, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11856, selfPID=11856, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=20004, selfPID=20004, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process iCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5080, selfPID=5080, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11672, selfPID=11672, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=13588, selfPID=13588, iMonCtr=2 CPDN Monitor - Quit request from BOINC... 14:58:41 (3476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=20256, selfPID=20256, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=20764, selfPID=20764, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=13836, selfPID=13836, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:55:56 (12820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=19404, selfPID=19404, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=17228, selfPID=17228, iMonCtr=2 CPDN Monitor - Quit request from BOINC... RCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=20456, selfPID=20456, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=18768, selfPID=18768, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12292, selfPID=12292, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=17584, selfPID=17584, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=17660, selfPID=17660, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=20144, selfPID=15000, iMonCtr=1 00:23:38 (18264): No heartbeat from core client for 30 sec - exiting 00:23:39 (18264): No heartbeat from core client for 30 sec - exiting 00:23:40 (18264): No heartbeat from core client for 30 sec - exiting 00:23:41 (18264): No heartbeat from core client for 30 sec - exiting 00:23:42 (18264): No heartbeat from core client for 30 sec - exiting 00:23:43 (18264): No heartbeat from core client for 30 sec - exiting 00:23:44 (18264): No heartbeat from core client for 30 sec - exiting 00:23:45 (18264): No heartbeat from core client for 30 sec - exiting 00:23:46 (18264): No heartbeat from core client for 30 sec - exiting 00:23:47 (18264): No heartbeat from core client for 30 sec - exiting 00:23:48 (18264): No heartbeat from core client for 30 sec - exiting 00:23:49 (18264): No heartbeat from core client for 30 sec - exiting 00:23:50 (18264): No heartbeat from core client for 30 sec - exiting 00:23:51 (18264): No heartbeat from core client for 30 sec - exiting 00:23:52 (18264): No heartbeat from core client for 30 sec - exiting 00:23:53 (18264): No heartbeat from core client for 30 sec - exiting 00:23:54 (18264): No heartbeat from core client for 30 sec - exiting 00:23:55 (18264): No heartbeat from core client for 30 sec - exiting 00:23:56 (18264): No heartbeat from core client for 30 sec - exiting 00:23:57 (18264): No heartbeat from core client for 30 sec - exiting 00:23:58 (18264): No heartbeat from core client for 30 sec - exiting 00:23:59 (18264): No heartbeat from core client for 30 sec - exiting 00:24:00 (18264): No heartbeat from core client for 30 sec - exiting 00:24:01 (18264): No heartbeat from core client for 30 sec - exiting 00:24:02 (18264): No heartbeat from core client for 30 sec - exiting 00:24:03 (18264): No heartbeat from core client for 30 sec - exiting 00:24:04 (18264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=22644, selfPID=22644, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:22:30 (23000): No heartbeat from core client for 30 sec - exiting 02:22:31 (23000): No heartbeat from core client for 30 sec - exiting 02:22:32 (23000): No heartbeat from core client for 30 sec - exiting 02:22:33 (23000): No heartbeat from core client for 30 sec - exiting 02:22:34 (23000): No heartbeat from core client for 30 sec - exiting 02:22:35 (23000): No heartbeat from core client for 30 sec - exiting 02:22:36 (23000): No heartbeat from core client for 30 sec - exiting 02:22:37 (23000): No heartbeat from core client for 30 sec - exiting 02:22:38 (23000): No heartbeat from core client for 30 sec - exiting 02:22:39 (23000): No heartbeat from core client for 30 sec - exiting 02:22:40 (23000): No heartbeat from core client for 30 sec - exiting 02:22:41 (23000): No heartbeat from core client for 30 sec - exiting 02:22:42 (23000): No heartbeat from core client for 30 sec - exiting 02:22:44 (23000): No heartbeat from core client for 30 sec - exiting 05:22:40 (5648): No heartbeat from core client for 30 sec - exiting 05:22:41 (5648): No heartbeat from core client for 30 sec - exiting 05:22:42 (5648): No heartbeat from core client for 30 sec - exiting 05:22:43 (5648): No heartbeat from core client for 30 sec - exiting 05:22:44 (5648): No heartbeat from core client for 30 sec - exiting 05:22:45 (5648): No heartbeat from core client for 30 sec - exiting 05:22:46 (5648): No heartbeat from core client for 30 sec - exiting 05:22:47 (5648): No heartbeat from core client for 30 sec - exiting 05:22:48 (5648): No heartbeat from core client for 30 sec - exiting 05:22:49 (5648): No heartbeat from core client for 30 sec - exiting 05:22:50 (5648): No heartbeat from core client for 30 sec - exiting 05:22:51 (5648): No heartbeat from core client for 30 sec - exiting 05:22:52 (5648): No heartbeat from core client for 30 sec - exiting 05:22:53 (5648): No heartbeat from core client for 30 sec - exiting 05:22:54 (5648): No heartbeat from core client for 30 sec - exiting 05:22:55 (5648): No heartbeat from core client for 30 sec - exiting 05:22:56 (5648): No heartbeat from core client for 30 sec - exiting 05:22:57 (5648): No heartbeat from core client for 30 sec - exiting 05:22:58 (5648): No heartbeat from core client for 30 sec - exiting 05:22:59 (5648): No heartbeat from core client for 30 sec - exiting 05:23:00 (5648): No heartbeat from core client for 30 sec - exiting 05:23:01 (5648): No heartbeat from core client for 30 sec - exiting 05:23:02 (5648): No heartbeat from core client for 30 sec - exiting 05:23:03 (5648): No heartbeat from core client for 30 sec - exiting 05:23:04 (5648): No heartbeat from core client for 30 sec - exiting 05:23:05 (5648): No heartbeat from core client for 30 sec - exiting 05:23:06 (5648): No heartbeat from core client for 30 sec - exiting 05:23:07 (5648): No heartbeat from core client for 30 sec - exiting 05:23:08 (5648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:39:50 (7976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:39:51 (7976): No heartbeat from core client for 30 sec - exiting 06:10:05 (5728): No heartbeat from core client for 30 sec - exiting 06:10:06 (5728): No heartbeat from core client for 30 sec - exiting 06:10:07 (5728): No heartbeat from core client for 30 sec - exiting 06:10:08 (5728): No heartbeat from core client for 30 sec - exiting 06:10:09 (5728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 09:06:23 (2956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:06:24 (2956): No heartbeat from core client for 30 sec - exiting 09:06:25 (2956): No heartbeat from core client for 30 sec - exiting 09:06:26 (2956): No heartbeat from core client for 30 sec - exiting 09:06:27 (2956): No heartbeat from core client for 30 sec - exiting 09:06:28 (2956): No heartbeat from core client for 30 sec - exiting 09:08:02 (4804): No heartbeat from core client for 30 sec - exiting 09:08:03 (4804): No heartbeat from core client for 30 sec - exiting 09:08:04 (4804): No heartbeat from core client for 30 sec - exiting 09:08:05 (4804): No heartbeat from core client for 30 sec - exiting 09:08:06 (4804): No heartbeat from core client for 30 sec - exiting 09:08:07 (4804): No heartbeat from core client for 30 sec - exiting 09:08:08 (4804): No heartbeat from core client for 30 sec - exiting 09:08:09 (4804): No heartbeat from core client for 30 sec - exiting 09:08:10 (4804): No heartbeat from core client for 30 sec - exiting 09:08:11 (4804): No heartbeat from core client for 30 sec - exiting 09:08:12 (4804): No heartbeat from core client for 30 sec - exiting 09:08:13 (4804): No heartbeat from core client for 30 sec - exiting 09:08:14 (4804): No heartbeat from core client for 30 sec - exiting 09:08:15 (4804): No heartbeat from core client for 30 sec - exiting 09:08:16 (4804): No heartbeat from core client for 30 sec - exiting 09:08:17 (4804): No heartbeat from core client for 30 sec - exiting 09:08:18 (4804): No heartbeat from core client for 30 sec - exiting 09:08:19 (4804): No heartbeat from core client for 30 sec - exiting 09:08:20 (4804): No heartbeat from core client for 30 sec - exiting 09:08:21 (4804): No heartbeat from core client for 30 sec - exiting 09:08:22 (4804): No heartbeat from core client for 30 sec - exiting 09:08:23 (4804): No heartbeat from core client for 30 sec - exiting 09:08:24 (4804): No heartbeat from core client for 30 sec - exiting 09:08:25 (4804): No heartbeat from core client for 30 sec - exiting 09:08:26 (4804): No heartbeat from core client for 30 sec - exiting 09:08:27 (4804): No heartbeat from core client for 30 sec - exiting 09:08:28 (4804): No heartbeat from core client for 30 sec - exiting 09:08:29 (4804): No heartbeat from core client for 30 sec - exiting 09:08:30 (4804): No heartbeat from core client for 30 sec - exiting 09:08:31 (4804): No heartbeat from core client for 30 sec - exiting 09:08:32 (4804): No heartbeat from core client for 30 sec - exiting 09:08:33 (4804): No heartbeat from core client for 30 sec - exiting 09:08:34 (4804): No heartbeat from core client for 30 sec - exiting 09:08:35 (4804): No heartbeat from core client for 30 sec - exiting 09:08:36 (4804): No heartbeat from core client for 30 sec - exiting 09:08:37 (4804): No heartbeat from core client for 30 sec - exiting 09:08:38 (4804): No heartbeat from core client for 30 sec - exiting 09:08:39 (4804): No heartbeat from core client for 30 sec - exiting 09:08:40 (4804): No heartbeat from core client for 30 sec - exiting 09:08:41 (4804): No heartbeat from core client for 30 sec - exiting 09:08:42 (4804): No heartbeat from core client for 30 sec - exiting 09:08:43 (4804): No heartbeat from core client for 30 sec - exiting 09:08:44 (4804): No heartbeat from core client for 30 sec - exiting 09:08:45 (4804): No heartbeat from core client for 30 sec - exiting 09:08:46 (4804): No heartbeat from core client for 30 sec - exiting 09:08:47 (4804): No heartbeat from core client for 30 sec - exiting 09:08:48 (4804): No heartbeat from core client for 30 sec - exiting 09:08:49 (4804): No heartbeat from core client for 30 sec - exiting 09:08:50 (4804): No heartbeat from core client for 30 sec - exiting 09:08:51 (4804): No heartbeat from core client for 30 sec - exiting 09:08:52 (4804): No heartbeat from core client for 30 sec - exiting 09:08:53 (4804): No heartbeat from core client for 30 sec - exiting 09:08:54 (4804): No heartbeat from core client for 30 sec - exiting 09:08:55 (4804): No heartbeat from core client for 30 sec - exiting 09:08:56 (4804): No heartbeat from core client for 30 sec - exiting 09:08:57 (4804): No heartbeat from core client for 30 sec - exiting 09:08:58 (4804): No heartbeat from core client for 30 sec - exiting 09:08:59 (4804): No heartbeat from core client for 30 sec - exiting 09:09:00 (4804): No heartbeat from core client for 30 sec - exiting 09:09:01 (4804): No heartbeat from core client for 30 sec - exiting 09:09:02 (4804): No heartbeat from core client for 30 sec - exiting 09:09:03 (4804): No heartbeat from core client for 30 sec - exiting 09:09:04 (4804): No heartbeat from core client for 30 sec - exiting 09:09:05 (4804): No heartbeat from core client for 30 sec - exiting 09:09:06 (4804): No heartbeat from core client for 30 sec - exiting 09:09:07 (4804): No heartbeat from core client for 30 sec - exiting 09:09:08 (4804): No heartbeat from core client for 30 sec - exiting 09:09:09 (4804): No heartbeat from core client for 30 sec - exiting 09:09:10 (4804): No heartbeat from core client for 30 sec - exiting 09:09:11 (4804): No heartbeat from core client for 30 sec - exiting 09:09:12 (4804): No heartbeat from core client for 30 sec - exiting 09:09:13 (4804): No heartbeat from core client for 30 sec - exiting 09:09:14 (4804): No heartbeat from core client for 30 sec - exiting 09:09:15 (4804): No heartbeat from core client for 30 sec - exiting 09:09:16 (4804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:09:17 (4804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=14084, selfPID=14084, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4468, selfPID=4468, iMonCtr=2 CPDN Monitor - Quit request from BOINC... 09:08:23 (4736): No heartbeat from core client for 30 sec - exiting 09:08:24 (4736): No heartbeat from core client for 30 sec - exiting 09:08:25 (4736): No heartbeat from core client for 30 sec - exiting 09:08:26 (4736): No heartbeat from core client for 30 sec - exiting 09:08:28 (4736): No heartbeat from core client for 30 sec - exiting 09:08:29 (4736): No heartbeat from core client for 30 sec - exiting 09:08:30 (4736): No heartbeat from core client for 30 sec - exiting 09:08:31 (4736): No heartbeat from core client for 30 sec - exiting 09:08:32 (4736): No heartbeat from core client for 30 sec - exiting 09:08:33 (4736): No heartbeat from core client for 30 sec - exiting 09:08:34 (4736): No heartbeat from core client for 30 sec - exiting 09:08:35 (4736): No heartbeat from core client for 30 sec - exiting 09:08:36 (4736): No heartbeat from core client for 30 sec - exiting 09:08:37 (4736): No heartbeat from core client for 30 sec - exiting 09:08:38 (4736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:33:58 (7836): No heartbeat from core client for 30 sec - exiting 09:33:59 (7836): No heartbeat from core client for 30 sec - exiting 09:34:00 (7836): No heartbeat from core client for 30 sec - exiting 09:34:01 (7836): No heartbeat from core client for 30 sec - exiting 09:34:03 (7836): No heartbeat from core client for 30 sec - exiting 09:34:04 (7836): No heartbeat from core client for 30 sec - exiting 09:34:05 (7836): No heartbeat from core client for 30 sec - exiting 09:34:06 (7836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7688, selfPID=7688, iMonCtr=2 CPDN Monitor - Quit request from BOINC... 12:59:41 (7300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8064, selfPID=8064, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7360, selfPID=7360, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11576, selfPID=11576, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11324, selfPID=11324, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:19:30 (12240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:19:31 (12240): No heartbeat from core client for 30 sec - exiting 08:19:32 (12240): No heartbeat from core client for 30 sec - exiting 08:19:33 (12240): No heartbeat from core client for 30 sec - exiting 08:19:34 (12240): No heartbeat from core client for 30 sec - exiting 08:19:35 (12240): No heartbeat from core client for 30 sec - exiting 08:19:36 (12240): No heartbeat from core client for 30 sec - exiting 08:19:37 (12240): No heartbeat from core client for 30 sec - exiting 08:19:38 (12240): No heartbeat from core client for 30 sec - exiting 08:19:39 (12240): No heartbeat from core client for 30 sec - exiting 08:19:40 (12240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=13532, selfPID=13532, iMonCtr=2 17:14:54 (2088): No heartbeat from core client for 30 sec - exiting 17:14:55 (2088): No heartbeat from core client for 30 sec - exiting 17:14:56 (2088): No heartbeat from core client for 30 sec - exiting 17:14:57 (2088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11656, selfPID=11656, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11648, selfPID=11648, iMonCtr=2 CPDN Monitor - Quit request from BOINC... 16:22:36 (10980): No heartbeat from core client for 30 sec - exiting 16:22:38 (10980): No heartbeat from core client for 30 sec - exiting 16:22:39 (10980): No heartbeat from core client for 30 sec - exiting 16:22:40 (10980): No heartbeat from core client for 30 sec - exiting 16:22:41 (10980): No heartbeat from core client for 30 sec - exiting 16:22:42 (10980): No heartbeat from core client for 30 sec - exiting 16:22:43 (10980): No heartbeat from core client for 30 sec - exiting 16:22:44 (10980): No heartbeat from core client for 30 sec - exiting 16:22:45 (10980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:24:06 (13572): No heartbeat from core client for 30 sec - exiting 16:24:07 (13572): No heartbeat from core client for 30 sec - exiting 16:24:08 (13572): No heartbeat from core client for 30 sec - exiting 16:24:10 (13572): No heartbeat from core client for 30 sec - exiting 16:24:11 (13572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Precis Restart file copy #1 failed on 2z3dga.dai0460 Atmos Restart file copy failed on atmos_restart.day Precis Restart file copy #1 failed on 2z3dga.dai0470 Atmos Restart file copy failed on atmos_restart.day Precis Restart file copy #1 failed on 2z3dga.dai0480 Atmos Restart file copy failed on atmos_restart.day Precis Restart file copy #1 failed on 2z3dga.dai0490 Atmos Restart file copy failed on atmos_restart.day Precis Restart file copy #1 failed on 2z3dga.dai04a0 Atmos Restart file copy failed on atmos_restart.day Precis Restart file copy #1 failed on 2z3dga.dai04b0 Atmos Restart file copy failed on atmos_restart.day Precis Restart file copy #1 failed on 2z3dga.dai04c0 Atmos Restart file copy failed on atmos_restart.day CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8924, selfPID=8924, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6008, selfPID=6008, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3648, selfPID=3648, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:08:35 (13552): No heartbeat from core client for 30 sec - exiting 04:08:36 (13552): No heartbeat from core client for 30 sec - exiting 04:08:37 (13552): No heartbeat from core client for 30 sec - exiting 04:08:38 (13552): No heartbeat from core client for 30 sec - exiting 04:08:39 (13552): No heartbeat from core client for 30 sec - exiting 04:08:40 (13552): No heartbeat from core client for 30 sec - exiting 04:08:41 (13552): No heartbeat from core client for 30 sec - exiting 04:08:42 (13552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:49:26 (14596): No heartbeat from core client for 30 sec - exiting 11:49:27 (14596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... RCPDN Monitor - Quit request from BOINC... forrtl: Not enough storage is available to process this command. CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/xaakm.pipe_dummy 2048 Leaving CPDN_Main::Monitor... Regional yearly means requires 12 input files got 4 01:49:00 (11732): called boinc_finish </stderr_txt> <message> <file_xfer_error> <file_name>hadam3p_pnw_2z3d_1979_1_007178977_0_5.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_pnw_2z3d_1979_1_007178977_0_6.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_pnw_2z3d_1979_1_007178977_0_7.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_pnw_2z3d_1979_1_007178977_0_8.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_pnw_2z3d_1979_1_007178977_0_9.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_pnw_2z3d_1979_1_007178977_0_10.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_pnw_2z3d_1979_1_007178977_0_11.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_pnw_2z3d_1979_1_007178977_0_12.zip</file_name> <error_code>-161</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Mar 2011 21:05:57	729212	12618321	hadam3p_pnw_2z3d_1979_1_007178977_0	46,177	231,681	5.0172
20 Mar 2011 20:29:21	729212	12618321	hadam3p_pnw_2z3d_1979_1_007178977_0	46,176	231,005	5.0027
16 Mar 2011 22:31:08	729212	12618321	hadam3p_pnw_2z3d_1979_1_007178977_0	34,656	174,550	5.0366
14 Mar 2011 19:29:13	729212	12618321	hadam3p_pnw_2z3d_1979_1_007178977_0	23,161	117,062	5.0543
14 Mar 2011 19:00:29	729212	12618321	hadam3p_pnw_2z3d_1979_1_007178977_0	23,146	116,255	5.0227
14 Mar 2011 18:37:14	729212	12618321	hadam3p_pnw_2z3d_1979_1_007178977_0	23,136	115,549	4.9943
11 Mar 2011 01:20:01	729212	12618321	hadam3p_pnw_2z3d_1979_1_007178977_0	11,616	58,330	5.0215