Task 15837204

Name	hadcm3n_4l8a_1940_40_008305000_3
Workunit	8456135
Created	10 Jun 2013, 6:01:09 UTC
Sent	10 Jun 2013, 6:20:51 UTC
Report deadline	9 Sep 2013, 13:48:02 UTC
Received	10 Oct 2013, 5:59:29 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1229098
Run time	48 days 7 hours 34 min 31 sec
CPU time	44 days 16 hours 29 min 9 sec
Validate state	Invalid
Credit	9,953.28
Device peak FLOPS	0.62 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> Das Gerät erkennt den Befehl nicht. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5020, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3312, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:04:38 (5400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:10:50 (2628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:17:00 (4572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:23:12 (4172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:32:19 (6664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:47:46 (5320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:12:27 (4620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:15:35 (3076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5748, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4636, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5040, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold 09:44:41 (4584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:34:55 (4452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:38:50 (6544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:56:53 (5164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:58:31 (2616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:00:09 (5960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:01:15 (4736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:13:56 (6172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 16:27:08 (6548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:27:43 (6016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4852, iMonCtr=1 Model crash detected, will try to restart... 08:20:38 (1364): Can't acquire lockfile (32) - waiting 35s 08:21:13 (1364): Can't acquire lockfile (32) - exiting 08:21:13 (1364): Error: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird. (0x20) 20:00:32 (5396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:03:21 (1624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:07:25 (860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:14:36 (7132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:14:37 (7132): No heartbeat from core client for 30 sec - exiting 22:15:33 (4828): No heartbeat from core client for 30 sec - exiting 22:15:34 (4828): No heartbeat from core client for 30 sec - exiting 22:15:35 (4828): No heartbeat from core client for 30 sec - exiting 22:15:37 (4828): No heartbeat from core client for 30 sec - exiting 22:15:38 (4828): No heartbeat from core client for 30 sec - exiting 22:15:40 (4828): No heartbeat from core client for 30 sec - exiting 22:15:41 (4828): No heartbeat from core client for 30 sec - exiting 22:15:43 (4828): No heartbeat from core client for 30 sec - exiting 22:15:44 (4828): No heartbeat from core client for 30 sec - exiting 22:15:45 (4828): No heartbeat from core client for 30 sec - exiting 22:15:46 (4828): No heartbeat from core client for 30 sec - exiting 22:15:48 (4828): No heartbeat from core client for 30 sec - exiting 22:15:49 (4828): No heartbeat from core client for 30 sec - exiting 22:15:50 (4828): No heartbeat from core client for 30 sec - exiting 22:15:51 (4828): No heartbeat from core client for 30 sec - exiting 22:15:52 (4828): No heartbeat from core client for 30 sec - exiting 22:15:54 (4828): No heartbeat from core client for 30 sec - exiting 22:15:55 (4828): No heartbeat from core client for 30 sec - exiting 22:15:56 (4828): No heartbeat from core client for 30 sec - exiting 22:15:57 (4828): No heartbeat from core client for 30 sec - exiting 22:15:59 (4828): No heartbeat from core client for 30 sec - exiting 22:16:00 (4828): No heartbeat from core client for 30 sec - exiting 22:16:01 (4828): No heartbeat from core client for 30 sec - exiting 22:16:02 (4828): No heartbeat from core client for 30 sec - exiting 22:16:04 (4828): No heartbeat from core client for 30 sec - exiting 22:16:06 (4828): No heartbeat from core client for 30 sec - exiting 22:16:07 (4828): No heartbeat from core client for 30 sec - exiting 22:16:08 (4828): No heartbeat from core client for 30 sec - exiting 22:16:09 (4828): No heartbeat from core client for 30 sec - exiting 22:16:10 (4828): No heartbeat from core client for 30 sec - exiting 22:16:12 (4828): No heartbeat from core client for 30 sec - exiting 22:16:13 (4828): No heartbeat from core client for 30 sec - exiting 22:16:14 (4828): No heartbeat from core client for 30 sec - exiting 22:16:15 (4828): No heartbeat from core client for 30 sec - exiting 22:16:17 (4828): No heartbeat from core client for 30 sec - exiting 22:16:18 (4828): No heartbeat from core client for 30 sec - exiting 22:16:19 (4828): No heartbeat from core client for 30 sec - exiting 22:16:20 (4828): No heartbeat from core client for 30 sec - exiting 22:16:22 (4828): No heartbeat from core client for 30 sec - exiting 22:16:23 (4828): No heartbeat from core client for 30 sec - exiting 22:16:25 (4828): No heartbeat from core client for 30 sec - exiting 22:16:26 (4828): No heartbeat from core client for 30 sec - exiting 22:16:27 (4828): No heartbeat from core client for 30 sec - exiting 22:16:28 (4828): No heartbeat from core client for 30 sec - exiting 22:16:29 (4828): No heartbeat from core client for 30 sec - exiting 22:16:31 (4828): No heartbeat from core client for 30 sec - exiting 22:16:32 (4828): No heartbeat from core client for 30 sec - exiting 22:16:33 (4828): No heartbeat from core client for 30 sec - exiting 22:16:34 (4828): No heartbeat from core client for 30 sec - exiting 22:16:36 (4828): No heartbeat from core client for 30 sec - exiting 22:16:37 (4828): No heartbeat from core client for 30 sec - exiting 22:16:38 (4828): No heartbeat from core client for 30 sec - exiting 22:16:39 (4828): No heartbeat from core client for 30 sec - exiting 22:16:40 (4828): No heartbeat from core client for 30 sec - exiting 22:16:42 (4828): No heartbeat from core client for 30 sec - exiting 22:16:43 (4828): No heartbeat from core client for 30 sec - exiting 22:16:44 (4828): No heartbeat from core client for 30 sec - exiting 22:16:45 (4828): No heartbeat from core client for 30 sec - exiting 22:16:47 (4828): No heartbeat from core client for 30 sec - exiting 22:16:48 (4828): No heartbeat from core client for 30 sec - exiting 22:16:49 (4828): No heartbeat from core client for 30 sec - exiting 22:16:50 (4828): No heartbeat from core client for 30 sec - exiting 22:16:52 (4828): No heartbeat from core client for 30 sec - exiting 22:16:53 (4828): No heartbeat from core client for 30 sec - exiting 22:16:54 (4828): No heartbeat from core client for 30 sec - exiting 22:16:55 (4828): No heartbeat from core client for 30 sec - exiting 22:16:56 (4828): No heartbeat from core client for 30 sec - exiting 22:16:58 (4828): No heartbeat from core client for 30 sec - exiting 22:16:59 (4828): No heartbeat from core client for 30 sec - exiting 22:17:00 (4828): No heartbeat from core client for 30 sec - exiting 22:17:01 (4828): No heartbeat from core client for 30 sec - exiting 22:17:03 (4828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:17:04 (4828): No heartbeat from core client for 30 sec - exiting 22:17:05 (4828): No heartbeat from core client for 30 sec - exiting 22:17:06 (4828): No heartbeat from core client for 30 sec - exiting 22:17:08 (4828): No heartbeat from core client for 30 sec - exiting 22:17:09 (4828): No heartbeat from core client for 30 sec - exiting 22:17:10 (4828): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:16:55 (5528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 11:07:39 (3440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold 11:34:19 (4516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4832, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:36:42 (4132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4l8aka.dah1cg0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4l8aka.dah1ch0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4l8aka.dah1ci0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4l8aka.dah1cj0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4l8aka.dah1ck0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4l8aka.dah1cl0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4l8aka.dah1cm0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4l8aka.dah1cn0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4l8aka.dah1co0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4l8aka.dah1cp0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4l8aka.dah1cq0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 4l8aka.dah1cr0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Oct 2013 07:11:45	1229098	15837204	hadcm3n_4l8a_1940_40_008305000_3	829,440	3,838,965	4.6284
07 Oct 2013 08:19:56	1229098	15837204	hadcm3n_4l8a_1940_40_008305000_3	803,520	3,758,951	4.6781
06 Oct 2013 11:40:55	1229098	15837204	hadcm3n_4l8a_1940_40_008305000_3	777,600	3,680,591	4.7333
06 Oct 2013 09:35:22	1229098	15837204	hadcm3n_4l8a_1940_40_008305000_3	751,680	3,602,314	4.7924
04 Oct 2013 17:05:12	1229098	15837204	hadcm3n_4l8a_1940_40_008305000_3	725,760	3,522,173	4.8531
02 Oct 2013 06:38:09	1229098	15837204	hadcm3n_4l8a_1940_40_008305000_3	699,840	3,371,649	4.8177
30 Sep 2013 08:55:23	1229098	15837204	hadcm3n_4l8a_1940_40_008305000_3	673,920	3,250,337	4.8230
26 Sep 2013 06:21:10	1229098	15837204	hadcm3n_4l8a_1940_40_008305000_3	648,000	3,153,750	4.8669
25 Sep 2013 09:23:12	1229098	15837204	hadcm3n_4l8a_1940_40_008305000_3	622,080	3,077,322	4.9468
18 Sep 2013 16:56:43	1229098	15837204	hadcm3n_4l8a_1940_40_008305000_3	596,160	2,916,265	4.8917
17 Sep 2013 21:29:53	1229098	15837204	hadcm3n_4l8a_1940_40_008305000_3	570,240	2,848,540	4.9953
16 Sep 2013 21:50:19	1229098	15837204	hadcm3n_4l8a_1940_40_008305000_3	544,320	2,767,840	5.0850
15 Sep 2013 05:48:10	1229098	15837204	hadcm3n_4l8a_1940_40_008305000_3	518,400	2,627,636	5.0687
11 Sep 2013 13:02:09	1229098	15837204	hadcm3n_4l8a_1940_40_008305000_3	492,480	2,517,510	5.1119
03 Sep 2013 19:10:12	1229098	15837204	hadcm3n_4l8a_1940_40_008305000_3	466,560	2,437,491	5.2244
02 Sep 2013 07:08:27	1229098	15837204	hadcm3n_4l8a_1940_40_008305000_3	440,640	2,334,255	5.2974
29 Aug 2013 17:13:58	1229098	15837204	hadcm3n_4l8a_1940_40_008305000_3	414,720	2,142,233	5.1655
14 Aug 2013 16:08:57	1229098	15837204	hadcm3n_4l8a_1940_40_008305000_3	388,800	1,980,065	5.0928
14 Aug 2013 16:08:57	1229098	15837204	hadcm3n_4l8a_1940_40_008305000_3	362,880	1,812,844	4.9957
14 Aug 2013 16:08:57	1229098	15837204	hadcm3n_4l8a_1940_40_008305000_3	336,960	1,735,548	5.1506