Task 15788973

Name	hadcm3n_3ejc_1940_40_008258419_4
Workunit	8413543
Created	18 May 2013, 8:00:48 UTC
Sent	18 May 2013, 8:00:49 UTC
Report deadline	17 Aug 2013, 15:28:00 UTC
Received	28 Jun 2013, 9:25:16 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1268577
Run time	2 days 7 hours 2 min 15 sec
CPU time	2 days 5 hours 38 min
Validate state	Invalid
Credit	2,177.28
Device peak FLOPS	2.80 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae13u0 Ocean Restart file copy failed on 3ejcko.dae1410 Ocean Restart file copy failed on 3ejcko.dae1420 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 3ejcka.dae1460 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae14q0 Ocean Restart file copy failed on 3ejcko.dae14r0 Ocean Restart file copy failed on 3ejcko.dae14s0 Ocean Restart file copy failed on 3ejcko.dae1560 Ocean Restart file copy failed on 3ejcko.dae1570 Ocean Restart file copy failed on 3ejcko.dae1580 Ocean Restart file copy failed on 3ejcko.dae1590 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae15p0 Ocean Restart file copy failed on 3ejcko.dae15q0 Ocean Restart file copy failed on 3ejcko.dae15r0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae16p0 Ocean Restart file copy failed on 3ejcko.dae16q0 Ocean Restart file copy failed on 3ejcko.dae16r0 Ocean Restart file copy failed on 3ejcko.dae16s0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae18b0 Ocean Restart file copy failed on 3ejcko.dae18c0 Ocean Restart file copy failed on 3ejcko.dae18d0 Ocean Restart file copy failed on 3ejcko.dae18e0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae18t0 Ocean Restart file copy failed on 3ejcko.dae18u0 Ocean Restart file copy failed on 3ejcko.dae1910 Ocean Restart file copy failed on 3ejcko.dae1920 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae19q0 Ocean Restart file copy failed on 3ejcko.dae19r0 Ocean Restart file copy failed on 3ejcko.dae19s0 Ocean Restart file copy failed on 3ejcko.dae19t0 Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3ejcko.dae24n0 Ocean Restart file copy failed on 3ejcko.dae24o0 Ocean Restart file copy failed on 3ejcko.dae24p0 Ocean Restart file copy failed on 3ejcko.dae24q0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae2670 Ocean Restart file copy failed on 3ejcko.dae2680 Ocean Restart file copy failed on 3ejcko.dae2690 Ocean Restart file copy failed on 3ejcko.dae26a0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae27j0 Ocean Restart file copy failed on 3ejcko.dae27k0 Ocean Restart file copy failed on 3ejcko.dae27l0 Ocean Restart file copy failed on 3ejcko.dae27m0 Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3ejcko.dae28r0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3ejcko.dae28s0 Ocean Restart file copy failed on 3ejcko.dae28t0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae29h0 Ocean Restart file copy failed on 3ejcko.dae29i0 Ocean Restart file copy failed on 3ejcko.dae29j0 Ocean Restart file copy failed on 3ejcko.dae29k0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae2ab0 Ocean Restart file copy failed on 3ejcko.dae2ac0 Ocean Restart file copy failed on 3ejcko.dae2ad0 Ocean Restart file copy failed on 3ejcko.dae2ae0 Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae2ba0 Ocean Restart file copy failed on 3ejcko.dae2bb0 Ocean Restart file copy failed on 3ejcko.dae2bc0 Ocean Restart file copy failed on 3ejcko.dae2cl0 Ocean Restart file copy failed on 3ejcko.dae2cm0 Ocean Restart file copy failed on 3ejcko.dae2cn0 Ocean Restart file copy failed on 3ejcko.dae3140 Ocean Restart file copy failed on 3ejcko.dae3150 Ocean Restart file copy failed on 3ejcko.dae3160 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae31m0 Ocean Restart file copy failed on 3ejcko.dae31n0 Ocean Restart file copy failed on 3ejcko.dae31o0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae32i0 Ocean Restart file copy failed on 3ejcko.dae32j0 Ocean Restart file copy failed on 3ejcko.dae32k0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae33b0 Ocean Restart file copy failed on 3ejcko.dae33c0 Ocean Restart file copy failed on 3ejcko.dae33d0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae34j0 Ocean Restart file copy failed on 3ejcko.dae34k0 Ocean Restart file copy failed on 3ejcko.dae34l0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae35u0 Ocean Restart file copy failed on 3ejcko.dae3610 Ocean Restart file copy failed on 3ejcko.dae3620 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae36p0 Ocean Restart file copy failed on 3ejcko.dae36q0 Ocean Restart file copy failed on 3ejcko.dae36r0 Ocean Restart file copy failed on 3ejcko.dae36s0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae3970 Ocean Restart file copy failed on 3ejcko.dae3980 Ocean Restart file copy failed on 3ejcko.dae3990 Ocean Restart file copy failed on 3ejcko.dae39a0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae3ao0 Ocean Restart file copy failed on 3ejcko.dae3ap0 Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3ejcko.dae3aq0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae3b20 Ocean Restart file copy failed on 3ejcko.dae3b30 Ocean Restart file copy failed on 3ejcko.dae3b40 Ocean Restart file copy failed on 3ejcko.dae3b50 Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae3cg0 Ocean Restart file copy failed on 3ejcko.dae3ch0 Ocean Restart file copy failed on 3ejcko.dae3ci0 Ocean Restart file copy failed on 3ejcko.dae3cj0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6348, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6348, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6348, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 3ejcka.dae41o0 Ocean Restart file copy failed on 3ejcko.dae41o0 Ocean Restart file copy failed on 3ejcko.dae41p0 Ocean Restart file copy failed on 3ejcko.dae41q0 Ocean Restart file copy failed on 3ejcko.dae41r0 Ocean Restart file copy failed on 3ejcko.dae4250 Ocean Restart file copy failed on 3ejcko.dae4260 Ocean Restart file copy failed on 3ejcko.dae4270 Ocean Restart file copy failed on 3ejcko.dae4280 Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3ejcko.dae45g0 Ocean Restart file copy failed on 3ejcko.dae45h0 Ocean Restart file copy failed on 3ejcko.dae45i0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae4890 Ocean Restart file copy failed on 3ejcko.dae48a0 Ocean Restart file copy failed on 3ejcko.dae48b0 Ocean Restart file copy failed on 3ejcko.dae48c0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae4910 Ocean Restart file copy failed on 3ejcko.dae4920 Ocean Restart file copy failed on 3ejcko.dae4930 Ocean Restart file copy failed on 3ejcko.dae4940 Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae4b10 Ocean Restart file copy failed on 3ejcko.dae4b20 Ocean Restart file copy failed on 3ejcko.dae4b30 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae4c70 Ocean Restart file copy failed on 3ejcko.dae4c80 Ocean Restart file copy failed on 3ejcko.dae4c90 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae5180 Ocean Restart file copy failed on 3ejcko.dae5190 Ocean Restart file copy failed on 3ejcko.dae51a0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae5210 Ocean Restart file copy failed on 3ejcko.dae5220 Ocean Restart file copy failed on 3ejcko.dae5230 Ocean Restart file copy failed on 3ejcko.dae5240 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae52f0 Ocean Restart file copy failed on 3ejcko.dae52g0 Ocean Restart file copy failed on 3ejcko.dae52h0 Ocean Restart file copy failed on 3ejcko.dae52i0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae53d0 Ocean Restart file copy failed on 3ejcko.dae53e0 Ocean Restart file copy failed on 3ejcko.dae53f0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae54c0 Ocean Restart file copy failed on 3ejcko.dae54d0 Ocean Restart file copy failed on 3ejcko.dae54e0 Ocean Restart file copy failed on 3ejcko.dae54f0 Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3ejcko.dae55o0 Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3ejcko.dae55p0 Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3ejcko.dae55q0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae56t0 Ocean Restart file copy failed on 3ejcko.dae56u0 Ocean Restart file copy failed on 3ejcko.dae5710 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae57d0 Ocean Restart file copy failed on 3ejcko.dae57e0 Ocean Restart file copy failed on 3ejcko.dae57f0 Ocean Restart file copy failed on 3ejcko.dae57g0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae58j0 Ocean Restart file copy failed on 3ejcko.dae58k0 Ocean Restart file copy failed on 3ejcko.dae58l0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae59g0 Ocean Restart file copy failed on 3ejcko.dae59h0 Ocean Restart file copy failed on 3ejcko.dae59i0 Ocean Restart file copy failed on 3ejcko.dae59j0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae5a60 Ocean Restart file copy failed on 3ejcko.dae5a70 Ocean Restart file copy failed on 3ejcko.dae5a80 Ocean Restart file copy failed on 3ejcko.dae5a90 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae5bf0 Ocean Restart file copy failed on 3ejcko.dae5bg0 Ocean Restart file copy failed on 3ejcko.dae5bh0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae5ca0 Ocean Restart file copy failed on 3ejcko.dae5cb0 Ocean Restart file copy failed on 3ejcko.dae5cc0 Ocean Restart file copy failed on 3ejcko.dae5cd0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae6150 Ocean Restart file copy failed on 3ejcko.dae6160 Ocean Restart file copy failed on 3ejcko.dae6170 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae62k0 Ocean Restart file copy failed on 3ejcko.dae62l0 Ocean Restart file copy failed on 3ejcko.dae62m0 Ocean Restart file copy failed on 3ejcko.dae62n0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae63j0 Ocean Restart file copy failed on 3ejcko.dae63k0 Ocean Restart file copy failed on 3ejcko.dae63l0 Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:50:42 (8000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:50:44 (8000): No heartbeat from core client for 30 sec - exiting 07:50:45 (8000): No heartbeat from core client for 30 sec - exiting 07:50:46 (8000): No heartbeat from core client for 30 sec - exiting 07:50:47 (8000): No heartbeat from core client for 30 sec - exiting 07:50:48 (8000): No heartbeat from core client for 30 sec - exiting 07:50:49 (8000): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3ejcko.dae6740 Ocean Restart file copy failed on 3ejcko.dae6750 Ocean Restart file copy failed on 3ejcko.dae6760 Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3ejcko.dae67o0 Ocean Restart file copy failed on 3ejcko.dae67p0 Ocean Restart file copy failed on 3ejcko.dae67q0 Ocean Restart file copy failed on 3ejcko.dae67r0 Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3ejcko.dae68a0 Ocean Restart file copy failed on 3ejcko.dae68b0 Ocean Restart file copy failed on 3ejcko.dae68c0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae6990 Ocean Restart file copy failed on 3ejcko.dae69a0 Ocean Restart file copy failed on 3ejcko.dae69b0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae69s0 Ocean Restart file copy failed on 3ejcko.dae69t0 Ocean Restart file copy failed on 3ejcko.dae69u0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae6ab0 Ocean Restart file copy failed on 3ejcko.dae6ac0 Ocean Restart file copy failed on 3ejcko.dae6ad0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae6ba0 Ocean Restart file copy failed on 3ejcko.dae6bb0 Ocean Restart file copy failed on 3ejcko.dae6bc0 Ocean Restart file copy failed on 3ejcko.dae6bd0 Ocean Restart file copy failed on 3ejcko.dae6ce0 Ocean Restart file copy failed on 3ejcko.dae6cf0 Ocean Restart file copy failed on 3ejcko.dae6cg0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 3ejcka.dae71l0 Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3ejcko.dae71u0 Ocean Restart file copy failed on 3ejcko.dae7210 Ocean Restart file copy failed on 3ejcko.dae7220 Ocean Restart file copy failed on 3ejcko.dae7230 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae73r0 Ocean Restart file copy failed on 3ejcko.dae73s0 Ocean Restart file copy failed on 3ejcko.dae73t0 Ocean Restart file copy failed on 3ejcko.dae73u0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae74a0 Ocean Restart file copy failed on 3ejcko.dae74b0 Ocean Restart file copy failed on 3ejcko.dae74c0 Ocean Restart file copy failed on 3ejcko.dae74d0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae7590 Ocean Restart file copy failed on 3ejcko.dae75a0 Ocean Restart file copy failed on 3ejcko.dae75b0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae76g0 Ocean Restart file copy failed on 3ejcko.dae76h0 Ocean Restart file copy failed on 3ejcko.dae76i0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 3ejcka.dae76q0 Ocean Restart file copy failed on 3ejcko.dae77o0 Ocean Restart file copy failed on 3ejcko.dae77p0 Ocean Restart file copy failed on 3ejcko.dae77q0 Ocean Restart file copy failed on 3ejcko.dae77r0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on 3ejcka.dae78d0 Ocean Restart file copy failed on 3ejcko.dae78p0 Ocean Restart file copy failed on 3ejcko.dae78q0 Ocean Restart file copy failed on 3ejcko.dae78r0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae79h0 Ocean Restart file copy failed on 3ejcko.dae79i0 Ocean Restart file copy failed on 3ejcko.dae79j0 Ocean Restart file copy failed on 3ejcko.dae79k0 Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae7b70 Ocean Restart file copy failed on 3ejcko.dae7b80 Ocean Restart file copy failed on 3ejcko.dae7b90 Ocean Restart file copy failed on 3ejcko.dae7ba0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3ejcko.dae7ca0 Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Jun 2013 19:24:06	1268577	15788973	hadcm3n_3ejc_1940_40_008258419_4	181,440	192,549	1.0612
19 Jun 2013 12:23:01	1268577	15788973	hadcm3n_3ejc_1940_40_008258419_4	155,520	165,118	1.0617
07 Jun 2013 16:21:54	1268577	15788973	hadcm3n_3ejc_1940_40_008258419_4	129,600	137,003	1.0571
04 Jun 2013 14:02:13	1268577	15788973	hadcm3n_3ejc_1940_40_008258419_4	103,680	109,796	1.0590
23 May 2013 08:17:58	1268577	15788973	hadcm3n_3ejc_1940_40_008258419_4	77,760	82,683	1.0633
21 May 2013 09:03:02	1268577	15788973	hadcm3n_3ejc_1940_40_008258419_4	51,840	53,868	1.0391
18 May 2013 16:32:01	1268577	15788973	hadcm3n_3ejc_1940_40_008258419_4	25,920	27,092	1.0452