Task 14088766

Name	hadcm3n_o4hb_1940_40_007617679_2
Workunit	7795809
Created	11 Feb 2012, 14:14:56 UTC
Sent	11 Feb 2012, 14:15:03 UTC
Report deadline	12 May 2012, 21:42:14 UTC
Received	19 Apr 2012, 23:14:48 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1132141
Run time	13 days 17 hours 42 min 51 sec
CPU time	12 days 7 hours 49 min 19 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.83 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3572, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o4hbko.pje6c10 Error converting file to netcdf: dataout/o4hbko.pie6c10 Error converting file to netcdf: dataout/o4hbko.pfe6c10 Error converting file to netcdf: dataout/o4hbka.phe6c10 Error converting file to netcdf: dataout/o4hbka.pge6c10 Error converting file to netcdf: dataout/o4hbka.pee6c10 Error converting file to netcdf: dataout/o4hbka.pde6c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3404, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3404, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=128, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=128, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2356, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4708, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:01:04 (3132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3604, iMonCtr=1 Model crash detected, will try to restart... 18:19:06 (5612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Apr 2012 22:14:34	1132141	14088766	hadcm3n_o4hb_1940_40_007617679_2	777,600	1,064,954	1.3695
15 Apr 2012 18:45:37	1132141	14088766	hadcm3n_o4hb_1940_40_007617679_2	751,680	1,029,518	1.3696
09 Apr 2012 13:14:23	1132141	14088766	hadcm3n_o4hb_1940_40_007617679_2	725,760	994,292	1.3700
08 Apr 2012 13:13:40	1132141	14088766	hadcm3n_o4hb_1940_40_007617679_2	699,840	959,098	1.3705
06 Apr 2012 15:59:06	1132141	14088766	hadcm3n_o4hb_1940_40_007617679_2	673,920	923,165	1.3698
06 Apr 2012 02:54:48	1132141	14088766	hadcm3n_o4hb_1940_40_007617679_2	648,000	889,193	1.3722
02 Apr 2012 19:29:01	1132141	14088766	hadcm3n_o4hb_1940_40_007617679_2	622,080	853,684	1.3723
31 Mar 2012 16:56:27	1132141	14088766	hadcm3n_o4hb_1940_40_007617679_2	596,160	818,904	1.3736
31 Mar 2012 06:17:14	1132141	14088766	hadcm3n_o4hb_1940_40_007617679_2	570,240	783,229	1.3735
28 Mar 2012 21:06:49	1132141	14088766	hadcm3n_o4hb_1940_40_007617679_2	544,320	748,023	1.3742
24 Mar 2012 19:38:18	1132141	14088766	hadcm3n_o4hb_1940_40_007617679_2	518,400	712,386	1.3742
24 Mar 2012 08:05:34	1132141	14088766	hadcm3n_o4hb_1940_40_007617679_2	492,480	675,216	1.3711
19 Mar 2012 21:20:49	1132141	14088766	hadcm3n_o4hb_1940_40_007617679_2	466,560	638,268	1.3680
18 Mar 2012 11:19:02	1132141	14088766	hadcm3n_o4hb_1940_40_007617679_2	440,640	600,957	1.3638
17 Mar 2012 08:59:22	1132141	14088766	hadcm3n_o4hb_1940_40_007617679_2	414,720	564,498	1.3612
16 Mar 2012 21:57:14	1132141	14088766	hadcm3n_o4hb_1940_40_007617679_2	388,800	528,232	1.3586
11 Mar 2012 18:39:57	1132141	14088766	hadcm3n_o4hb_1940_40_007617679_2	362,880	491,952	1.3557
10 Mar 2012 15:40:34	1132141	14088766	hadcm3n_o4hb_1940_40_007617679_2	336,960	456,680	1.3553
10 Mar 2012 05:11:52	1132141	14088766	hadcm3n_o4hb_1940_40_007617679_2	311,040	420,990	1.3535
07 Mar 2012 19:51:56	1132141	14088766	hadcm3n_o4hb_1940_40_007617679_2	285,120	385,382	1.3516