Task 16071348

Name	hadcm3n_4bzy_2020_40_008390275_4
Workunit	8541134
Created	21 Oct 2013, 12:15:48 UTC
Sent	21 Oct 2013, 12:16:08 UTC
Report deadline	20 Jan 2014, 19:43:19 UTC
Received	21 Dec 2013, 4:27:42 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1297903
Run time	18 days 3 hours 55 min 57 sec
CPU time	17 days 22 hours 54 min 51 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.58 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> 09:08:54 (2504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:08:55 (2504): No heartbeat from core client for 30 sec - exiting 09:08:56 (2504): No heartbeat from core client for 30 sec - exiting 09:08:57 (2504): No heartbeat from core client for 30 sec - exiting 09:08:58 (2504): No heartbeat from core client for 30 sec - exiting 09:09:00 (2504): No heartbeat from core client for 30 sec - exiting 09:09:01 (2504): No heartbeat from core client for 30 sec - exiting 09:09:02 (2504): No heartbeat from core client for 30 sec - exiting 09:09:03 (2504): No heartbeat from core client for 30 sec - exiting 09:09:04 (2504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2052, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2672, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2544, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2544, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 09:20:52 (3788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:20:53 (3788): No heartbeat from core client for 30 sec - exiting 09:20:54 (3788): No heartbeat from core client for 30 sec - exiting 09:20:55 (3788): No heartbeat from core client for 30 sec - exiting 09:20:56 (3788): No heartbeat from core client for 30 sec - exiting 09:20:57 (3788): No heartbeat from core client for 30 sec - exiting 09:20:58 (3788): No heartbeat from core client for 30 sec - exiting 09:20:59 (3788): No heartbeat from core client for 30 sec - exiting 09:21:00 (3788): No heartbeat from core client for 30 sec - exiting 09:21:01 (3788): No heartbeat from core client for 30 sec - exiting 09:21:02 (3788): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3356, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2924, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3168, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3676, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2880, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1560, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1560, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:40:51 (3324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3884, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3212, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3172, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: C I/O Error feof - Unit 62 - Return code = 16 BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/4bzyko.pjq0c10 Error converting file to netcdf: dataout/4bzyko.piq0c10 Error converting file to netcdf: dataout/4bzyko.pfq0c10 Error converting file to netcdf: dataout/4bzyko.pcq0c10 Error converting file to netcdf: dataout/4bzyko.pbq0c10 Error converting file to netcdf: dataout/4bzyko.paq0c10 Error converting file to netcdf: dataout/4bzyka.phq0c10 Error converting file to netcdf: dataout/4bzyka.pgq0c10 Error converting file to netcdf: dataout/4bzyka.peq0c10 Error converting file to netcdf: dataout/4bzyka.pdq0c10 BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: C I/O Error feof - Unit 62 - Return code = 16 BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/4bzyko.pjq0c10 Error converting file to netcdf: dataout/4bzyko.piq0c10 Error converting file to netcdf: dataout/4bzyko.pfq0c10 Error converting file to netcdf: dataout/4bzyko.pcq0c10 Error converting file to netcdf: dataout/4bzyko.pbq0c10 Error converting file to netcdf: dataout/4bzyko.paq0c10 Error converting file to netcdf: dataout/4bzyka.phq0c10 Error converting file to netcdf: dataout/4bzyka.pgq0c10 Error converting file to netcdf: dataout/4bzyka.peq0c10 Error converting file to netcdf: dataout/4bzyka.pdq0c10 Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
21 Dec 2013 04:30:20	1297903	16071348	hadcm3n_4bzy_2020_40_008390275_4	1,036,800	1,551,289	1.4962
19 Dec 2013 08:05:14	1297903	16071348	hadcm3n_4bzy_2020_40_008390275_4	1,010,880	1,502,592	1.4864
17 Dec 2013 13:07:54	1297903	16071348	hadcm3n_4bzy_2020_40_008390275_4	984,960	1,454,302	1.4765
16 Dec 2013 10:14:20	1297903	16071348	hadcm3n_4bzy_2020_40_008390275_4	959,040	1,412,263	1.4726
13 Dec 2013 12:44:21	1297903	16071348	hadcm3n_4bzy_2020_40_008390275_4	933,120	1,375,103	1.4737
12 Dec 2013 11:46:44	1297903	16071348	hadcm3n_4bzy_2020_40_008390275_4	907,200	1,341,695	1.4789
11 Dec 2013 11:29:07	1297903	16071348	hadcm3n_4bzy_2020_40_008390275_4	881,280	1,307,288	1.4834
10 Dec 2013 09:06:34	1297903	16071348	hadcm3n_4bzy_2020_40_008390275_4	855,360	1,271,871	1.4869
09 Dec 2013 08:24:17	1297903	16071348	hadcm3n_4bzy_2020_40_008390275_4	829,440	1,234,358	1.4882
06 Dec 2013 10:03:18	1297903	16071348	hadcm3n_4bzy_2020_40_008390275_4	803,520	1,194,840	1.4870
05 Dec 2013 06:56:59	1297903	16071348	hadcm3n_4bzy_2020_40_008390275_4	777,600	1,152,674	1.4823
03 Dec 2013 14:44:40	1297903	16071348	hadcm3n_4bzy_2020_40_008390275_4	751,680	1,109,779	1.4764
02 Dec 2013 09:19:04	1297903	16071348	hadcm3n_4bzy_2020_40_008390275_4	725,760	1,061,511	1.4626
29 Nov 2013 10:19:25	1297903	16071348	hadcm3n_4bzy_2020_40_008390275_4	699,840	1,014,534	1.4497
28 Nov 2013 06:26:50	1297903	16071348	hadcm3n_4bzy_2020_40_008390275_4	673,920	967,616	1.4358
26 Nov 2013 13:25:14	1297903	16071348	hadcm3n_4bzy_2020_40_008390275_4	648,000	921,880	1.4227
25 Nov 2013 10:55:55	1297903	16071348	hadcm3n_4bzy_2020_40_008390275_4	622,080	879,202	1.4133
22 Nov 2013 11:20:41	1297903	16071348	hadcm3n_4bzy_2020_40_008390275_4	596,160	836,408	1.4030
21 Nov 2013 08:17:43	1297903	16071348	hadcm3n_4bzy_2020_40_008390275_4	570,240	793,976	1.3924
20 Nov 2013 05:20:50	1297903	16071348	hadcm3n_4bzy_2020_40_008390275_4	544,320	751,317	1.3803